Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Distributed Stochastic Optimization for Non-Smooth and Weakly Convex Problems under Heavy-Tailed Noise

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proves that a gradient-clipping distributed subgradient algorithm converges almost surely to a stationary point of non-smooth weakly convex objectives under heavy-tailed noise, with rate O(M/log T) on the Moreau-envelope…

desk verdict The paper is the first to combine clipping, consensus, non-smooth weakly convex objectives, and heavy-tailed noise, but the proof of the main theorem has a repairable gap that must be fixed before the rate claim is trustworthy. read the letter →

arxiv 2505.09279 v1 pith:X5MN6JMY submitted 2025-05-14 math.OC

Jun Hu , Chao Sun , Bo Chen , Jianzheng Wang , Zheming Wang This is my paper · ORCID
classification math.OC MSC 90C1590C26
keywords distributedstochasticoptimizationweaklyconvexproblemsheavy-tailednoisegradientclippingMoreauenvelopeconsensussubgradientmethodphaseretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a purely decentralized, consensus-based stochastic subgradient method—where each agent clips its local gradient before communicating—can solve distributed optimization problems whose objectives are only weakly convex and non-smooth, and whose stochastic subgradient noise may have unbounded variance. It proves that the Moreau envelope of the average iterate converges almost surely, and that the expected squared norm of its gradient decays like O(M/log T) for stepsize alpha_k = 1/(k+1) and clipping threshold tau_k = 2C0(k+1)^0.4. This matters because most existing heavy-tailed distributed algorithms assume convexity and smoothness, while many machine-learning tasks are weakly convex and non-smooth, and real gradient noise is often heavy-tailed. A phase-retrieval experiment on MNIST supports the claim.

What carries the argument

The load-bearing object is the Moreau envelope $\varphi_\mu(x)=\min_{y\in\Omega}\{f(y)+\frac{1}{2\mu}\|y-x\|^2\}$ and its proximal map $\widehat{x}=\arg\min_{y\in\Omega}\{f(y)+\frac{1}{2\mu}\|y-x\|^2\}$, whose gradient satisfies $\nabla\varphi_\mu(x)=\frac{1}{\mu}(x-\widehat{x})$; a small gradient norm certifies near-stationarity. The second component is the clipping map $\widehat{g}_{i,k}=g_{i,k}\min\{1,\tau_k/\|g_{i,k}\|\}$, which truncates each local subgradient at level $\tau_k$ and, under the $\alpha$-th moment bound, has bias at most $(2\gamma)^\alpha \tau_k^{1-\alpha}$. The consensus estimate of Lemma 5 keeps the disagreement $\|\bar{x}_k-x_{i,k}\|$ small enough that all consensus error terms are summable, so the Moreau potential decreases modulo bias terms that vanish as $\tau_k\to\infty$.

What would settle it

Run the proposed algorithm on a single agent ($N=1$, so consensus is trivial) with synthetic subgradient noise drawn from a symmetric Lévy-stable distribution with tail index $\alpha=1.5$ and a non-smooth weakly convex objective such as $f(x)=|x^2-1|$; record $\inf_{k\le T}\|\nabla\varphi_\mu(\bar{x}_k)\|^2$ over many seeds. The theorem predicts decay at least $O(M/\log T)$, so observing the metric level off at a positive constant, or decay slower than logarithmic, would refute Theorem 2.

Watch

Extended reading notes

Core claim

The central assertion is Theorem 2: under Assumptions 1–8 and condition (15), the averaged iterates $\bar{x}_k$ satisfy $$E\Big[\inf_{k_0\le k\le \infty}\|\nabla \varphi_\mu(\bar{x}_k)\|^2\Big]\le \frac{2}{1-2\mu(\rho+1)}\Big(\frac{\|V_{k_0}-\$\theta$\|+\sum_{k=k_0}^\infty b_k}{\sum_{k=k_0}^\infty \alpha_k}+\frac{\sum_{k=k_0}^\infty \$tau_k^{2}$\$alpha_k^{2}$/(2\mu)+\sum_{k=k_0}^\infty \alpha_k(2\gamma)^{2\$\alpha$}\$tau_k^{{2-2\alpha}}$/(2\mu)}{\sum_{k=k_0}^\infty \alpha_k}\Big),$$ and with $\alpha_k=1/(k+1)$, $\tau_k=2C_0(k+1)^{0.4}$, this is $O(M/\log T)$. In words, the sequence approaches a stationary point of the non-smooth weakly convex global objective even though each local stochastic subgradient may have infinite variance; only a finite $\alpha$-th moment for some $\alpha\in(1,2]$ is required.

Load-bearing premise

The load-bearing premise is Assumption 8: each local stochastic subgradient must have a finite moment of some order $\alpha>1$ (variance may be infinite); if even that moment fails, the clipping bias term $(2\gamma)^\alpha\tau_k^{1-\alpha}$ does not vanish as $\tau_k\to\infty$, so the convergence argument collapses.

Editorial extensions

If this is right

  • The algorithm reaches a stationary point almost surely for any distributed weakly convex non-smooth problem whose local stochastic subgradients have a finite moment of order $\alpha>1$; neither smoothness nor bounded variance is required.
  • With stepsize $\alpha_k=1/(k+1)$ and clipping threshold $\tau_k=2C_0(k+1)^{0.4}$, the expected squared Moreau gradient of the averaged iterate decays as $O(M/\log T)$, so only logarithmically many iterations are needed to drive the stationarity metric to a given tolerance up to the constant $M$.
  • Consensus and clipping coexist: the result holds over strongly connected, time-varying, doubly stochastic graphs, so the method needs no central server.
  • The heavy-tailed regime includes infinite-variance noise ($\alpha<2$), and the bounded-variance case is recovered as the limit $\alpha=2$.
  • In the robust phase-retrieval experiment on MNIST, the clipped method reconstructs the image where unclipped distributed baselines do not, indicating practical usefulness beyond the proof.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper leaves implicit: per-agent clipping levels $\tau_{i,k}$ could be tuned to the local tail index $\alpha_i$, which may improve constants when agents observe noise of different heaviness.
  • The logarithmic rate is governed by the constant $M$, which depends on the initial Moreau gap and consensus transients; one could test whether a short warm-up with larger clipping thresholds reduces $M$ and accelerates practical convergence.
  • Because the proof only needs the tail probability of $\|g_{i,k}-G_{i,k}\| > \tau_k/2$, it suggests convergence may survive for noise with slowly varying tails whose $\alpha$-th moment exists only asymptotically, but the rate would degrade—a claim not made in the paper.
  • The analysis treats the averaging step as one consensus round per gradient step; for federated settings with multiple local updates between communications, the consensus-error bound would need reworking, but the clipping-bias argument would carry over.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a distributed stochastic subgradient method with gradient clipping for non-smooth, weakly convex objectives over time-varying networks, allowing heavy-tailed gradient noise with only a finite alpha-th moment. The main theoretical claims are Theorem 1, an almost-sure convergence of the Moreau envelope along the averaged iterates, and Theorem 2, an expectation bound on the squared norm of the Moreau-envelope gradient together with an O(M/log T) rate for a particular choice of parameters. The paper also reports numerical experiments on a phase-retrieval problem using MNIST measurements.

Significance. If the results were rigorously established, this would be a useful contribution: it extends clipping-based heavy-tailed stochastic optimization from smooth/convex settings to non-smooth, weakly convex distributed problems, and it avoids the bounded-variance assumption that much of the distributed stochastic optimization literature relies on. The paper also provides explicit parameter conditions and a direct comparison with centralized or server-based clipping algorithms. The Moreau-envelope potential-function argument is appropriate for the problem class, and the experimental section gives evidence of heavy-tailed noise in the phase-retrieval application. However, the central rate statement in Theorem 2 is not proved as written, and one technical lemma is stated inconsistently; these issues affect the main claims and require repair.

major comments (3)
  1. [Appendix F, Theorem 2, and Remark 4] The proof of Theorem 2 sums inequality (F.1) from k0 to infinity and then divides by the infinite sum of the stepsizes. This step is not justified as written. For a finite horizon T, the key numerator term is sum_{k=k0}^T (V_k - E[V_{k+1}|F_k]), which equals V_{k0} - V_{T+1} plus a martingale-difference sum; only after taking expectations does the martingale term vanish, and then the numerator is E[V_{k0}] - E[V_{T+1}], not the random quantity ||V_{k0}-theta|| displayed in Theorem 2. Dividing by an infinite denominator would also formally send the displayed upper bound to zero under condition (15), rather than producing the rate stated in Remark 4. The finite-T inequality should be written first, then expectations taken, then monotone convergence applied to pass from inf_{k0<=k<=T} to inf_{k>=k0}; Remark 4's O(M/log T) claim should be stated as a finite-T bound with explicit T dependence. The theorem statement should also define theta, since Theorem 1 as stated does not introduce it.
  2. [Section 4.2, Lemma 4] Lemma 4 is dimensionally inconsistent: the left-hand side is a sum of norms, sum_i ||x_{i,k+1} - \hat v_{i,k}||, while the first term on the right-hand side is a sum of squared norms, sum_i ||v_{i,k} - \hat v_{i,k}||^2, and the final term N tau_k^2 alpha_k^2 also has squared units. The proof in Appendix C derives an inequality with squared norms on the left. The lemma statement should be corrected to the squared-norm form actually proved, and the subsequent use of the lemma in Appendix E should be checked against the corrected statement.
  3. [Theorem 2 and Appendix E] The quantity b_k is introduced in Appendix E as a random bound involving the consensus errors, and later replaced by a deterministic O(...) bound. In Theorem 2, however, sum b_k appears as a term in the bound without a clear definition or an explicit replacement by the deterministic bound. The theorem is only well-defined if b_k is a deterministic sequence; otherwise the right-hand side of Theorem 2 contains unintegrated random variables. Please state the definition of b_k in the theorem, or make explicit that the O(...) bound from Appendix E is substituted and spell out the resulting constants.
minor comments (5)
  1. [Section 2.2, Eq. (1)] The definition of the subgradient in (1), ∂f(x) = ∂h(x) - ρ||x||, is not dimensionally consistent: the term should be -ρx, not -ρ||x||. The inequality in (2) uses the inner-product form, so the intended definition is clear, but the displayed formula should be corrected.
  2. [Section 2.3, Eq. (4)] The condition 1 < μ < 1/ρ in the definition of the Moreau envelope should likely be 0 < μ < 1/ρ, since Theorem 1 later uses 0 < μ < 1/(2(ρ+1)). The printed interval also excludes all cases with ρ >= 1, which would be inconsistent with the rest of the paper.
  3. [Assumptions 8 and Theorem 1] The symbol α is used both for the moment exponent in Assumption 8 and, as α_k, for the stepsize sequence. This creates confusion in condition (15), where α_k τ_k^{2-2α} mixes the two uses. A different symbol for the moment exponent would improve readability.
  4. [Section 2.3, Lemma 2] The statement 'there exists μ < 1/ρ satisfies ...' should be 'for all μ < 1/ρ' or 'whenever μ < 1/ρ', since the proof and applications use the inequality for arbitrary admissible μ, not merely for one specially chosen value.
  5. [Section 5, numerical experiments] The baseline algorithms 'DPSM' and 'stoDPSM' are not defined in the text; a brief description or reference is needed. In addition, the captions of Fig. 2 and Fig. 3 should indicate what is plotted on the axes, especially the meaning of 'linear rate' in Fig. 3.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: Theorems 1 and 2 follow from explicit assumptions and independently proved lemmas; the sole self-citation [33] is used only as a contrast, not as a load-bearing premise.

full rationale

The derivation chain is self-contained rather than circular. Lemma 3 bounds the clipping bias directly from Assumption 8 using Hölder and Markov inequalities; Lemma 4 is proved from weak convexity, Lipschitz continuity, projection nonexpansiveness, and the proximal mapping property; Lemma 5 is a consensus estimate adapted from the external reference [48]. Theorem 1 is a supermartingale convergence argument, and Theorem 2 converts the Moreau-envelope stationarity measure via the identity ∇φ_μ(¯x_k) = (1/μ)(¯x_k − z_k), which is equation (7), not an input assumption. No fitted parameter is relabeled as a prediction: the constants α, γ, L, ρ, C0 are assumption constants, and M in Remark 4 is an abstract upper bound for convergent series, not a value calibrated to data. The only author self-citation is [33], cited in the Introduction as a contrast ('In contrast to [33, 34]...') and in Algorithm Design as motivation ('Motivated by [22], [33]'); it is not invoked as proof support, so it does not make the central claim circular. The Appendix F step that sums (F.1) to infinity and divides by the infinite stepsize sum has a technical gap concerning martingale-difference terms and the infinite denominator; this is a correctness/handling issue, not a circular reduction of the theorem to its assumptions.

Assumptions & free parameters 0 free parameters · 7 assumptions · 0 invented entities

The central claim rests on Assumptions 1-8, which are stated as given, and on two lemmas imported from [32]. No free parameters are fit to data: gamma, rho, L, C0 are assumption constants or derived bounds, and M in Theorem 2 is an upper bound on convergent sums, not a fitted value. No new entities (particles, forces, dimensions) are introduced. This is a theory paper; the numerical experiment is illustrative and does not contribute to the proof.

assumptions (7)
  • standard math Lemma 1 in [32]: interpolation inequality for rho-weakly convex functions (Eq. 3)
    Imported from Chen et al. [32] and used in Appendix E to bound phi_mu(v_i,k) in terms of phi_mu(x_j,k).
  • standard math Lemma 2 in [32]: prox map of a rho-weakly convex function is 1/(1-mu rho)-Lipschitz
    Used in Appendix C and E; assumed without reproof.
  • domain assumption Assumptions 1-3: strongly connected graph, doubly stochastic adjacency matrix, bounded intercommunication interval
    Network model; Assumption 1 is used to keep v_i,k in Omega and to obtain consensus bounds in Lemma 5.
  • domain assumption Assumption 4: Omega is compact, convex, nonempty
    Used to bound subgradients by C0 and to ensure prox is well-defined and projections are nonexpansive.
  • domain assumption Assumptions 5-6: each f_i is rho-weakly convex and L-Lipschitz
    Central problem class; used in Lemma 4 and to bound subgradients.
  • domain assumption Assumption 7: unbiased stochastic subgradient estimator
    Ensures G_i,k is a true subgradient on average; used throughout the analysis.
  • domain assumption Assumption 8: E[||g - G||^alpha] <= gamma^alpha for some alpha in (1,2]
    Heavy-tailed noise model; load-bearing for Lemma 3's clipping bias bound and for condition (15).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distributed Stochastic Optimization for Non-Smooth and Weakly Convex Problems under Heavy-Tailed Noise." pith.science (2026). https://pith.science/paper/X5MN6JMY

@misc{pith2026250509279,
  author       = {Pith},
  title        = {Pith review of: Distributed Stochastic Optimization for Non-Smooth and Weakly Convex Problems under Heavy-Tailed Noise},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X5MN6JMY}},
  note         = {Machine review of arXiv:2505.09279}
}
read the original abstract

In existing distributed stochastic optimization studies, it is usually assumed that the gradient noise has a bounded variance. However, recent research shows that the heavy-tailed noise, which allows an unbounded variance, is closer to practical scenarios in many tasks. Under heavy-tailed noise, traditional optimization methods, such as stochastic gradient descent, may have poor performance and even diverge. Thus, it is of great importance to study distributed stochastic optimization algorithms applicable to the heavy-tailed noise scenario. However, most of the existing distributed algorithms under heavy-tailed noise are developed for convex and smooth problems, which limits their applications. This paper proposes a clipping-based distributed stochastic algorithm under heavy-tailed noise that is suitable for non-smooth and weakly convex problems. The convergence of the proposed algorithm is proven, and the conditions on the parameters are given. A numerical experiment is conducted to demonstrate the effectiveness of the proposed algorithm.

Figures

Figures reproduced from arXiv: 2505.09279 by the authors.

Figure 3
Figure 3. (a) Linear rate of stoDPSM, DPSM and the proposed [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 1
Figure 1. (a) Histogram of gradient noise samples for Phase-re [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Phase retrieval results on a digit image from the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 39 canonical work pages

  1. [1]

    A survey on distributed machine learning

    Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan S Rellermeyer. A survey on distributed machine learning. Acm computing surveys (csur), 53(2):1–33, 2020

    2020

  2. [2]

    Modern robotics: Mechanics, planning, and control

    Andreas Mueller. Modern robotics: Mechanics, planning, and control. IEEE Control Systems Magazine , 39(6):100–102, 2019

    work page 2019

  3. [3]

    Delay effects on consensus- based distributed economic dispatch algorithm in microgrid

    Gang Chen and Zhongyuan Zhao. Delay effects on consensus- based distributed economic dispatch algorithm in microgrid. IEEE Transactions on Power Systems , 33(1):602–612, 2017

    work page 2017

  4. [4]

    A survey of distributed optimization

    Tao Yang, Xinlei Yi, Junfeng Wu, Ye Yuan, Di Wu, Ziyang Meng, Yiguang Hong, Hong Wang, Zongli Lin, and Karl H Johansson. A survey of distributed optimization. Annual Reviews in Control, 47:278–305, 2019

    work page 2019

  5. [5]

    Machine learning at the wireless edge: Distributed stochastic gradient descent over-the-air

    Mohammad Mohammadi Amiri and Deniz G¨ und¨ uz. Machine learning at the wireless edge: Distributed stochastic gradient descent over-the-air. IEEE Transactions on Signal Processing, 68:2155–2169, 2020

    work page 2020

  6. [6]

    Convergence analysis of distributed stochastic gradient descent with shuffling

    Qi Meng, Wei Chen, Yue Wang, Zhi-Ming Ma, and Tie-Yan Liu. Convergence analysis of distributed stochastic gradient descent with shuffling. Neurocomputing, 337:46–57, 2019

    work page 2019

  7. [7]

    Distributed stochastic optimization and learning

    Ohad Shamir and Nathan Srebro. Distributed stochastic optimization and learning. In 2014 52nd Annual Allerton Conference on Communication, Control, and Computing (Allerton), 850–857. IEEE, 2014

    work page 2014

  8. [8]

    Distributed stochastic gradient descent with event-triggered communication

    Jemin George and Prudhvi Gurram. Distributed stochastic gradient descent with event-triggered communication. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 7169–7178, 2020

    work page 2020

Show all 48 references
  1. [9]

    Distributed stochastic gradient descent: Nonconvexity, nonsmoothness, and convergence to local minima

    Brian Swenson, Ryan Murray, H Vincent Poor, and Soummya Kar. Distributed stochastic gradient descent: Nonconvexity, nonsmoothness, and convergence to local minima. Journal of Machine Learning Research, 23(328):1–62, 2022

    2022

  2. [10]

    A sharp estimate on the transient time of distributed stochastic 10 gradient descent

    Shi Pu, Alex Olshevsky, and Ioannis Ch Paschalidis. A sharp estimate on the transient time of distributed stochastic 10 gradient descent. IEEE Transactions on Automatic Control, 67(11):5900–5915, 2021

    2021

  3. [11]

    Distributed learning in wireless networks: Recent progress and future challenges

    Mingzhe Chen, Deniz G¨ und¨ uz, Kaibin Huang, Walid Saad, Mehdi Bennis, Aneta Vulgarakis Feljan, and H Vincent Poor. Distributed learning in wireless networks: Recent progress and future challenges. IEEE Journal on Selected Areas in Communications, 39(12):3579–3605, 2021

    2021

  4. [12]

    On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization

    Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. In International Conference on Machine Learning, 7184–7193. PMLR, 2019

    2019

  5. [13]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  6. [14]

    A primal-dual sgd algorithm for distributed nonconvex optimization

    Xinlei Yi, Shengjun Zhang, Tao Yang, Tianyou Chai, and Karl Henrik Johansson. A primal-dual sgd algorithm for distributed nonconvex optimization. IEEE/CAA Journal of Automatica Sinica, 9(5):812–833, 2022

    2022

  7. [15]

    Stochastic gradient push for distributed deep learning

    Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Mike Rabbat. Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning, 344–353. PMLR, 2019

    2019

  8. [16]

    Distributed stochastic subgradient projection algorithms for convex optimization

    S Sundhar Ram, Angelia Nedi´ c, and Venugopal V Veeravalli. Distributed stochastic subgradient projection algorithms for convex optimization. Journal of optimization theory and applications, 147:516–545, 2010

    2010

  9. [17]

    The heavy-tail phenomenon in sgd

    Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning, 3964–3975. PMLR, 2021

    2021

  10. [18]

    High- probability convergence bounds for nonlinear stochastic gradient descent under heavy-tailed noise

    Aleksandar Armacki, Pranay Sharma, Gauri Joshi, Dragana Bajovic, Dusan Jakovetic, and Soummya Kar. High- probability convergence bounds for nonlinear stochastic gradient descent under heavy-tailed noise. arXiv preprint arXiv:2310.18784, 2023

    arXiv

  11. [19]

    On proximal policy optimization’s heavy-tailed gradients

    Saurabh Garg, Joshua Zhanson, Emilio Parisotto, Prasad Adarsh, Kolter Zico, Lipton Zachary, Balakrishnan Sivaraman, Salakhutdinov Ruslan, and Ravikumar Pradeep. On proximal policy optimization’s heavy-tailed gradients. In International Conference on Machine Learning , 3610–361...

    2021

  12. [20]

    Revisiting the noise model of stochastic gradient descent

    Barak Battash, Lior Wolf, and Ofir Lindenbaum. Revisiting the noise model of stochastic gradient descent. In International Conference on Artificial Intelligence and Statistics, 4780–4788. PMLR, 2024

    2024

  13. [21]

    Heavy-tail phenomenon in decentralized sgd

    Mert G¨ urb¨ uzbalaban, Yuanhan Hu, Umut S ¸im¸ sekli, Kun Yuan, and Lingjiong Zhu. Heavy-tail phenomenon in decentralized sgd. IISE Transactions, 1–15, 2024

    2024

  14. [22]

    Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33:15383–15393, 2020

    Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33:15383–15393, 2020

    2020

  15. [23]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021

    2021 arXiv

  16. [24]

    High probability guarantees for nonconvex stochastic gradient descent with heavy tails

    Shaojie Li and Yong Liu. High probability guarantees for nonconvex stochastic gradient descent with heavy tails. In International Conference on Machine Learning , 12931– 12963. PMLR, 2022

    2022

  17. [25]

    heavier-tailed

    Haibo Yang, Peiwen Qiu, and Jia Liu. Taming fat-tailed (“heavier-tailed” with potentially infinite variance) noise in federated learning. Advances in Neural Information Processing Systems, 35:17017–17029, 2022

    2022

  18. [26]

    A communication-efficient distributed gradient clipping algorithm for training deep neural networks

    Mingrui Liu, Zhenxun Zhuang, Yunwen Lei, and Chunyang Liao. A communication-efficient distributed gradient clipping algorithm for training deep neural networks. Advances in Neural Information Processing Systems , 35:26204–26217, 2022

    2022

  19. [27]

    Stochastic model- based minimization of weakly convex functions

    Damek Davis and Dmitriy Drusvyatskiy. Stochastic model- based minimization of weakly convex functions. SIAM Journal on Optimization , 29(1):207–239, 2019

    2019

  20. [28]

    Solving (most) of a set of quadratic equalities: Composite optimization for robust phase retrieval

    John C Duchi and Feng Ruan. Solving (most) of a set of quadratic equalities: Composite optimization for robust phase retrieval. Information and Inference: A Journal of the IMA, 8(3):471–529, 2019

    2019

  21. [29]

    First-order convergence theory for weakly-convex-weakly- concave min-max problems

    Mingrui Liu, Hassan Rafique, Qihang Lin, and Tianbao Yang. First-order convergence theory for weakly-convex-weakly- concave min-max problems. Journal of Machine Learning Research, 22(169):1–34, 2021

    2021

  22. [30]

    Weakly convex regularisers for inverse problems: Convergence of critical points and primal- dual optimisation

    Zakhar Shumaylov, Jeremy Budd, Subhadip Mukherjee, and Carola-Bibiane Sch¨ onlieb. Weakly convex regularisers for inverse problems: Convergence of critical points and primal- dual optimisation. arXiv preprint arXiv:2402.01052 , 2024

    2024 arXiv

  23. [31]

    Delayed algorithms for distributed stochastic weakly convex optimization

    Wenzhi Gao and Qi Deng. Delayed algorithms for distributed stochastic weakly convex optimization. Advances in Neural Information Processing Systems, 36:22754–22793, 2023

    2023

  24. [32]

    On distributed nonconvex optimization: Projected subgradient method for weakly convex problems in networks

    Shixiang Chen, Alfredo Garcia, and Shahrampour Shahin. On distributed nonconvex optimization: Projected subgradient method for weakly convex problems in networks. IEEE Transactions on Automatic Control , 67(2):662–675, 2021

    2021

  25. [33]

    Distributed stochastic optimization under heavy- tailed noises

    Chao Sun. Distributed stochastic optimization under heavy- tailed noises. arXiv preprint arXiv:2312.15847 , 2023

    2023 arXiv

  26. [34]

    Online distributed optimization with clipped stochastic gradients: High probability bound of regrets

    Yuchen Yang, Kaihong Lu, and Long Wang. Online distributed optimization with clipped stochastic gradients: High probability bound of regrets. arXiv preprint arXiv:2401.14776, 2024

    2024 arXiv

  27. [35]

    High-probability convergence for composite and distributed stochastic minimization and variational inequalities with heavy-tailed noise

    Eduard Gorbunov, Abdurakhmon Sadiev, Marina Danilova, Samuel Horv´ ath, Gauthier Gidel, Pavel Dvurechensky, Alexander Gasnikov, and Peter Richt´ arik. High-probability convergence for composite and distributed stochastic minimization and variational inequalities with heavy-tai...

    arXiv

  28. [36]

    Convex analysis

    R Tyrrell Rockafellar. Convex analysis. SIAM Review , 13(2):233–238, 1971

    1971

  29. [37]

    Efficiency of minimizing compositions of convex functions and smooth maps

    Dmitriy Drusvyatskiy and Courtney Paquette. Efficiency of minimizing compositions of convex functions and smooth maps. Mathematical Programming, 178:503–558, 2019

    2019

  30. [38]

    Distributed stochastic optimization with gradient tracking over strongly-connected networks

    Ran Xin, Anit Kumar Sahu, Usman A Khan, and Soummya Kar. Distributed stochastic optimization with gradient tracking over strongly-connected networks. In 2019 IEEE 58th Conference on Decision and Control (CDC), 8353–8358. IEEE, 2019

    2019

  31. [39]

    Distributed subgradient methods for multi-agent optimization

    Angelia Nedic and Asuman Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control, 54(1):48–61, 2009

    2009

  32. [40]

    Distributed smooth convex optimization with coupled constraints

    Shu Liang, George Yin, et al. Distributed smooth convex optimization with coupled constraints. IEEE Transactions on Automatic Control, 65(1):347–353, 2019

    2019

  33. [41]

    Variational analysis, volume 317

    R Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science & Business Media, 2009

    2009

  34. [42]

    The nonsmooth landscape of phase retrieval

    Damek Davis, Dmitriy Drusvyatskiy, and Courtney Paquette. The nonsmooth landscape of phase retrieval. IMA Journal of Numerical Analysis , 40(4):2652–2695, 2020. 11

    2020

  35. [43]

    Gradient descent with random initialization: Fast global convergence for nonconvex phase retrieval

    Yuxin Chen, Yuejie Chi, Jianqing Fan, and Cong Ma. Gradient descent with random initialization: Fast global convergence for nonconvex phase retrieval. Mathematical Programming, 176:5–37, 2019

    2019

  36. [44]

    Gradient-based learning applied to document recognition

    Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278–2324, 1998

    1998

  37. [45]

    Heavy- tailed distributions in combinatorial search

    Carla P Gomes, Bart Selman, and Nuno Crato. Heavy- tailed distributions in combinatorial search. In International Conference on Principles and Practice of Constraint Programming, 121–135. Springer, 1997

    1997

  38. [46]

    Proximal algorithms

    Stephen Boyd and Neal Parikh. Proximal algorithms. Foundations and Trends in optimization, 1(3):123–231, 2013

    2013

  39. [47]

    Convergence rate analysis of distributed optimization with projected subgradient algorithm

    Shuai Liu, Zhirong Qiu, and Lihua Xie. Convergence rate analysis of distributed optimization with projected subgradient algorithm. Automatica, 83:162–169, 2017

    2017

  40. [48]

    Constrained consensus and optimization in multi-agent networks

    Angelia Nedic, Asuman Ozdaglar, and Pablo A Parrilo. Constrained consensus and optimization in multi-agent networks. IEEE Transactions on Automatic Control , 55(4):922–938, 2010. 12

    2010

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.