Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

A Fast Newton Method Under Local Lipschitz Smoothness

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Newton-type method for nonconvex optimization finds an $\epsilon$-stationary point in $O(|\log\epsilon|\,\epsilon^{-3/2})$ iterations under only a local Hessian Lipschitz condition, and a second-order variant reaches near-optimal rates…

desk verdict Careful and honest extension of adaptive regularized Newton to local Hessian smoothness, with an explicit global negative-curvature assumption carrying the logarithmic factor. read the letter →

arxiv 2505.04807 v2 pith:RSVQKA46 submitted 2025-05-07 math.OC

Serge Gratton , Sadok Jerad , Philippe L.Toint This is my paper · ORCID
classification math.OC MSC 90C3090C5390C6065K05
keywords second-orderoptimizationNewtonmethodlocalLipschitzsmoothnessnegativecurvatureworst-casecomplexityadaptiveregularizationKrylovsubspacesnonconvex
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes AN2CLS, an adaptive Newton method with negative-curvature steps, and proves that it produces a point with gradient norm at most $\epsilon$ in $O(|\log\epsilon|\,\epsilon^{-3/2})$ successful iterations. The novelty is that the proof does not assume a globally Lipschitz Hessian: a local condition, $\|\nabla^2 f(y)-\nabla^2 f(x)\|\le(L_0+L_1\|\nabla f(x)\|)\|x-y\|$ for $\|x-y\|\le\delta$, suffices. That matters because global Lipschitz smoothness fails for simple functions such as polynomials and exponentials, while the local condition covers them. The paper also gives an inexact Krylov-subspace implementation retaining the same bound, and an extended algorithm, SOAN2CLS, that finds approximate second-order critical points with complexity within a log factor of the optimal rate. Numerical experiments on standard test problems are reported as initial evidence that the method is competitive.

What carries the argument

The load-bearing mechanism is the doubly regularized quadratic model $m_k(s)=g_k^\top s+\frac12 s^\top(H_k+(\sqrt{\sigma_k}\|g_k\|+\mu_k)I_n)s$, where $g_k$ and $H_k$ are the gradient and Hessian at the current iterate, $\sigma_k$ is adaptively updated, and $\mu_k$ carries eigenvalue information. Step computation is regulated by Assumption 0: for moderate $\mu_k$ the step approximately minimizes this model with residual bounds and orthogonality (1.15)–(1.16); for large $\mu_k$ the algorithm takes a negative-curvature step. The two acceptance tests (1.8) and (1.12) are what make the local smoothness condition sufficient, because they prevent acceptance of small steps that cause the gradient to jump and they control the gradient ratio after each step. Lemma 1.2 provides Taylor bounds with coefficients $L_0+L_1\|\nabla f(x)\|$, and Lemma 2.4 uses the negative-curvature bound AS.4 to keep $\|g_{k+1}\|/\|g_k\|$ under control; together they yield the $|\log\epsilon|$ factor in the final complexity.

What would settle it

Run the exact AN2CLS variant on a family of univariate polynomials of high degree, starting near a stationary point, over a range of $\epsilon$, and count successful iterations: if the count grows faster than $\epsilon^{-3/2}\log(1/\epsilon)$, Theorem 2.5 is false. A sharper test targets Lemma 2.4 directly: track $\|g_{k+1}\|/\|g_k\|$ after each negative-curvature step and check whether it exceeds the paper's bound $\kappa_{\rm upncurv}/\epsilon$; the lemma forbids such a spike under AS.4.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that near-optimal second-order complexity for nonconvex optimization does not require global Hessian Lipschitz continuity. Theorem 2.5 states that, under Assumptions AS.1–AS.4, AN2CLS terminates with $\|\nabla f(x)\|\le\epsilon$ after at most $O(|\log\epsilon|\,\epsilon^{-3/2})$ successful iterations, and the same order of total iterations, using only the local smoothness condition and a bound on negative curvature. The algorithm alternates between a Newton step computed from a doubly regularized quadratic model and a negative-curvature step, with adaptive regularization governed by the current gradient norm; no prior knowledge of the Lipschitz constants is required. The inexact variant based on nested Krylov subspaces satisfies the same step-quality conditions, and the second-order variant SOAN2CLS terminates at a point with $\|\nabla f(x)\|\le\epsilon_1$ and $\lambda_{\min}(\nabla^2 f(x))\ge-\epsilon_2$ in $O(\epsilon_1^{-3/2}+\epsilon_2^{-3})$ iterations up to log factors. The rate matches the complexity of methods that assume a globally Lipschitz Hessian, up to the $|\log\epsilon|$ factor, while applying to a strictly broader class of functions.

Load-bearing premise

The load-bearing premise is that the Hessian's negative curvature never becomes unboundedly large on the set of points the algorithm can visit: a finite $\kappa_B$ caps how negative the smallest eigenvalue can get, and this cap is what prevents the gradient from exploding after a negative-curvature step.

Editorial extensions

If this is right

  • The same near-optimal rate previously known for globally Lipschitz Hessians now holds for functions whose Hessians satisfy only the local condition, including univariate polynomials and exponentials.
  • The inexact Krylov variant keeps the $O(|\log\epsilon|\epsilon^{-3/2})$ guarantee while computing steps with matrix-vector products, so the bound is realistic for large-scale problems where exact solves are unavailable.
  • Without prior knowledge of $L_0$, $L_1$, $\delta$, or the criticality threshold, the algorithm adapts its regularization internally and still terminates with the stated complexity.
  • The SOAN2CLS extension achieves approximate second-order criticality in $O(\epsilon_1^{-3/2}+\epsilon_2^{-3})$ iterations (up to log factors) under the same local smoothness assumption, matching the order of optimal global-smoothness methods.
  • On the standard test problems used in the paper, the new method's efficiency and reliability profiles are at least as competitive as those of trust-region, cubic-regularization, and earlier Newton variants in the reported experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the step-computation conditions in Assumption 0 are already inexact, the same $O(|\log\epsilon|\epsilon^{-3/2})$ bound should transfer to stochastic or matrix-free settings where gradients and Hessians are approximated, provided the approximation errors respect the residual tolerances.
  • Beyond the paper: the proof isolates the ratio $\|g_{k+1}\|/\|g_k\|$ after a negative-curvature step as the quantity that produces the $|\log\epsilon|$ factor, so a sharper potential-function argument might reduce that factor to $|\log\log(1/\epsilon)|$, as in the globally smooth case; the paper does not attempt this.
  • Beyond the paper: the reported experiments use negative-curvature steps in less than one percent of iterations, which raises the testable question of whether a simplified regularized-Newton variant with only the two acceptance tests would achieve the same practical rate without explicit negative-curvature steps.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes AN2CLS, an adaptive second-order method for unconstrained nonconvex optimization that alternates between regularized Newton steps and negative-curvature steps. Its main contribution is a worst-case complexity bound of O(|log ε| ε^{-3/2}) iterations for finding a first-order ε-stationary point under a local Hessian Lipschitz condition (AS.3) rather than the usual global Lipschitz Hessian assumption. The step computation is abstracted through Assumption 0, and two implementations are given: one using exact linear algebra (AN2CLSE) and one based on nested Krylov subspaces (AN2CLSK). A second-order variant, SOAN2CLS, is also analyzed and claimed to have complexity O(ε1^{-3/2} + ε2^{-3}) for finding (ε1,ε2)-second-order critical points. Numerical experiments on CUTEst problems compare the proposed methods with AN2C, AR2, and TR2 variants.

Significance. If the main theorem stands, the paper makes a valuable contribution: it shows that the fast second-order complexity O(ε^{-3/2}) up to a logarithmic factor is achievable without global Hessian Lipschitz continuity, replacing it by a local smoothness condition that covers polynomials and exponentials. The abstract Assumption 0 formulation cleanly separates the convergence analysis from the linear-algebra implementation, and the explicit attention to inexact Krylov solves is practically relevant. The numerical study is honest in scope and shows the new method is competitive. However, the significance is tempered by two correctness gaps: the proof of the second-order variant (Theorem 3.1) appears to rely on an invalid implication, and the Krylov implementation's validity is not established for the full stated parameter range.

major comments (3)
  1. [Appendix B, Lemma B.3] The bound ∥g_{p_i}∥ ≤ κ_gpi is derived from condition (3.4), but (3.4) is a conjunction: REJECT is set only if ρ_k < η1 AND ∥∇f(x_k+s_k)∥ > κ_{k,hess}. An accepted second-order step with ρ_k ≥ η1 is not subject to the gradient bound, so the proof does not establish the claimed bound on the gradient at the start of the next first-order phase. This is load-bearing for Theorem 3.1; either the acceptance condition must be changed to reject on the disjunction (ρ_k < η1 OR gradient exceeds the threshold), or a separate argument must bound ∥g_{p_i}∥ for accepted second-order steps with ρ_k ≥ η1.
  2. [Section 4.2, StepcompK Step 4 and Lemma 4.2] The assertion that 'since θ≤1, u_p can be chosen as the negative curvature of T_p' is false. Condition (4.10) requires u_p^T T_p^2 u_p ≤ λ_min(T_p)^2/(2θ^2), and for a unit vector satisfying u_p^T T_p u_p ≤ θ λ_min(T_p) with λ_min(T_p)<0 one has u_p^T T_p^2 u_p ≥ (u_p^T T_p u_p)^2 ≥ θ^2 λ_min(T_p)^2. For θ>2^{-1/2} these two inequalities can be incompatible; for example, with T_p = diag(-1,-0.9) and θ=0.8, no unit vector satisfies both. Lemma 4.2 therefore proves Assumption 0 for StepcompK only for a restricted range of θ (the numerical choice θ=1/2 is safe), not for all θ∈(0,1] as stated. The condition in Step 4 should be revised or θ restricted.
  3. [Lemma 2.1, Eq. (2.1)] The displayed lower bound in (2.1) is algebraically incorrect. Solving the quadratic inequality in the proof gives ∥s_trial∥ ≥ (√(a^2+Y)-a)/(√σ Y) with Y=(L0/∥g∥+L1)/σ, not the expression in (2.1). As printed, (2.1) is false; for instance, take a=1 and Y=4, where the correct bound gives a smaller value than the printed one by a factor of 4. The subsequent sufficient condition (2.2) is nevertheless correct, so the main complexity proof can be repaired by correcting (2.1), but the lemma as stated is not a valid theorem.
minor comments (5)
  1. [Abstract] The phrase 'optimal O(|log ε| ε^{-3/2})' is inaccurate: since the known lower bound is Ω(ε^{-3/2}), the stated bound is within a logarithmic factor of optimal, as the authors correctly note later in the paper. The abstract should be reworded to avoid claiming optimality.
  2. [Section 1, Eq. (1.4)] Equation (1.4) has a stray superscript '⊺' after 's⊺∇²f(x)s'; the intended expression is f(x+s)-f(x)-s^T∇f(x)-½s^T∇²f(x)s.
  3. [Lemma 2.2, proof] In the negative-curvature case, the passage from terms like κ_C μ_k/√σ to (κ_C μ_k/√σ)∥g∥/ε uses both ∥g∥≥ε and ε≤1; adding an explicit sentence would improve readability.
  4. [References] Reference [29] is missing author names and formatting, and reference [39] has inconsistent punctuation in its title. These should be corrected.
  5. [Section 5.1] The baseline 'AN2C' is described as a modified variant that suppresses conditions (1.12) and (1.8) and changes the negative-curvature formula; the comparison should state more clearly that this is not exactly the published AN2C algorithm but an adapted implementation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the complexity results are proved from stated assumptions with self-contained lemmas; self-citations are contextual only.

full rationale

The paper's central results are mathematical theorems, not empirical predictions: Theorem 2.5 and Theorem 3.1 are derived from the explicitly stated assumptions AS.1–AS.4 and from Assumption 0, which is a contract on the step-computation subroutine rather than a restatement of the conclusions. Every load-bearing step in the proof is carried out in the paper: Lemma 2.3 bounds σ_k via Taylor error estimates (1.4)–(1.5) and the acceptance-ratio mechanism; Lemma 2.4 bounds gradient-growth using AS.4 and the definitions of κ_upnewt and κ_upncurv; the final bound combines these with function-decrease arguments. The negative-curvature bound AS.4 is global but is an explicit hypothesis, not an unstated one, and its use in Lemma 2.4 (via u_kᵀH_k u_k ≤ −θμ_k and λ_min(H_k) ≤ u_kᵀH_k u_k) is valid. Self-citations to [18], [16], [17], [19], [20], and [27] provide inspiration, comparison baselines, and software context, but none is load-bearing: even the lemma said to be 'in the spirit of [18, Lemma 3.4]' is proved in full. The numerical section tunes hyperparameters for experiments only and does not use the complexity bound to fit anything; the theoretical rate holds for all allowed parameter values. No fitted input is renamed as a prediction, no theorem is imported by self-citation as a substitute for proof, and the local-smoothness framework is not a renaming of the global-Lipschitz result because the proof genuinely replaces global Lipschitz estimates with the local ones in AS.3. The derivation chain is therefore self-contained and non-circular.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central complexity theorem depends on the four domain assumptions AS.1-AS.4 and on the algorithmic contract Assumption 0. No free parameters are fitted to derive the theory; the only tuned values are numerical hyperparameters for the Krylov variant, which do not affect the complexity result. No new physical or mathematical entities are introduced.

free parameters (1)
  • Krylov hyperparameters κθ, θ = κθ=1, θ=0.5
    Selected by search on a subset of small CUTEst problems (Section 5.2); affects empirical performance only, not the complexity theorem.
assumptions (5)
  • domain assumption AS.1: f is twice continuously differentiable on R^n.
    Standard smoothness requirement for algorithms using Hessian information; invoked throughout Section 1.
  • domain assumption AS.2: f is bounded below by f_low.
    Needed for the function-decrease arguments in Theorem 2.5 and Theorem 3.1 to bound the number of successful iterations.
  • domain assumption AS.3: local Lipschitz Hessian condition (1.2): ||∇²f(y)-∇²f(x)|| ≤ (L0 + L1||∇f(x)||)||x-y|| whenever ||x-y|| ≤ δ.
    This is the central relaxed smoothness condition replacing global Lipschitz continuity of the Hessian; it drives all Taylor error bounds in Lemmas 1.2, 2.1, and 2.2.
  • domain assumption AS.4: there exists κ_B ≥ 0 such that max(0,-λ_min(∇²f(x))) ≤ κ_B for all x in the sublevel set {y | f(y) ≤ f(x0)}.
    Used in Lemma 2.4 and Lemma B.3 to bound the gradient growth factor after negative curvature steps; without AS.4 the |log ε| term in the complexity bound is uncontrolled.
  • domain assumption Assumption 0: the Stepcomp procedure must return a trial step satisfying (1.14)-(1.19) in the Newton case and (1.18)-(1.19) in the negative curvature case.
    This is an algorithmic contract, not a mathematical assumption on f. The exact and Krylov implementations in Section 4 are proved to satisfy it, but Theorem 2.5 is conditional on its validity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Fast Newton Method Under Local Lipschitz Smoothness." pith.science (2026). https://pith.science/paper/RSVQKA46

@misc{pith2026250504807,
  author       = {Pith},
  title        = {Pith review of: A Fast Newton Method Under Local Lipschitz Smoothness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RSVQKA46}},
  note         = {Machine review of arXiv:2505.04807}
}
abstract

A new, fast second-order method is proposed that achieves the optimal $\mathcal{O}\left(|\log(\epsilon)|\epsilon^{-3/2}\right)$ complexity to obtain first-order $\epsilon$-stationary points. Crucially, this is deduced without assuming the standard global Lipschitz Hessian continuity condition, but only using an appropriate local smoothness requirement. The algorithm exploits Hessian information to compute a Newton step and a negative curvature step when needed, in an approach similar to that of the AN2C method.Inexact versions of the Newton step and negative curvature are proposed in order to reduce the cost of evaluating second-order information. Details are given of such an iterative implementation using Krylov subspaces. An extended algorithm for finding second-order critical points is also developed and its complexity is again shown to be within a log factor of the optimal one. Initial numerical experiments are discussed for both factorised and Krylov variants, which demonstrate the competitiveness of the proposed algorithm.

Figures

Figures reproduced from arXiv: 2505.04807 by the authors.

Figure 1
Figure 1. Performance profile of both AN2CLSE and AN2CE on three different set of problems (small, medium, large) iterations (or, equivalently, function and possibly derivatives’ evaluations): the fewer the more efficient the algorithm. We also add an additional global metric πalgo following [27] which denotes 1/10 of the curve corresponding to algo in the performance profile, for abscissas in the interval [1, 10]. We also ad… view at source ↗
Figure 2
Figure 2. Performance profile of both AN2CLSK and AN2CK on three different sets of problems (small, medium, large) [PITH_FULL_IMAGE:figures/full_fig_p025_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Gradient-Normalized Smoothness for Optimization with Approximate Hessians

    math.OC 2025-06 conditional novelty 7.0 of 10

    Gradient-Normalized Smoothness lets a gradient-regularized Newton method with approximate Hessians achieve fast global convergence rates, recovering and extending known rates in convex and non-convex optimization.

Reference graph

Works this paper leans on

39 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [33]

    Trust region methods for nonconvex stochastic optimization beyond Lipschitz smoothness.Proceedings of the AAAI Conference on Artificial Intelligence, 38(14):16049–16057, Mar

    Chenghan Xie, Chenxi Li, Chuwen Zhang, Qi Deng, Dongdong Ge, and Yinyu Ye. Trust region methods for nonconvex stochastic optimization beyond Lipschitz smoothness.Proceedings of the AAAI Conference on Artificial Intelligence, 38(14):16049–16057, Mar. 2024

    work page 2024

  2. [18]

    Yet another fast variant of newton’s method for nonconvex optimization.IMA Journal of Numerical Analysis, 45(2):971–1008, 2025

    Serge Gratton, Sadok Jerad, and Philippe L Toint. Yet another fast variant of newton’s method for nonconvex optimization.IMA Journal of Numerical Analysis, 45(2):971–1008, 2025

    work page 2025

  3. [1]

    Solving the Trust-Region Subproblem By a Generalized Eigenvalue Problem.SIAM Journal on Optimization, 27:269–291, 2017

    Satoru Adachin, Satoru Iwata, Yuji Nakatsuka, and Akiko Takeda. Solving the Trust-Region Subproblem By a Generalized Eigenvalue Problem.SIAM Journal on Optimization, 27:269–291, 2017

    work page 2017

  4. [2]

    E. G. Birgin, J. L. Gardenghi, J. M. Mart´ ınez, S. A. Santos, and Ph. L. Toint. Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models.Mathematical Programming, 163(1-2):359–368, August 2016

    work page 2016

  5. [3]

    Duchi, Oliver Hinder, and Aaron Sidford

    Yair Carmon, John C. Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points i.Mathematical Programming, 184(1-2):71–120, 2019

    2019

  6. [4]

    Coralia Cartis, Nicholas I. M. Gould, and Philippe L. Toint. Adaptive cubic regularisation methods for unconstrained optimization. part II: worst-case function- and derivative-evaluation complexity.Mathe- matical Programming, 130(2):295–319, 2010

    work page 2010

  7. [5]

    Coralia Cartis, Nicholas I. M. Gould, and Philippe L. Toint. Sharp worst-case evaluation complexity bounds for arbitrary-order nonconvex optimization with inexpensive constraints.SIAM Journal on Op- timization, 30(1):513–541, January 2020

    2020

  8. [6]

    MOS-SIAM Series on Optimization

    Coralia Cartis, Nicholas I M Gould, and Philippe L Toint.Evaluation complexity of algorithms for non- convex optimization. MOS-SIAM Series on Optimization. Society for Industrial & Applied Mathematics, New York, NY, April 2022

    2022

Show all 39 references
  1. [7]

    Generalized-smooth nonconvex optimization is as efficient as smooth nonconvex optimization

    Ziyi Chen, Yi Zhou, Yingbin Liang, and Zhaosong Lu. Generalized-smooth nonconvex optimization is as efficient as smooth nonconvex optimization. InProceedings of the 40th International Conference on Machine Learning, volume 202, pages 5396–5427. PMLR, 2023

    2023

  2. [8]

    Coakley and Vladimir Rokhlin

    Ed S. Coakley and Vladimir Rokhlin. A fast divide-and-conquer algorithm for computing the spectra of real symmetric tridiagonal matrices.Applied and Computational Harmonic Analysis, 34(3):379–414, May 2013

    2013

  3. [9]

    Conn, Nicholas I

    Andrew R. Conn, Nicholas I. M. Gould, and Ph. L. Toint.Trust Region Methods. Society for Industrial and Applied Mathematics, January 2000

    2000

  4. [10]

    Minimizing quasi-self-concordant functions by gradient regularization of newton method

    Nikita Doikov. Minimizing quasi-self-concordant functions by gradient regularization of newton method. arxiv:2308.14742, 2023

    2023 arXiv

  5. [11]

    Super-universal regularized newton method

    Nikita Doikov, Konstantin Mishchenko, and Yurii Nesterov. Super-universal regularized newton method. SIAM Journal on Optimization, 34(1):27–56, January 2024

    2024

  6. [12]

    Gradient regularization of newton method with bregman distances

    Nikita Doikov and Yurii Nesterov. Gradient regularization of newton method with bregman distances. Mathematical Programming, March 2023

    2023

  7. [13]

    E. D. Dolan, J. J. Mor´ e, and T. S. Munson. Optimality measures for performance profiles.SIAM Journal on Optimization, 16(3):891–909, 2006. Gratton, Jerad, Toint: Fast Newton Method Under Local Smoothness27

    2006

  8. [14]

    Scalable adaptive cubic regularization meth- ods.Mathematical Programming, October 2023

    Jean-Pierre Dussault, Tangi Migot, and Dominique Orban. Scalable adaptive cubic regularization meth- ods.Mathematical Programming, October 2023

    2023

  9. [15]

    Beyond uniform smoothness: A stopped analysis of adaptive sgd

    Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Beyond uniform smoothness: A stopped analysis of adaptive sgd. InProceedings of Thirty Sixth Conference on Learning Theory, volume 195 ofProceedings of Machine Learning Research, pages 89–160, 2023

    2023

  10. [16]

    Serge Gratton, Sadok Jerad, and Philippe L. Toint. Convergence properties of an objective-function- free optimization regularization algorithm, including anO(ϵ −3/2)complexity bound.SIAM Journal on Optimization, 33(3), 1621–1646, 2023

    2023

  11. [17]

    Serge Gratton, Sadok Jerad, and Philippe L. Toint. A Stochastic Objective-Function-Free Adaptive Regularization Method with Optimal ComplexityOpen Journal of Mathematical Optimization, 6(5):1– 24, 2025

    2025

  12. [19]

    S2MPJ and CUTEst optimization problems for Matlab, Python and Julia

    Serge Gratton and Philippe L Toint. S2MPJ and CUTEst optimization problems for Matlab, Python and Julia. arxiv:2407.07812, 2024

    2024 arXiv

  13. [20]

    Adaptive regularization minimization algorithms with nonsmooth norms.IMA Journal of Numerical Analysis, 43(2):920—949 2022

    Serge Graton and Philippe L Toint. Adaptive regularization minimization algorithms with nonsmooth norms.IMA Journal of Numerical Analysis, 43(2):920—949 2022

    2022

  14. [21]

    Revisiting gradient clipping: Stochastic bias and tight convergence guarantees

    Anastasia Koloskova, Hadrien Hendrikx, and Sebastian U Stich. Revisiting gradient clipping: Stochastic bias and tight convergence guarantees. InProceedings of the 40th International Conference on Machine Learning, pages 17343–17363. PMLR, 2023

    2023

  15. [22]

    Convex and non-convex op- timization under generalized smoothness

    Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. Convex and non-convex op- timization under generalized smoothness. InAdvances in Neural Information Processing Systems, vol- ume 36, pages 40238–40271, 2023

    2023

  16. [23]

    Convergence of adam under relaxed assumptions

    Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. Convergence of adam under relaxed assumptions. InAdvances in Neural Information Processing Systems, volume 36, pages 52166–52196, 2023

    2023

  17. [24]

    Adaptive gradient descent without descent

    Yura Malitsky and Konstantin Mishchenko. Adaptive gradient descent without descent. InProceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pages 6702–6712. PMLR, 2020

    2020

  18. [25]

    Regularized newton method with globalo(1/k 2) convergence.SIAM Journal on Optimization, 33(3):1440–1462, July 2023

    Konstantin Mishchenko. Regularized newton method with globalo(1/k 2) convergence.SIAM Journal on Optimization, 33(3):1440–1462, July 2023

    2023

  19. [26]

    Roman A. Polyak. Regularized newton method for unconstrained convex optimization.Mathematical Programming, 120(1):125–145, June 2007

    2007

  20. [27]

    Margherita Porcelli and Philippe L. Toint. A note on using performance and data profiles for training algorithms.ACM Transactions on Mathematical Software, 45(2):1–10, April 2019

    2019

  21. [28]

    Variance-reduced clipping for non-convex optimization

    Amirhossein Reisizadeh, Haochuan Li, Subhro Das, and Ali Jadbabaie. Variance-reduced clipping for non-convex optimization. arXiv:2303.00883, 2023

    2023 arXiv

  22. [29]

    A Newton-CG algorithm with complexity guarantees for smooth unconstrained optimizationMathematical Programming, 180(1-2):451–488, 2019

    2019

  23. [30]

    Convergence of steinvariational gradient descent under a weaker smoothness condition

    Lukang Sun, Avetik Karagulyan, and Peter Richtarik. Convergence of steinvariational gradient descent under a weaker smoothness condition. InProceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206, pages 3693–3717. PMLR, 2023

    2023

  24. [31]

    A regularized newton method without line search for unconstrained optimization.Computational Optimization and Applications, 59(1-2):321–351, 2014

    Kenji Ueda and Nobuo Yamashita. A regularized newton method without line search for unconstrained optimization.Computational Optimization and Applications, 59(1-2):321–351, 2014

    2014

  25. [32]

    Gradient norm regularization second-order algorithms for solving nonconvex- strongly concave minimax problems.ArXiv, abs/2411.15769, 2024

    Junlin Wang and Zi Xu. Gradient norm regularization second-order algorithms for solving nonconvex- strongly concave minimax problems.ArXiv, abs/2411.15769, 2024

    2024 arXiv

  26. [34]

    Zhewei Yao, Peng Xu, Fred Roosta, and Michael W. Mahoney. Inexact nonconvex newton-type methods. INFORMS Journal on Optimization, 3(2):154–182, 2021. Gratton, Jerad, Toint: Fast Newton Method Under Local Smoothness28

    2021

  27. [35]

    Inexact newton-CG algorithms with complexity guarantees.IMA Journal of Numerical Analysis, 43(3):1855–1897, 2022

    Zhewei Yao, Peng Xu, Fred Roosta, Stephen J Wright, and Michael W Mahoney. Inexact newton-CG algorithms with complexity guarantees.IMA Journal of Numerical Analysis, 43(3):1855–1897, 2022

    2022

  28. [36]

    Improved analysis of clipping algorithms for non- convex optimization

    Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non- convex optimization. InAdvances in Neural Information Processing Systems, volume 33, pages 15511– 15521. Curran Associates, Inc., 2020

    2020

  29. [37]

    Adabb: Adaptive Barzilai-Borwein method for convex optimization

    Danqing Zhou, Shiqian Ma, and Junfeng Yang. Adabb: Adaptive Barzilai-Borwein method for convex optimization. arXiv:2401.08024, 2024

    2024 arXiv

  30. [38]

    Why gradient clipping accelerates training: A theoretical justification for adaptivity

    Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. InInternational Conference on Learning Representations, 2020

    2020

  31. [39]

    Yuhao Zhou, Jintao Xu, Bingrui Li, Chenglong Bao Chao Ding, and Jun Zhu A Regularized Newton Method for Nonconvex Optimization with Global and Local Complexity Guarantees InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. A Proof of Lemma 1.2 ...

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.