Pith. sign in

REVIEW 5 major objections 6 minor 58 references

Permutation Randomization on Nonsmooth Nonconvex Optimization: A Theoretical and Experimental Study

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Permutation randomization lets gradient-based optimizers approximate the global optimum on nonsmooth nonconvex objectives while preserving their convergence rate.

desk verdict The central theorem assumes the conclusion and is false as stated; the paper is not salvageable in its current form, though the empirical question it asks is a real one. read the letter →

arxiv 2505.11752 v1 pith:3NWJ4GE2 submitted 2025-05-16 cs.LG

Wei Zhang , Arif Hassan Zidan , Afrar Jahin , Yu Bao , Tianming Liu This is my paper · ORCID
classification cs.LG
keywords permutationrandomizationgradient-basedoptimizationnonsmoothnonconvexLipschitzcontinuousglobaloptimumconvergenceraterandomizedADAMoperatornorm
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a simple coordinate-permutation randomization can make gradient-based optimizers global optimizers for nonsmooth nonconvex Lipschitz objectives. The mechanism is coverage: after enough iterations and in high enough dimension, the closed cubes around the permuted iterates cover the whole domain, hence a closed cube around the global optimum; the permutation has operator norm one, so the underlying optimizer's convergence rate is preserved. If correct, this gives a dimension-free theoretical role for randomization in a setting where deterministic first-order methods face known limitations. The authors demonstrate the idea by adding permutation to ADAM and reporting lower reconstruction loss than ADAM, ADMM, and SVRG on deep matrix factorization, a deep belief network, and noisy objectives.

What carries the argument

The load-bearing object is the permutation randomization operator $R:\mathbb{R}^D\to\mathbb{R}^D$, which reorders all coordinates of the current point; because it is a finite-dimensional isometry, $\|R\|=1$. $R$ is interleaved with a gradient-based optimizer operator $G$, so the iterate is $R_t G_t f(I_0)$. The argument uses the contraction and shrinkage property of $G$ to explain why plain optimizers stall, and Heine-Borel and Vitali covering theorems to argue that the union of cubes around randomized iterates fills $I$. The norm-one property is what prevents $R$ from slowing the base optimizer.

What would settle it

Run a two-dimensional nonsmooth nonconvex Lipschitz function whose global minimizer has a coordinate value that never appears in any coordinate of any iterate of the deterministic optimizer; since permutation only reorders existing coordinate values, that minimizer can never be covered, and observing that the randomized optimizer cannot approach it would refute the coverage claim of Theorem 4.3.

Watch

Extended reading notes

Core claim

The central claim is Theorem 4.3: given a Lipschitz continuous nonsmooth nonconvex $f:\mathbb{R}^D\to\mathbb{R}$, interleaving a gradient-based optimizer $G$ with a permutation operator $R$ that reorders coordinates produces iterates $\hat{x}_{i,t}$ whose closed cubes $B(\hat{x}_{i,t},\delta_{i,t})$ cover the domain $I$ when $T$ and $D$ are sufficiently large, and therefore cover the closed cubes around the global optimum. The proof splits into overlapping cubes, handled by the Heine-Borel theorem, and non-overlapping cubes, handled by the Vitali covering theorem. Theorem 4.5 claims the permutation has operator norm $\|R\|=1$, so the convergence rate of the included optimizer is unchanged; Theorem 4.6 states the upper bound $\frac{1}{\sqrt{T}}\le 2\,|\psi(B(x_{\mathrm{gbest}},\delta))|$ for a one-dimensional $\psi$. The paper's experiments corroborate the claim by showing randomized ADAM keeps updating late in training and outperforms ADAM, ADMM, and SVRG on the chosen tasks.

Load-bearing premise

The proof's key step is assuming that, once the iteration count and dimension are large enough, the small boxes around the points the optimizer has visited cover the entire search space; this coverage is asserted rather than derived, and a permutation only reorders coordinate values the optimizer has already produced.

Editorial extensions

If this is right

  • If Theorem 4.3 holds, any gradient-based optimizer augmented with coordinate permutation can approach a closed cube around the global optimum on Lipschitz nonsmooth nonconvex problems, given enough iterations.
  • Because $\|R\|=1$, the random permutation adds no asymptotic slowdown: the randomized optimizer inherits the base optimizer's convergence rate.
  • Permutation can be applied only when the gradient difference is small, and the experiments indicate this lets the optimizer escape stationary points and keep updating late in training.
  • The empirical results imply that randomized ADAM is a practical replacement for ADAM on deep stacked architectures and noisy objectives, with better reconstruction accuracy and lower runtime than SVRG.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof of coverage is combinatorial rather than dynamical: permutation never creates new coordinate magnitudes, it only rearranges existing ones, so the real engine of Theorem 4.3 is the assumption that the visited coordinate values are dense enough in the domain; a natural test is to measure visited-cube volume directly and see whether sparse permutation triggering preserves the guarantee.
  • If the norm-one argument is the reason convergence is preserved, then any measure-preserving coordinate transformation, such as cyclic shifts or sign flips, should behave like permutation; this is an untested prediction of the paper's reasoning.
  • The algorithms in the experiments trigger permutation only when $\|g_t-g_{t-1}\|<\epsilon$, while the theorems consider permutation at every step, so closing that gap would decide whether the practical algorithm is fully covered by Theorem 4.3.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper studies permutation randomization as a wrapper around gradient-based optimizers for nonsmooth nonconvex Lipschitz objectives. It claims (Theorem 4.3) that, with sufficiently many iterations and sufficiently high dimension, the union of cubes centered at permuted iterates covers the domain, and hence approximates a closed cube around the global optimum; it also claims (Theorem 4.5) that permutation randomization preserves the underlying optimizer's convergence rate. The authors support the theoretical claims with experiments comparing a 'randomized ADAM' against ADAM, ADMM, and SVRG on deep matrix factorization, deep belief networks, noisy objectives, and logistic regression.

Significance. If the central theorem were correct, the result would be remarkable: it would imply that a purely combinatorial coordinate permutation, applied to any gradient-based method, turns a local method into a global optimizer for all Lipschitz nonsmooth nonconvex functions, in contrast to known lower bounds for deterministic nonsmooth nonconvex optimization. The paper also provides pseudo-code and experiments on public data. However, the proof of the main theorem is circular, and the theorem is false as stated; a simple two-dimensional counterexample shows that permutation randomization cannot escape a fixed point whose coordinate set contains only the starting values. The contraction lemma underpinning the negative result is asserted without proof, and several auxiliary statements are mathematically incorrect. The central claims are therefore not established.

major comments (5)
  1. [Theorem 4.3, Eq. (2) and Appendix B, Eqs. (B1), (B8)] The proof of the main global-coverage claim assumes the conclusion. In Eq. (2) and Appendix B (B8), the authors state that when T and D are sufficiently large, the union of cubes centered at the permuted iterates covers the whole domain I, and they cite Lemma 4.1 (Heine-Borel) for this step. But a union of finitely many cubes covers I only if the iterates happen to visit a set of points whose cubes cover I; this is precisely the statement to be proved. No argument is given to show that permutation randomization forces the iterates to visit such a set. The proof is therefore circular.
  2. [Theorem 4.3, counterexample] The claimed guarantee is false as stated. Take D=2, I=[-2,2]^2, and f(x,y)=g(x)+(y-1)^2 with g(x)=min((x-1)^2+1,(x+1)^2). The function is Lipschitz, nonsmooth, and nonconvex on I. Starting at (1,1), gradient descent has zero partial derivative in both coordinates, so every unrandomized iterate is (1,1); every permutation of (1,1) is (1,1). The global minimizer is (-1,1), and for any fixed radius delta<1, no cube centered at (1,1) covers B((-1,1),delta). Thus Theorem 4.3's conclusion fails despite arbitrarily large T (and D=2). Permutation randomization only reorders coordinate values that already appear in the iterate; it cannot create new coordinate values or visit unvisited regions.
  3. [Lemma 3.1 and Appendix A] Lemma 3.1 asserts that any gradient-based optimizer without randomization satisfies G(f(I_t)) = I_{t+1} subset of I_t for all t, i.e., nested contraction of the iterate set. This is not a property of gradient-based optimizers in general: gradient descent with a poorly chosen step size can increase the norm of the iterate or oscillate. The 'proof by contradiction' assumes that if the set ever expanded, iterates would diverge to infinity, which conflates set-theoretic expansion with norm growth of a sequence. The lemma is used to prove Theorem 3.2, so the claimed limitation of non-randomized optimizers is not established.
  4. [Lemma 4.1 and Lemma 4.2] Both lemmas are mathematically incorrect as stated. Lemma 4.1 claims that for a closed bounded set Gamma and a closed set G = {g_i}_{i=1}^K, the union of finitely many closed sets g_i covers Gamma; this is false without a covering assumption (e.g., take two disjoint closed intervals in [0,2]). Lemma 4.2's statement of the Vitali covering theorem is also not a standard formulation: finitely many disjoint closed sets with small outer measure of the complement do not constitute a Vitali covering in the usual sense, and the countable/measure-theoretic conditions are missing. Since these lemmas are the cited justification for the covering step in Theorem 4.3, the proof collapses.
  5. [Theorem 4.5 and Theorem 4.6, Appendix B] The convergence-rate preservation result is not proven. In Appendix B, Eq. (B17) states ||G(f(X)-f(Y))|| <= c||G(f(X)-f(Y))|| with 0<c<1, which is trivially false unless the left side is zero; the intended contraction inequality should involve ||f(X)-f(Y)|| on the right. The subsequent argument only shows that the operator norm of R is 1, which does not imply that the composed iteration preserves a specific convergence rate. Theorem 4.6's statement and proof are incomprehensible: the set equality involves unbound variables and 'Cauchy Theorem' is invoked without a clear statement; the final bound 1/sqrt(T) <= 2|f(x_{i,gbest},delta)| appears without derivation from the preceding lines. These results do not constitute a valid proof.
minor comments (6)
  1. [Definition 2.1] The notation \(x,y \subseteq \mathbb{R}^D\) is incorrect; x and y should be elements, not subsets. Also, \(\|f(x)-f(y)\|\le L\|x-y\|\) with f scalar-valued typically uses absolute values, and the L should be assumed nonnegative.
  2. [Assumption 2.1] Assumption 2.1 is nearly vacuous: every point is the center of some cube of positive radius. It does not constrain the iterates or the covering property, so it cannot support the theorem's coverage conclusion.
  3. [Definition 2.4 and 2.5] The terminology 'closed interval' is used for what is a finite set of D points, and notations like \(G_t \cdot f(I_0)\) are used inconsistently (an operator acting on a function value rather than on an iterate). This makes the definitions hard to parse.
  4. [Table 3 (Appendix C)] The pseudo-code for randomized ADAM updates the second moment as \(V_t \leftarrow \beta_1 V_{t-1} + (1-\beta_1) g_t\), which should involve \(\beta_2\) and \((1-\beta_2)\) in standard ADAM. As written, the second moment estimate uses the wrong decay factor.
  5. [Introduction and Related Work] The statement that 'deterministic, dimension-free, first-order optimizers cannot outperform randomized methods' mischaracterizes Jordan et al. (2023), which gives lower bounds on deterministic methods; it does not establish that randomized methods achieve dimension-free global convergence on nonsmooth nonconvex problems.
  6. [Section 5] The experiments show lower reconstruction loss on specific datasets but do not measure whether the optimizer approximates a global optimum or covers the domain; thus the empirical results do not corroborate Theorem 4.3, which is the paper's central claim.

Circularity Check

3 steps flagged · score 8.0 of 10

Theorem 4.3's global-coverage conclusion is assumed rather than derived: Eq. (B8) states the conclusion as a 'sufficiently large' condition, and Lemma 4.1 is stated so that the covering relation is already its conclusion.

  1. self definitional [Section 4, Theorem 4.3, Eq. (2); Appendix B, Eq. (B8)]
    "Suppose T and D are sufficiently large, inferring from Lemma 4.1, according to Definition 2.6, if we denote the closed cubes of global optimum as B(x_1,gbest,δ_1), B(x_2,gbest,δ_2), ..., we have: I ⊆ ⋃_{t=1}^{T} ⋃_{i=1}^{D} B(ˆx_{i,t},δ_{i,t}) (B8)."

    The theorem's conclusion is that the union of cubes centered at permuted iterates covers I and hence contains the global-optimum cube. The proof obtains this exact inclusion in Eq. (B8) by first stipulating 'Suppose T and D are sufficiently large'; no mechanism is given by which large T and D force the permuted iterates into every region of I. Since permutation only reorders the coordinate values already present in the current iterate, the permuted iterates are confined to the coordinates visited by the unrandomized optimizer, and regions never visited are never covered. Thus Eq. (B8) is the conclusion restated as an assumption, not a derived result.

  2. self definitional [Section 4, Lemma 4.1]
    "Lemma 4.1 (Heine-Borel Theorem). [Royden, 1968] Assume Γ is a close and bounded set and {g_i}_{i=1}^K = G is a closed set. Then ⋃_{i=1}^K g_i ⊇ Γ, and G = ℵ0."

    The proof of Theorem 4.3 says 'inferring from Lemma 4.1' to obtain the covering I ⊆ ⋃_{t,i} B(ˆx_{i,t},δ_{i,t}). But Lemma 4.1, as stated, already contains the desired covering relation ⋃ g_i ⊇ Γ as its conclusion, with no hypothesis that the g_i actually cover Γ. The theorem's global-cover conclusion is therefore not derived from the permutation-randomized dynamics; it is borrowed from a lemma whose statement is the same covering assertion. This is a self-definitional reduction: the 'external theorem' is being used to supply the very inclusion that the theorem is supposed to prove.

1 more flagged steps
  1. self definitional [Section 4, Lemma 4.4 and Theorem 4.5; Appendix B, Eq. (B21)]
    "Inferring from Lemma 4.4, it is obvious that we have: ||R||·||G·(f(I_{t+1})−f(I_t))|| = ||G·(f(I_{t+1})−f(I_t))|| ≤ c·||f(I_{t+1})−f(I_t)|| (B21)."

    Theorem 4.5's conclusion that permutation randomization 'preserves the convergence rate' is derived entirely from Lemma 4.4, whose content is that a permutation has operator norm 1 because it only reorders coordinates. That is a definitional property of permutations under the Euclidean norm: the equality in (B21) holds by construction. The theorem therefore labels a built-in isometry property of the operator R as a proved 'preservation' result; it does not establish any rate statement beyond the norm equality already contained in Definition 2.3.

full rationale

The central theoretical claim, Theorem 4.3, is circular in the strongest sense: its proof (Eq. (B8), also Eq. (2)) assumes the covering of I by permuted-iterate cubes and then concludes that the global-optimum cube is covered. The invoked Lemma 4.1 is stated with the covering relation already as its conclusion, so the 'inference' is a restatement of the target. The alternative Vitali branch similarly assumes the complement has arbitrarily small measure, which is the same coverage conclusion in measure form. Because permutation only reorders the coordinate values already present in the iterates, large T and D cannot by themselves force coverage of regions the unrandomized optimizer never visits; a two-coordinate example with a fixed point at (1,1) and a global minimum at (-1,1) exhibits the claimed guarantee failing. The secondary result, Theorem 4.5, is also definitional rather than derived: it follows from permutations being norm-preserving rearrangements, so the 'preservation of convergence rate' is a tautological consequence of Definition 2.3. The empirical section is self-contained and not circular, but experiments cannot repair a proof that assumes its main conclusion. No self-citation chain is load-bearing here; the circularity is internal to the proof structure. Overall score 8: the central result is forced by the stated assumptions and by the shape of Lemma 4.1, not derived from the optimizer dynamics.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central proof rests on two unstated assumptions: the nested contraction of unrandomized optimizers (Lemma 3.1) and the coverage of the domain by permuted iterates (Theorem 4.3). Neither is derived from the Lipschitz assumption. The empirical method adds a hand-chosen trigger threshold. No new physical entities are introduced.

free parameters (3)
  • Permutation trigger threshold = 1.0e-2
    Permutation randomization is applied only when the difference between current and previous gradient is below this threshold (Section 5). The value is chosen by hand and no sensitivity analysis is reported.
  • Closed cube radii delta_i = unspecified
    Assumption 2.1 and Definition 2.6 posit closed cubes B(x_i, delta_i) with positive radii for each coordinate; the theorem's coverage conclusion depends on these radii but they are never set, estimated, or bounded.
  • Maximum iterations T = 200 (stated) / 2000 (figure captions)
    Section 5 says only maximum iterations are manually determined as two hundred iterations; figure captions say results are within 2000 iterations. The number is a chosen experimental setting, not derived from any convergence requirement.
assumptions (4)
  • ad hoc to paper A gradient-based optimizer without randomization satisfies G·f(I_t) = I_{t+1} subset of I_t for all t (nested contraction of the iterate set).
    Lemma 3.1 and its proof in Appendix A assume this nested-set contraction. It does not hold for generally used optimizers such as GD or ADAM, and it is not implied by Lipschitz continuity or standard convergence assumptions.
  • ad hoc to paper For sufficiently large T and D, the union of closed cubes centered at the permuted iterates covers the whole domain I.
    Theorem 4.3 (Eq. (2)/(B1)) asserts this coverage and then concludes the global optimum cube is covered; the coverage assertion is effectively the conclusion.
  • domain assumption The objective function f is Lipschitz continuous on a closed domain I, with a smooth subset I' and nonsmooth nonconvex behavior on I\I'.
    Assumption 2.1 and Definition 2.1; standard background for nonsmooth analysis but not sufficient for the claimed global coverage.
  • standard math The covering theorems (Heine-Borel, Lemma 4.1; Vitali, Lemma 4.2) are applied as if they imply that finite families of the constructed closed cubes cover I.
    The paper invokes these theorems to justify the coverage conclusion; as stated, the applications do not follow from the theorems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Permutation Randomization on Nonsmooth Nonconvex Optimization: A Theoretical and Experimental Study." pith.science (2026). https://pith.science/paper/3NWJ4GE2

@misc{pith2026250511752,
  author       = {Pith},
  title        = {Pith review of: Permutation Randomization on Nonsmooth Nonconvex Optimization: A Theoretical and Experimental Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3NWJ4GE2}},
  note         = {Machine review of arXiv:2505.11752}
}
read the original abstract

While gradient-based optimizers that incorporate randomization often showcase superior performance on complex optimization, the theoretical foundations underlying this superiority remain insufficiently understood. A particularly pressing question has emerged: What is the role of randomization in dimension-free nonsmooth nonconvex optimization? To address this gap, we investigate the theoretical and empirical impact of permutation randomization within gradient-based optimization frameworks, using it as a representative case to explore broader implications. From a theoretical perspective, our analyses reveal that permutation randomization disrupts the shrinkage behavior of gradient-based optimizers, facilitating continuous convergence toward the global optimum given a sufficiently large number of iterations. Additionally, we prove that permutation randomization can preserve the convergence rate of the underlying optimizer. On the empirical side, we conduct extensive numerical experiments comparing permutation-randomized optimizer against three baseline methods. These experiments span tasks such as training deep neural networks with stacked architectures and optimizing noisy objective functions. The results not only corroborate our theoretical insights but also highlight the practical benefits of permutation randomization. In summary, this work delivers both rigorous theoretical justification and compelling empirical evidence for the effectiveness of permutation randomization. Our findings and evidence lay a foundation for extending analytics to encompass a wide array of randomization.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 51 canonical work pages

  1. [1]

    Allen-Zhu

    Z. Allen-Zhu. The first direct acceleration of stochastic gradient methods. Journal of Machine Learning Research, 18 0 (1): 0 8194--8244, 2017

    work page 2017

  2. [2]

    Allen-Zhu and Y

    Z. Allen-Zhu and Y. Yuan. Improved svrg for non-strongly-convex or sum-of-non-convex objectives. In International conference on machine learning, pages 1080--1089, 2016

    work page 2016

  3. [3]

    Arjevani and O

    Y. Arjevani and O. Shamir. Communication complexity of distributed convex learning and optimization. In Advances in neural information processing systems, volume 28, 2015

    work page 2015

  4. [4]

    Arjevani, Y

    Y. Arjevani, Y. Carmon, J. C. Duchi, D. J. Foster, N. Srebro, and B. Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199 0 (1-2): 0 165--214, 2023

    work page 2023

  5. [5]

    Atenas, C

    F. Atenas, C. Sagastiz \'a bal, P. J. Silva, and M. Solodov. A unified analysis of descent sequences in weakly convex optimization, including convergence rates for bundle methods. SIAM Journal on Optimization, 33 0 (1): 0 89--115, 2023

    work page 2023

  6. [6]

    J. J. Bartko. On various intraclass correlation reliability coefficients. Psychological bulletin, 83 0 (5): 0 762, 1976

    work page 1976

  7. [7]

    Bena \" m, J

    M. Bena \" m, J. Hofbauer, and S. Sorin. Stochastic approximations and differential inclusions. SIAM Journal on Control and Optimization, 44 0 (1): 0 328--348, 2005

    work page 2005

  8. [8]

    Bolte and E

    J. Bolte and E. Pauwels. Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming, 188: 0 19--51, 2021

    work page 2021

Show all 58 references
  1. [9]

    M. A. Bujang and N. Baharum. A simplified guide to determination of sample size requirements for estimating the value of intraclass correlation coefficient: a review. Archives of Orofacial Science, 12 0 (1), 2017

    2017

  2. [10]

    J. V. Burke, A. S. Lewis, and M. L. Overton. Approximating subdifferentials by random sampling of gradients. Mathematics of Operations Research, 27 0 (3): 0 567--584, 2002

    2002

  3. [11]

    J. V. Burke, A. S. Lewis, and M. L. Overton. A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM Journal on Optimization, 15 0 (3): 0 751--779, 2005

    2005

  4. [12]

    J. V. Burke, F. E. Curtis, A. S. Lewis, M. L. Overton, and L. E. Sim \ o es. Gradient sampling methods for nonsmooth optimization. Numerical nonsmooth optimization: State of the art algorithms, pages 201--225, 2020

    2020

  5. [13]

    F. H. Clarke. Necessary conditions for nonsmooth variational problems. In Optimal Control Theory and its Applications: Proceedings of the Fourteenth Biennial Seminar of the Canadian Mathematical Congress University of Western Ontario, August 12--25, 1973, pages 70--91. Springer, 1974

    1973

  6. [14]

    F. H. Clarke. Generalized gradients and applications. Transactions of the American Mathematical Society, 205: 0 247--262, 1975

    1975

  7. [15]

    F. H. Clarke. Generalized gradients of lipschitz functionals. Advances in Mathematics, 40 0 (1): 0 52--67, 1981

    1981

  8. [16]

    F. H. Clarke. Optimization and nonsmooth analysis. SIAM, 1990

    1990

  9. [17]

    A. D. Cohen and Y. Wang. Multiband multi-echo bold fmri. https://openfmri.org/dataset/ds000254/, 2018. Accessed: 2018-07-17

    2018

  10. [18]

    Cutkosky and F

    A. Cutkosky and F. Orabona. Momentum-based variance reduction in nonconvex sgd. In Advances in neural information processing systems, volume 32, 2019

    2019

  11. [19]

    Daniilidis and D

    A. Daniilidis and D. Drusvyatskiy. Pathological subgradient dynamics. SIAM Journal on Optimization, 30 0 (2): 0 1327--1338, 2020

    2020

  12. [20]

    Defazio, F

    A. Defazio, F. Bach, and S. Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in neural information processing systems, volume 27, 2014

    2014

  13. [21]

    Dubois-Taine, S

    B. Dubois-Taine, S. Vaswani, R. Babanezhad, M. Schmidt, and S. Lacoste-Julien. Svrg meets adagrad: Painless variance reduction. Machine Learning, 111 0 (12): 0 4359--4409, 2022

    2022

  14. [22]

    A. Field. Discovering statistics using IBM SPSS statistics. sage, 2013

    2013

  15. [23]

    Forneron

    J.-J. Forneron. Noisy, non-smooth, non-convex estimation of moment condition models. arXiv preprint arXiv:2301.07196, 2023

    2023 arXiv

  16. [24]

    Gaudioso, S

    M. Gaudioso, S. Taheri, A. M. Bagirov, and N. Karmitsa. Bundle enrichment method for nonsmooth difference of convex programming problems. Algorithms, 16 0 (8): 0 394, 2023

    2023

  17. [25]

    Goldstein

    A. Goldstein. Optimization of lipschitz continuous functions. Mathematical Programming, 13: 0 14--22, 1977

    1977

  18. [26]

    Hazan, A

    E. Hazan, A. Agarwal, and S. Kale. Logarithmic regret algorithms for online convex optimization.machine learning. Machine Learning, 69 0 (2): 0 169--192, 2007

    2007

  19. [27]

    G. E. Hinton. Deep belief networks. Scholarpedia, 4 0 (5): 0 5947, 2009

    2009

  20. [28]

    Johnson and T

    R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information processing systems, volume 26, 2013

    2013

  21. [29]

    Jordan, G

    M. Jordan, G. Kornowski, T. Lin, O. Shamir, and M. Zampetakis. Deterministic nonsmooth nonconvex optimization. In The Thirty Sixth Annual Conference on Learning Theory, pages 4570--4597. PMLR, 2023

    2023

  22. [30]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014

    2014 arXiv

  23. [31]

    K. C. Kiwiel. Restricted step and levenberg--marquardt techniques in proximal bundle methods for nonconvex nondifferentiable optimization. SIAM Journal on Optimization, 6 0 (1): 0 227--249, 1996

    1996

  24. [32]

    K. C. Kiwiel. Convergence of the gradient sampling algorithm for nonsmooth nonconvex optimization. SIAM Journal on Optimization, 18 0 (2): 0 379--388, 2007

    2007

  25. [33]

    LeCun, Y

    Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521 0 (7553): 0 436--444, 2015

    2015

  26. [34]

    Lei and M

    L. Lei and M. Jordan. Less than a single pass: Stochastically controlled stochastic gradient. In Artificial Intelligence and Statistics, pages 148--156, 2017

    2017

  27. [35]

    Y. Lei. Stability and generalization of stochastic optimization with nonconvex and nonsmooth problems. In The Thirty Sixth Annual Conference on Learning Theory, pages 191--227. PMLR, 2023

    2023

  28. [36]

    X. Lian, M. Wang, and J. Liu. Finite-sum composition optimization via variance reduced gradient descent. In Artificial Intelligence and Statistics, pages 1159--1167. PMLR, 2017

    2017

  29. [37]

    H. Lin, J. Mairal, and Z. Harchaoui. A universal catalyst for first-order optimization. In Advances in neural information processing systems, volume 28, 2015

    2015

  30. [38]

    J. Lin, C. Song, K. He, L. Wang, and J. E. Hopcroft. Nesterov accelerated gradient and scale invariance for adversarial attacks. arXiv preprint arXiv:1908.06281, 2019

    1908 arXiv

  31. [39]

    Nair and G

    V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807--814, 2010

    2010

  32. [40]

    T. Nan, Y. Gao, and C. Kroer. Extragradient svrg for variational inequalities: Error bounds and increasing iterate averaging. arXiv preprint arXiv:2306.01796, 2023

    2023 arXiv

  33. [41]

    Nemirovski, A

    A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 19 0 (4): 0 1574--1609, 2009

    2009

  34. [42]

    Nesterov

    Y. Nesterov. Smooth minimization of non-smooth functions. Mathematical programming, 103: 0 127--152, 2005

    2005

  35. [43]

    Nesterov et al

    Y. Nesterov et al. Lectures on convex optimization, volume 137. Springer, 2018

    2018

  36. [44]

    Nishihara, L

    R. Nishihara, L. Lessard, B. Recht, A. Packard, and M. Jordan. A general analysis of the convergence of admm. In International Conference on Machine Learning, pages 343--352, 2015

    2015

  37. [45]

    S. Pei, C. Wang, S. Cao, and Z. Lv. Data augmentation for fmri-based functional connectivity and its application to cross-site adhd classification. IEEE Transactions on Instrumentation and Measurement, 72: 0 1--15, 2022

    2022

  38. [46]

    H. L. Royden. Real analysis. Pearson, 1968

    1968

  39. [47]

    W. Rudin. Functional analysis. McGraw-Hill, University of Michigan, 2 edition, 1973

    1973

  40. [48]

    Schober and T

    P. Schober and T. R. Vetter. Logistic regression in medical research. Anesthesia & Analgesia, 132 0 (2): 0 365--366, 2021

    2021

  41. [49]

    Shalev-Shwartz and T

    S. Shalev-Shwartz and T. Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14 0 (2), 2013

    2013

  42. [50]

    Y. Shen, Z. Wen, and Y. Zhang. Augmented lagrangian alternating direction method for matrix separation based on low-rank factorization. Optimization Methods and Software, 29: 0 236--263, 2014

    2014

  43. [51]

    M. Shut. Breast cancer data. https://www.kaggle.com/datasets/marshuu/breast-cancer?resource=download, 2023. Accessed: 2023-01-01

    2023

  44. [52]

    Trigeorgis, K

    G. Trigeorgis, K. Bousmalis, S. Zafeiriou, and B. Schuller. A deep semi-nmf model for learning hidden representations. In International conference on machine learning, pages 1692--1700. PMLR, 2014

    2014

  45. [53]

    Trigeorgis, K

    G. Trigeorgis, K. Bousmalis, S. Zafeiriou, and B. W. Schuller. A deep matrix factorization method for learning attribute representations. IEEE transactions on pattern analysis and machine intelligence, 39 0 (3): 0 417--429, 2016

    2016

  46. [54]

    Von Oswald, E

    J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151--35174. PMLR, 2023

    2023

  47. [55]

    Wang and B

    L. Wang and B. Shen. On the parallelization upper bound for asynchronous stochastic gradients descent in non-convex optimization. Journal of Optimization Theory and Applications, 196 0 (3): 0 900--935, 2023

    2023

  48. [56]

    Z. Wen, W. Yin, and Y. Zhang. Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm. Mathematical Programming Computation, 4: 0 333--361, 2012

    2012

  49. [57]

    K. Yosida. Functional analysis. Springer Science & Business Media, 2012

    2012

  50. [58]

    Zhang, M

    Y. Zhang, M. J. Wainwright, and J. C. Duchi. Communication-efficient algorithms for statistical optimization. In Advances in neural information processing systems, volume 25, 2012

    2012

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.