Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that layer-wise preconditioning, the optimizer family behind KFAC and Shampoo, is statistically necessary for feature learning once input covariates are anisotropic.

desk verdict The linear representation learning result is a real, mostly solid contribution; the single-index half proves feature alignment but leaves the bridge to generalization as an admitted sketch, and the abstract's 'provably necessary' claim overstates what is shown. read the letter →

arxiv 2502.01763 v1 pith:OZMXQ4JY submitted 2025-02-03 cs.LG math.OCstat.ML

Thomas T. Zhang , Behrad Moniri , Ansh Nagwekar , Faraz Rahman , Anton Xue , Hamed Hassani , Nikolai Matni This is my paper · ORCID
classification cs.LGmath.OCstat.ML MSC 68Q3268T07
keywords layer-wisepreconditioningKFACfeaturelearninganisotropiccovariateslinearrepresentationsingle-indexmodelSGDlowerboundcondition-number-freeconvergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that layer-wise preconditioning—the optimizer family behind KFAC and Shampoo—is not merely a practical accelerator but statistically necessary for feature learning once input covariates depart from the isotropic Gaussian ideal. In two prototypical models, linear representation learning and single-index learning, the paper proves that fixed-step-size SGD converges at a rate tied to the conditioning of the problem or learns an incorrectly correlated feature direction, while a stylized KFAC update removes those bottlenecks. The same update arises from first principles in both settings, which the paper presents as explaining why Kronecker-factored methods beat diagonal preconditioners like Adam on real optimization tasks. The reader should care because the result converts a practical folklore—that Shampoo and KFAC train better—into a precise statistical statement about what SGD leaves on the table.

What carries the argument

The central object is the stylized KFAC update (5), $G_+ = G - \eta_G (F^\top F)^{-1}\nabla_G\hat L(F,G)\,(\hat\Sigma_x+\lambda_G I)^{-1}$, with $F_+$ updated the same way using the post-activation covariance $\hat\Sigma_z$. It is Kronecker-factored because it pre- and post-multiplies the gradient by inverse second-moment matrices, the layer-wise analogue of multiplying by an inverse Fisher information. In the linear-representation proof, the left factor $(F^\top F)^{-1}$ removes the dependence on the task-head condition number $\kappa(F_*)$, and the right factor $\hat\Sigma_x^{-1}$ removes the dependence on input anisotropy; in the single-index proof, the right factor converts the SGD direction $X^\top y/n$ into the ridge-regression direction, and Lemma 3.1 shows the $F$-update is an exponential moving average of the least-squares estimator. These two moves carry the whole argument.

What would settle it

Run the single-index experiment of Section 4.2 with $\Sigma_x = \mathrm{diag}(1+\epsilon,\dots,1+\epsilon,1-\epsilon,\dots,1-\epsilon)$ and $\epsilon$ large, $n \gg d_X$, $\lambda_G \to 0$. Lemma 3.10 predicts the one-step SGD learned direction has limiting correlation $(\mathrm{Tr}(\Sigma_x)/d_X)/\sqrt{\mathrm{Tr}(\Sigma_x^2)/d_X}$, which is $<1$ for $\epsilon>0$; Lemma 3.12 predicts one-step KFAC correlation tends to 1. If the simulated SGD correlation stays at 1 (or KFAC's stays below 1), the paper's central claim is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the two preconditioners of the stylized KFAC update are exactly the objects that cancel SGD's failures. In linear representation learning with anisotropic $\mathbf{x}\sim\mathcal{N}(0,\Sigma_x)$, SGD's convergence rate has a lower bound $1-4\lambda_{\min}(F_*^\top F_*)/\lambda_{\max}(F_*^\top F_*)$ (Proposition 3.5), so severely ill-conditioned task heads make SGD arbitrarily slow. With left preconditioner $(F^\top F)^{-1}$ and right preconditioner $\hat\Sigma_x^{-1}$, the subspace distance contracts by $(1-0.9\eta_G)$ per step, independent of $\kappa(F_*)$ and $\kappa(\Sigma_x)$ (Theorem 3.6), under a local and sample-split condition. In single-index learning, one-step SGD's learned direction $\beta_{\mathrm{SGD}}=X^\top y/n$ has limiting correlation with $\beta_*$ equal to $\mathrm{Tr}(\Sigma_x)/\sqrt{d_X\,\mathrm{Tr}(\Sigma_x^2)}$, which is 1 if and only if $\Sigma_x$ is a scalar multiple of the identity (Lemma 3.10); the KFAC direction $(\hat\Sigma_x+\lambda_G I)^{-1}X^\top y/n$ is ridge regression and achieves correlation tending to 1 as $n\gg d_X$ and $\lambda_G\to 0$ (Lemma 3.12).

Load-bearing premise

The main theorems apply only when the current representation is already close to the true subspace (within $0.01/(\kappa(\Sigma_x)\kappa(F_*))$) and when the two layer updates are computed on independent data batches; if either fails, the paper does not prove a condition-number-free rate, and the claimed advantage could disappear.

Editorial extensions

If this is right

  • If Theorem 3.6 is right, then for ill-conditioned task heads KFAC-style preconditioning gives a universal-constant contraction rate on the representation subspace, whereas SGD's rate degrades with $\kappa(F_*)$; this makes the optimizer choice matter for representation recovery, not just optimization speed.
  • If Lemma 3.12 is right, then in the proportional high-dimensional limit a single KFAC preconditioned step on the first layer recovers the single-index direction $\beta_*$ under anisotropic covariates, while a single SGD step does not.
  • The transfer-learning bound (Lemma 3.7) decomposes fine-tuning error into a term proportional to the squared subspace distance of the learned representation plus a noise term; improving representation recovery therefore translates directly into better transfer with fewer target samples.
  • The paper's numerical comparison indicates that Adam-style diagonal preconditioning and batch norm do not close the gap observed in these models, supporting the claim that the layer-wise (Kronecker) structure is the operative ingredient.
  • Because the stylized KFAC update coincides with alternating least-squares minimization when $\eta_F=\eta_G=1$, the rate result also retroactively explains the convergence of min-min schemes; the descent view adds the flexibility of tuning $\eta_G$, which experiments show is generally better than 1.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the condition-number-free rate holds globally or with shared data batches; a natural test is to start from a random initialization with a single shared batch per layer update and check whether the experimental advantage survives, since the theory only covers the local sample-split regime.
  • The ridge-regression equivalence suggests a broader principle: any optimizer whose one-step update acts as ridge regression in the input covariance (e.g., certain full-matrix AdaGrad variants) should recover $\beta_*$ where SGD fails; this could be tested by comparing right-preconditioned-only methods against full KFAC.
  • The whitening analysis predicts a concrete transfer-learning failure mode: input normalization before pretraining should reduce downstream alignment whenever source and target covariances differ, even while improving training loss; this is testable in the paper's two-layer setup with mismatched $\Sigma_x$ and $\Sigma_x^{(t)}$.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that layer-wise Kronecker-factored preconditioning is a statistically motivated fix for SGD in two stylized feature-learning problems when inputs are anisotropic. In Section 3.1, for linear representation learning with Gaussian covariates, it proves a local contraction bound for a stylized KFAC update (Theorem 3.6) whose rate is independent of the condition numbers of the task head and input covariance, and contrasts this with a lower bound for fixed-step SGD (Proposition 3.5). In Section 3.2, for single-index learning in a proportional high-dimensional limit, it shows that one step of SGD produces a learned direction with limiting correlation strictly below 1 for non-isotropic inputs (Lemma 3.10), while one step of KFAC matches a ridge-regression direction whose correlation tends to 1 as the ridge parameter and sample-aspect ratio vanish (Lemma 3.12). Numerical experiments on the two models corroborate the correlation formulas and show that KFAC outperforms SGD, Adam, NGD, and batch-normalized variants in the linear representation learning task.

Significance. The paper makes a genuine conceptual contribution by deriving KFAC-type preconditioners from feature-learning desiderata rather than from curvature approximation, and it supplies a concrete condition-number-free contraction result in the linear representation problem. The proofs of the main linear-theory statements are detailed and algebraically coherent, and the asymptotic single-index correlation formulas are explicit and match simulations well. The lower bound in Proposition 3.5 is also a useful formalization of the limitation of fixed-step SGD. If the local and sample-split assumptions and the single-index generalization gap are clearly stated, the paper provides a valuable bridge between optimization folklore and feature-learning theory. However, as written, two load-bearing claims outrun the theorems: the single-index comparison stops at one-step weight alignment rather than generalization, and the word 'necessary' in the abstract is not supported by a theorem that rules out other preconditioning families.

major comments (3)
  1. [§3.2, Remark 3.13, Appendix C.4] The single-index advantage of KFAC is established only at the level of one-step weight alignment. Lemmas 3.10 and 3.12 characterize the correlation between the learned rank-one direction and β*, but the paper's own Appendix C.4 states that 'a complete analysis require tedious computations and is beyond the scope of this work' for converting alignment to a generalization guarantee. Since the experiments in Section 4.2 report correlation rather than test error or sample complexity, the conclusion that KFAC 'provably' fixes SGD in the nonlinear model is not a theorem: the random-feature component σ(XG0^T) and the nonlinear teacher component σ_{*,⊥} are not controlled in a rigorous test-error bound. I recommend either supplying a rigorous generalization statement in the proportional limit or explicitly rephrasing the single-index contribution as a feature-alignment result with a proof sketch, not as a provable generalization benefit.
  2. [§3.1, Theorem 3.6] The condition-number-free contraction rate is local and sample-split. Theorem 3.6 requires dist(G, G*) ≤ 0.01/(κ(Σx)κ(F*)) before the bound applies, and the G and F updates are computed on independent batches, as stated in Section 3. With random initialization in high dimension, dist(G0, G*) is typically near 1, so the theorem does not by itself establish the global advantage illustrated in Figure 1; the observed global convergence of KFAC is outside the proven regime. Because Proposition 3.5 is a worst-case lower bound over initializations, the claim that KFAC 'subverts' the lower bound should be qualified by the basin of attraction and the fresh-batch assumption, and the paper should either prove a basin-entry result or weaken the wording.
  3. [Abstract and Section 1] The abstract claims that layer-wise preconditioning methods are 'provably necessary' from a statistical perspective, but no theorem in the paper formalizes necessity. The formal results are a fixed-step SGD lower bound (Proposition 3.5), a local KFAC upper bound (Theorem 3.6), and one-step correlation comparisons (Lemmas 3.10 and 3.12). There is no lower bound excluding diagonal or other non-layer-wise preconditioners, and the Adam/batch-norm evidence in Section 4 is empirical rather than provable. I suggest replacing 'provably necessary' with a formulation such as 'provably beneficial in two prototypical models' or proving a formal necessity statement relative to a precisely defined class of preconditioners.
minor comments (5)
  1. [Section 3.2, Lemmas 3.10 and 3.12] The wording 'fails to learn useful features' in the contributions is too strong: Lemma 3.10 shows the SGD correlation can be substantial for moderate anisotropy and only goes below 1. A quantitative statement such as 'learns a direction with correlation strictly below the information-theoretically optimal value' would be more accurate.
  2. [Section 3.2, Lemma 3.12] The quantities Ψ1, Ψ2, Ψ3 appear in the statement of Lemma 3.12 but are defined only later in equation (34) of Appendix C.3. Add a forward reference or define them inline to make the main-text statement self-contained.
  3. [Section 3.2, Lemma 3.12] The double limit 'as λG → 0 and dX/n → 0' should specify the order or give explicit rates, since both limits act on the same ridge operator and the conclusion requires dX/n to vanish before λG becomes negligible for the unregularized limit to be meaningful.
  4. [Appendix B.1 and Appendix C.4] There are several typos and grammatical slips, including 'irregardless' in the proof of Lemma B.2 and 'the reminder term' in Appendix C.4; these should be corrected before publication.
  5. [Section 3.1, equation (4)] The definition of PG as bE[(∂f/∂h)^T (∂f/∂h)] with the option PG = I_dh is a little confusing for the linear network, where the theorem uses PG = F^T F; a sentence stating that for linear f, ∂f/∂h = F and hence PG = F^T F would improve clarity.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the KFAC updates are derived from the algebraic form of the gradient and then analyzed; self-citations serve as baselines, not as inputs to the derivation.

full rationale

The derivation chain is self-contained. In the linear model, Proposition 3.5 is an explicit adversarial lower bound rather than an assumption. The stylized KFAC update (5)/(8) is obtained by pre- and post-multiplying the SGD gradient by (F^T F)^{-1} and (Sigma_x-hat)^{-1}, and the key identity G_+ P_perp_* = (1 - eta_G) G P_perp_* + noise is exact algebra (eq. (16) and the proof of Theorem 3.6), so the condition-number-free contraction factor follows from the update's definition rather than being assumed. The noise and orthonormalization bounds are proved from concentration inequalities (Lemmas B.3-B.5 and Propositions B.6-B.7). In the single-index setting, Theorem 3.9 derives G_SGD ≈ G0 + alpha eta f0 beta_SGD^T from a gradient Taylor decomposition, and Lemmas 3.10 and 3.12 compute the limiting correlations of beta_SGD = X^T y/n and beta_KFAC = (Sigma_x-hat + lambda_G I)^{-1} X^T y/n; the latter is explicitly identified as ridge regression, and the correlation-to-1 limit is proved, not postulated. No fitted parameters are renamed as predictions; the theory curves in Section 4 are evaluated from population quantities. Self-citations (Zhang et al. 2024b; Moniri et al. 2024) appear as baselines, sample-splitting precedents, and related-work context, but the paper's own Proposition 3.5, Theorem 3.9, and Lemmas 3.10/3.12 carry the load independently. The genuine weaknesses are non-circular: Theorem 3.6 is local (dist(G, G*) ≤ 0.01/(kappa(Sigma_x) kappa(F*))) and sample-split, and the bridge from one-step weight alignment to generalization in the single-index model is explicitly a proof sketch (Appendix C.4: 'a complete analysis require tedious computations and is beyond the scope of this work'). These are gaps or overclaims rather than reductions of the claims to their inputs.

Assumptions & free parameters 1 free parameters · 8 assumptions · 0 invented entities

The proofs rely on standard random matrix and concentration tools plus several modeling assumptions: Gaussian data, random beta*, independent batches, and a local initialization regime for the linear representation theorem. No data-fitted free parameters or invented entities are introduced; the regularization lambda_G is an algorithm hyperparameter swept in experiments, not fitted. The main theorems are non-asymptotic or asymptotic with explicit constants.

free parameters (1)
  • lambda_G (ridge parameter in KFAC single-index update) = not fitted; swept in experiments, theory considers lambda_G -> 0
    Hyperparameter of the preconditioner QG = Sigma_x-hat + lambda_G I. The exact-recovery claim in Lemma 3.12 is in the limit lambda_G -> 0; no data fitting is used to select it.
assumptions (8)
  • domain assumption Gaussian data: x ~ N(0, Sigma_x), epsilon ~ N(0, Sigma_epsilon)
    Assumed throughout Section 3, model (6) and (12). Extensions to subgaussian are only sketched in Appendix E.3 and are not used in the main proofs.
  • domain assumption Assumption 3.8: beta* ~ N(0, d_X^-1 I), independent of covariates and noise
    Needed in Lemmas 3.10 and 3.12 to apply Hanson-Wright and to compute expectations of quadratic forms in beta*.
  • domain assumption Activations sigma and sigma* are O(1)-Lipschitz and differentiable enough for Stein's lemma and gradient computations
    Used in Theorem 3.9 and the correlation lemmas; differentiability is not stated explicitly but is required for the derivative-based decompositions.
  • domain assumption Independent batches / sample-splitting for G and F updates
    Stated in Section 3 before (5); required for the self-normalized martingale bounds in Lemmas B.3 and B.4 and to make F independent of the noise in the G-update.
  • ad hoc to paper Local convergence premise dist(G, G*) <= 0.01/(kappa(Sigma_x) kappa(F*)) in Theorem 3.6
    Assumption of the main linear-representation guarantee; the result does not cover global initialization.
  • domain assumption High-dimensional proportional limit n, d_X, d_h -> infinity with fixed ratios for single-index results
    Theorem 3.9 and Lemmas 3.10 and 3.12 are statements in this limit; finite-sample versions are not given.
  • standard math Standard concentration inequalities: Hanson-Wright, Gaussian covariance concentration, self-normalized martingale bounds
    Used throughout Appendices B and C; cited from Vershynin, Hsu et al., and Ziemann et al.
  • standard math Silverstein fixed-point equation for the Stieltjes transform of the sample covariance
    Used in Lemma 3.12 to express Psi_1, Psi_2, Psi_3 for anisotropic Sigma_x.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning." pith.science (2026). https://pith.science/paper/OZMXQ4JY

@misc{pith2026250201763,
  author       = {Pith},
  title        = {Pith review of: On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OZMXQ4JY}},
  note         = {Machine review of arXiv:2502.01763}
}
abstract

Layer-wise preconditioning methods are a family of memory-efficient optimization algorithms that introduce preconditioners per axis of each layer's weight tensors. These methods have seen a recent resurgence, demonstrating impressive performance relative to entry-wise ("diagonal") preconditioning methods such as Adam(W) on a wide range of neural network optimization tasks. Complementary to their practical performance, we demonstrate that layer-wise preconditioning methods are provably necessary from a statistical perspective. To showcase this, we consider two prototypical models, linear representation learning and single-index learning, which are widely used to study how typical algorithms efficiently learn useful features to enable generalization. In these problems, we show SGD is a suboptimal feature learner when extending beyond ideal isotropic inputs $\mathbf{x} \sim \mathsf{N}(\mathbf{0}, \mathbf{I})$ and well-conditioned settings typically assumed in prior work. We demonstrate theoretically and numerically that this suboptimality is fundamental, and that layer-wise preconditioning emerges naturally as the solution. We further show that standard tools like Adam preconditioning and batch-norm only mildly mitigate these issues, supporting the unique benefits of layer-wise preconditioning.

Figures

Figures reproduced from arXiv: 2502.01763 by the authors.

Figure 1
Figure 1. From left to right: the training loss, subspace distance, and transfer loss induced by various algorithms on a linear representation learning task. We note that various algorithms converge in training loss, but negligibly in subspace distance, and thus transfer loss. This lemma shows that when n ≫ dX, and λG → 0, the one-step stylized KFAC update—unlike the one-step full-batch SGD—perfectly recovers the target direc… view at source ↗
Figure 2
Figure 2. Subspace distance and the training loss of [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. The correlation of the direction learned by [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The effect of batch normalization (on AMGD) vs. KFAC in our experiment settings (Left) Uniform with low anisotropy. (Middle) Gaussian with low anisotropy. (Right) Gaussian with high anisotropy. As discussed in the main paper Section 3.1.1, we expect AMGD with batch-nor…
Figure 5
Figure 5. Figure 5: The subspace distance of representations learned by different algorithms after 1000 [PITH_FULL_IMAGE:figures/full_fig_p052_5.png]
Figure 6
Figure 6. Figure 6: Gaussian with low anisotropy 0 500 1000 1500 2000 Number of iterations k 10 0 2 × 10 0 3 × 10 0 4 × 10 0 6 × 10 0 Train Loss SGD Adam DFW AMGD KFAC NG 0 500 1000 1500 2000 Number of iterations k 0.25 0.50 0.75 1.00 Subspace Distance SGD Adam DFW AMGD KFAC NG 0 500 1000…
Figure 7
Figure 7. Figure 7: Gaussian with high anisotropy 0 500 1000 1500 2000 Number of iterations k 10 0 10 1 Train Loss SGD Adam DFW AMGD KFAC NG 0 500 1000 1500 2000 Number of iterations k 0.00 0.25 0.50 0.75 1.00 Subspace Distance SGD Adam DFW AMGD KFAC NG 0 500 1000 1500 2000 Number of iter…
Figure 8
Figure 8. Figure 8: Bernoulli with low anisotropy [PITH_FULL_IMAGE:figures/full_fig_p053_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reassessing Muon for Matrix Factorization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Muon's advantage over AdamW is problem-dependent: it loses or ties on plain low-rank factorization and completion but wins on nonnegative matrix factorization.

Reference graph

Works this paper leans on

124 extracted references · 50 canonical work pages · cited by 1 Pith paper

  1. [1]

    and Szepesv \'a ri, C

    Abbasi-Yadkori, Y. and Szepesv \'a ri, C. Regret bounds for the adaptive control of linear quadratic systems. In Conference on Learning Theory, 2011

    2011

  2. [2]

    B., and Misiakiewicz, T

    Abbe, E., Adsera, E. B., and Misiakiewicz, T. The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks. In Conference on Learning Theory, 2022

    2022

  3. [3]

    B., and Misiakiewicz, T

    Abbe, E., Adsera, E. B., and Misiakiewicz, T. SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics. In Conference on Learning Theory, 2023

    2023

  4. [4]

    Optimization algorithms on matrix manifolds

    Absil, P.-A., Mahony, R., and Sepulchre, R. Optimization algorithms on matrix manifolds. Princeton University Press, 2008

    2008

  5. [5]

    and Pennington, J

    Adlam, B. and Pennington, J. Understanding double descent requires a fine-grained bias-variance decomposition. In Advances in Neural Information Processing Systems, 2020

    2020

  6. [6]

    and Parrilo, P

    Altschuler, J. and Parrilo, P. Acceleration by stepsize hedging: Multi-step descent and the silver stepsize schedule. Journal of the ACM, 2024

    2024

  7. [7]

    When does preconditioning help or hurt generalization? arXiv preprint arXiv:2006.10732, 2020

    Amari, S.-i., Ba, J., Grosse, R., Li, X., Nitanda, A., Suzuki, T., Wu, D., and Xu, J. When does preconditioning help or hurt generalization? arXiv preprint arXiv:2006.10732, 2020

    arXiv 2006

  8. [8]

    Locoprop: Enhancing backprop via local loss optimization

    Amid, E., Anil, R., and Warmuth, M. Locoprop: Enhancing backprop via local loss optimization. In International Conference on Artificial Intelligence and Statistics, 2022

    2022

Show all 124 references
  1. [9]

    Scalable second order optimization for deep learning

    Anil, R., Gupta, V., Koren, T., Regan, K., and Singer, Y. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020

    2002 arXiv

  2. [10]

    Repetita iuvant: Data repetition allows SGD to learn high-dimensional multi-index functions

    Arnaboldi, L., Dandi, Y., Krzakala, F., Pesce, L., and Stephan, L. Repetita iuvant: Data repetition allows SGD to learn high-dimensional multi-index functions. arXiv preprint arXiv:2405.15459, 2024

    2024 arXiv

  3. [11]

    Implicit regularization in deep matrix factorization

    Arora, S., Cohen, N., Hu, W., and Luo, Y. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems, 2019

    2019

  4. [12]

    B., and Martens, J

    Ba, J., Grosse, R. B., and Martens, J. Distributed second-order optimization using Kronecker -factored approximations. In International Conference on Learning Representations, 2017

    2017

  5. [13]

    A., Suzuki, T., Wang, Z., Wu, D., and Yang, G

    Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., Wu, D., and Yang, G. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. In Advances in Neural Information Processing Systems, 2022

    2022

  6. [14]

    A., Suzuki, T., Wang, Z., and Wu, D

    Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., and Wu, D. Learning in the presence of low-dimensional structure: a spiked random matrix perspective. In Advances in Neural Information Processing Systems, 2024

    2024

  7. [15]

    Scaling laws of optimization, 2024

    Bach, F. Scaling laws of optimization, 2024. URL https://francisbach.com/scaling-laws-of-optimization/

    2024

  8. [16]

    and Lee, J

    Bai, Y. and Lee, J. D. Beyond linearization: On quadratic and higher-order approximation of wide neural networks. In International Conference on Learning Representations, 2020

    2020

  9. [17]

    Hidden progress in deep learning: SGD learns parities near the computational limit

    Barak, B., Edelman, B., Goel, S., Kakade, S., Malach, E., and Zhang, C. Hidden progress in deep learning: SGD learns parities near the computational limit. In Advances in Neural Information Processing Systems, 2022

    2022

  10. [18]

    Online stochastic gradient descent on non-convex losses from high-dimensional inference

    Ben Arous, G., Gheissari, R., and Jagannath, A. Online stochastic gradient descent on non-convex losses from high-dimensional inference. Journal of Machine Learning Research, 22 0 (106): 0 1--51, 2021

    2021

  11. [19]

    Gradient descent on neurons and its link to approximate second-order optimization

    Benzing, F. Gradient descent on neurons and its link to approximate second-order optimization. In International Conference on Machine Learning, 2022

    2022

  12. [20]

    and Newhouse, L

    Bernstein, J. and Newhouse, L. Modular duality in deep learning. arXiv preprint arXiv:2410.21265, 2024 a

    2024 arXiv

  13. [21]

    and Newhouse, L

    Bernstein, J. and Newhouse, L. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024 b

    2024 arXiv

  14. [22]

    Learning time-scales in two-layers neural networks

    Berthier, R., Montanari, A., and Zhou, K. Learning time-scales in two-layers neural networks. Foundations of Computational Mathematics, pp.\ 1--84, 2024

    2024

  15. [23]

    Bollapragada, R., Nocedal, J., Mudigere, D., Shi, H.-J., and Tang, P. T. P. A progressive batching l-bfgs method for machine learning. In International Conference on Machine Learning, 2018

    2018

  16. [24]

    and Mondelli, M

    Bombari, S. and Mondelli, M. How spurious features are memorized: Precise analysis for random and NTK features. In International Conference on Machine Learning, 2024 a

    2024

  17. [25]

    and Mondelli, M

    Bombari, S. and Mondelli, M. Privacy for free in the over-parameterized regime. arXiv preprint arXiv:2410.14787, 2024 b

    2024 arXiv

  18. [26]

    Beyond the universal law of robustness: Sharper laws for random features and neural tangent kernels

    Bombari, S., Kiyani, S., and Mondelli, M. Beyond the universal law of robustness: Sharper laws for random features and neural tangent kernels. In International Conference on Machine Learning, 2023

    2023

  19. [27]

    Practical Gauss- Netwon optimisation for deep learning

    Botev, A., Ritter, H., and Barber, D. Practical Gauss- Netwon optimisation for deep learning. In International Conference on Machine Learning, pp.\ 557--565, 2017

    2017

  20. [28]

    H., Hansen, S

    Byrd, R. H., Hansen, S. L., Nocedal, J., and Singer, Y. A stochastic quasi- Netwon method for large-scale optimization. SIAM Journal on Optimization, 26 0 (2): 0 1008--1031, 2016

    2016

  21. [29]

    Gram-Gauss- Netwon method: Learning overparameterized neural networks for regression problems

    Cai, T., Gao, R., Hou, J., Chen, S., Wang, D., He, D., Zhang, Z., and Wang, L. Gram-Gauss- Netwon method: Learning overparameterized neural networks for regression problems. arXiv preprint arXiv:1905.11675, 2019

    1905 arXiv

  22. [30]

    Exploiting shared representations for personalized federated learning

    Collins, L., Hassani, H., Mokhtari, A., and Shakkottai, S. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning, 2021

    2021

  23. [31]

    Provable multi-task representation learning by two-layer ReLU neural networks

    Collins, L., Hassani, H., Soltanolkotabi, M., Mokhtari, A., and Shakkottai, S. Provable multi-task representation learning by two-layer ReLU neural networks. In International Conference on Machine Learning, 2024

    2024

  24. [32]

    Asymptotics of feature learning in two-layer networks after one gradient-step

    Cui, H., Pesce, L., Dandi, Y., Krzakala, F., Lu, Y., Zdeborova, L., and Loureiro, B. Asymptotics of feature learning in two-layer networks after one gradient-step. In International Conference on Machine Learning, 2024

    2024

  25. [33]

    E., Schneider, F., Nado, Z., Agarwal, N., Sastry, C

    Dahl, G. E., Schneider, F., Nado, Z., Agarwal, N., Sastry, C. S., Hennig, P., Medapati, S., Eschenhagen, R., Kasimbeg, P., Suo, D., et al. Benchmarking neural network training algorithms. arXiv preprint arXiv:2306.07179, 2023

    2023 arXiv

  26. [34]

    Neural networks can learn representations with gradient descent

    Damian, A., Lee, J., and Soltanolkotabi, M. Neural networks can learn representations with gradient descent. In Conference on Learning Theory, 2022

    2022

  27. [35]

    How two-layer neural networks learn, one (giant) step at a time

    Dandi, Y., Krzakala, F., Loureiro, B., Pesce, L., and Stephan, L. How two-layer neural networks learn, one (giant) step at a time. Journal of Machine Learning Research, 25 0 (349): 0 1--65, 2024 a

    2024

  28. [36]

    M., and Loureiro, B

    Dandi, Y., Pesce, L., Cui, H., Krzakala, F., Lu, Y. M., and Loureiro, B. A random matrix theory perspective on the spectrum of learned features and asymptotic generalization capabilities. arXiv preprint arXiv:2410.18938, 2024 b

    arXiv

  29. [37]

    The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents

    Dandi, Y., Troiani, E., Arnaboldi, L., Pesce, L., Zdeborova, L., and Krzakala, F. The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents. In International Conference on Machine Learning, 2024 c

    2024

  30. [38]

    Modular block-diagonal curvature approximations for feedforward architectures

    Dangel, F., Harmeling, S., and Hennig, P. Modular block-diagonal curvature approximations for feedforward architectures. In International Conference on Artificial Intelligence and Statistics, 2020

    2020

  31. [39]

    Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates

    Dereich, S., Graeber, R., and Jentzen, A. Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates. arXiv preprint arXiv:2407.08100, 2024

    2024 arXiv

  32. [40]

    Dicker, L. H. Ridge regression and asymptotic minimax estimation over spheres of growing dimension. Bernoulli, pp.\ 1--37, 2016

    2016

  33. [41]

    and Wager, S

    Dobriban, E. and Wager, S. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics, 46 0 (1): 0 247--279, 2018

    2018

  34. [42]

    S., Hu, W., Kakade, S

    Du, S. S., Hu, W., Kakade, S. M., Lee, J. D., and Lei, Q. Few-shot learning via learning the representation, provably. In International Conference on Learning Representations, 2021

    2021

  35. [43]

    Adaptive subgradient methods for online learning and stochastic optimization

    Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12 0 (7), 2011

    2011

  36. [44]

    Proximal backpropagation

    Frerix, T., M \"o llenhoff, T., Moeller, M., and Cremers, D. Proximal backpropagation. In International Conference on Learning Representations, 2018

    2018

  37. [45]

    Fu, H., Wang, Z., Nichani, E., and Lee, J. D. Learning hierarchical polynomials of multiple nonlinear features with three-layer networks. arXiv preprint arXiv:2411.17201, 2024

    2024 arXiv

  38. [46]

    Linearized two-layers neural networks in high dimension

    Ghorbani, B., Mei, S., Misiakiewicz, T., and Montanari, A. Linearized two-layers neural networks in high dimension. The Annals of Statistics, 49 0 (2): 0 1029--1054, 2021 a

    2021

  39. [47]

    When do neural networks outperform kernel methods? Journal of Statistical Mechanics: Theory and Experiment, 2021 0 (12), 2021 b

    Ghorbani, B., Mei, S., Misiakiewicz, T., and Montanari, A. When do neural networks outperform kernel methods? Journal of Statistical Mechanics: Theory and Experiment, 2021 0 (12), 2021 b

    2021

  40. [48]

    A family of variable-metric methods derived by variational means

    Goldfarb, D. A family of variable-metric methods derived by variational means. Mathematics of computation, 24 0 (109): 0 23--26, 1970

    1970

  41. [49]

    Practical quasi- Netwon methods for training deep neural networks

    Goldfarb, D., Ren, Y., and Bahamou, A. Practical quasi- Netwon methods for training deep neural networks. In Advances in Neural Information Processing Systems, 2020

    2020

  42. [50]

    The G aussian equivalence of generative models for learning with shallow neural networks

    Goldt, S., Loureiro, B., Reeves, G., Krzakala, F., M \'e zard, M., and Zdeborov \'a , L. The G aussian equivalence of generative models for learning with shallow neural networks. In Mathematical and Scientific Machine Learning, pp.\ 426--471, 2022

    2022

  43. [51]

    Spectral phase transitions in non-linear wigner spiked models

    Guionnet, A., Ko, J., Krzakala, F., Mergny, P., and Zdeborov \'a , L. Spectral phase transitions in non-linear wigner spiked models. arXiv preprint arXiv:2310.14055, 2023

    2023 arXiv

  44. [52]

    Gupta, A., Ramanath, R., Shi, J., and Keerthi, S. S. Adam vs. SGD : Closing the generalization gap on image classification. In OPT2021: 13th Annual Workshop on Optimization for Machine Learning, 2021

    2021

  45. [53]

    Shampoo: Preconditioned stochastic tensor optimization

    Gupta, V., Koren, T., and Singer, Y. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, 2018

    2018

  46. [54]

    and Nica, M

    Hanin, B. and Nica, M. Finite depth and width corrections to the neural tangent kernel. In International Conference on Learning Representations, 2020

    2020

  47. [55]

    Hanson, D. L. and Wright, F. T. A bound on tail probabilities for quadratic forms in independent random variables. The Annals of Mathematical Statistics, 42 0 (3): 0 1079--1083, 1971

    1971

  48. [56]

    and Javanmard, A

    Hassani, H. and Javanmard, A. The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regression. The Annals of Statistics, 52 0 (2): 0 441--465, 2024

    2024

  49. [57]

    Horn, R. A. and Johnson, C. R. Matrix analysis. Cambridge university press, 2012

    2012

  50. [58]

    M., and Zhang, T

    Hsu, D., Kakade, S. M., and Zhang, T. Random design analysis of ridge regression. In Conference on Learning Theory, 2012

    2012

  51. [59]

    and Lu, Y

    Hu, H. and Lu, Y. M. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 69 0 (3), 2023

    2023

  52. [60]

    and Szegedy, C

    Ioffe, S. and Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, pp.\ 448–456. JMLR.org, 2015

    2015

  53. [61]

    and Karakida, R

    Ishikawa, S. and Karakida, R. On the parameterization of second-order optimization effective towards the infinite width. arXiv preprint arXiv:2312.12226, 2023

    2023 arXiv

  54. [62]

    Neural tangent kernel: Convergence and generalization in neural networks

    Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, 2018

    2018

  55. [63]

    Low-rank matrix completion using alternating minimization

    Jain, P., Netrapalli, P., and Sanghavi, S. Low-rank matrix completion using alternating minimization. In ACM Symposium on Theory of Computing, pp.\ 665--674, 2013

    2013

  56. [64]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Jordan, K., Jin, Y., Boza, V., Jiacheng, Y., Cecista, F., Newhouse, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/

    2024

  57. [65]

    Keskar, N. S. and Socher, R. Improving generalization performance by switching from Adam to SGD . arXiv preprint arXiv:1712.07628, 2017

    2017 arXiv

  58. [66]

    Kingma, D. P. and Ba, J. Adam : A method for stochastic optimization. In International Conference on Learning Representations, 2015

    2015

  59. [67]

    M., Ma, T., and Liang, P

    Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022

    2022

  60. [68]

    Scalable optimization in the modular norm

    Large, T., Liu, Y., Huh, M., Bahng, H., Isola, P., and Bernstein, J. Scalable optimization in the modular norm. arXiv preprint arXiv:2405.14813, 2024

    2024 arXiv

  61. [69]

    and Massart, P

    Laurent, B. and Massart, P. Adaptive estimation of a quadratic functional by model selection. The Annals of Statistics, pp.\ 1302--1338, 2000

    2000

  62. [70]

    Demystifying disagreement-on-the-line in high dimensions

    Lee, D., Moniri, B., Huang, X., Dobriban, E., and Hassani, H. Demystifying disagreement-on-the-line in high dimensions. In International Conference on Machine Learning, 2023 a

    2023

  63. [71]

    D., Oko, K., Suzuki, T., and Wu, D

    Lee, J. D., Oko, K., Suzuki, T., and Wu, D. Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit. arXiv preprint arXiv:2406.01581, 2024

    2024 arXiv

  64. [72]

    S., Tajwar, F., Kumar, A., Yao, H., Liang, P., and Finn, C

    Lee, Y., Chen, A. S., Tajwar, F., Kumar, A., Yao, H., Liang, P., and Finn, C. Surgical fine-tuning improves adaptation to distribution shifts. In International Conference on Learning Representations, 2023 b

    2023

  65. [73]

    and Dobriban, E

    Lin, L. and Dobriban, E. What causes the test error? going beyond bias-variance via ANOVA . Journal of Machine Learning Research, 22: 0 155--1, 2021

    2021

  66. [74]

    E., and Makhzani, A

    Lin, W., Dangel, F., Eschenhagen, R., Bae, J., Turner, R. E., and Makhzani, A. Can we remove the square-root in adaptive gradient methods? a second-order perspective. In Forty-first International Conference on Machine Learning, 2024

    2024

  67. [75]

    Liu, D. C. and Nocedal, J. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45 0 (1): 0 503--528, 1989

    1989

  68. [76]

    Deep learning via Hessian -free optimization

    Martens, J. Deep learning via Hessian -free optimization. In International Conference on Machine Learning, 2010

    2010

  69. [77]

    New insights and perspectives on the natural gradient method

    Martens, J. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 0 (146): 0 1--76, 2020

    2020

  70. [78]

    and Grosse, R

    Martens, J. and Grosse, R. Optimizing neural networks with Kronecker -factored approximate curvature. In International Conference on Machine Learning, 2015

    2015

  71. [79]

    The benefit of multitask representation learning

    Maurer, A., Pontil, M., and Romera-Paredes, B. The benefit of multitask representation learning. Journal of Machine Learning Research, 17 0 (81): 0 1--32, 2016

    2016

  72. [80]

    and Montanari, A

    Mei, S. and Montanari, A. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75 0 (4): 0 667--766, 2022

    2022

  73. [81]

    Announcing the results of the inaugural AlgoPerf : Training algorithms benchmark competition, 2024

    MLCommons. Announcing the results of the inaugural AlgoPerf : Training algorithms benchmark competition, 2024. URL https://mlcommons.org/2024/08/mlc-algoperf-benchmark-competition/

    2024

  74. [82]

    and Hassani, H

    Moniri, B. and Hassani, H. Asymptotics of linear regression with linearly dependent data. arXiv preprint arXiv:2412.03702, 2024 a

    2024 arXiv

  75. [83]

    and Hassani, H

    Moniri, B. and Hassani, H. Signal-plus-noise decomposition of nonlinear spiked random matrix models. arXiv preprint arXiv:2405.18274, 2024 b

    2024 arXiv

  76. [84]

    A theory of non-linear feature learning with one gradient step in two-layer neural networks

    Moniri, B., Lee, D., Hassani, H., and Dobriban, E. A theory of non-linear feature learning with one gradient step in two-layer neural networks. In International Conference on Machine Learning, 2024

    2024

  77. [85]

    The generalization error of max-margin linear classifiers: High-dimensional asymptotics in the overparametrized regime

    Montanari, A., Ruan, F., Sohn, Y., and Yan, J. The generalization error of max-margin linear classifiers: High-dimensional asymptotics in the overparametrized regime. arXiv preprint arXiv:1911.01544, 2019

    1911 arXiv

  78. [86]

    A new perspective on shampoo's preconditioner

    Morwani, D., Shapira, I., Vyas, N., Malach, E., Kakade, S., and Janson, L. A new perspective on shampoo's preconditioner. arXiv preprint arXiv:2406.17748, 2024

    2024 arXiv

  79. [87]

    Mousavi-Hosseini, A., Wu, D., Suzuki, T., and Erdogdu, M. A. Gradient-based feature learning under structured data. In Advances in Neural Information Processing Systems, 2023

    2023

  80. [88]

    Nakhleh, J., Shenouda, J., and Nowak, R. D. The effects of multi-task learning on ReLU neural network functions. arXiv preprint arXiv:2410.21696, 2024

    2024 arXiv

  81. [89]

    and Vaswani, N

    Nayer, S. and Vaswani, N. Fast and sample-efficient federated low rank matrix recovery from column-wise linear and quadratic projections. IEEE Transactions on Information Theory, 69 0 (2): 0 1177--1202, 2022

    2022

  82. [90]

    Nichani, E., Damian, A., and Lee, J. D. Provable guarantees for nonlinear feature learning in three-layer neural networks. In Advances in Neural Information Processing Systems, 2024 a

    2024

  83. [91]

    Nichani, E., Damian, A., and Lee, J. D. How transformers learn causal structure with gradient descent. In International Conference on Machine Learning, 2024 b

    2024

  84. [92]

    and Wright, S

    Nocedal, J. and Wright, S. J. Numerical optimization. Springer, 1999

    1999

  85. [93]

    Pearlmutter, B. A. Fast exact multiplication by the hessian. Neural computation, 6 0 (1): 0 147--160, 1994

    1994

  86. [94]

    and Recht, B

    Rahimi, A. and Recht, B. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, 2007

    2007

  87. [95]

    J., Kale, S., and Kumar, S

    Reddi, S. J., Kale, S., and Kumar, S. On the convergence of Adam and beyond. In International Conference on Learning Representations, 2018

    2018

  88. [96]

    and Vershynin, R

    Rudelson, M. and Vershynin, R. Hanson-wright inequality and sub-gaussian concentration. Electronic Communications in Probability, 18: 0 1--9, 2013

    2013

  89. [97]

    M., Schneider, F., and Hennig, P

    Schmidt, R. M., Schneider, F., and Hennig, P. Descending through a crowded valley-benchmarking deep learning optimizers. In International Conference on Machine Learning, 2021

    2021

  90. [98]

    Schraudolph, N. N. Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14 0 (7): 0 1723--1738, 2002

    2002

  91. [99]

    M., Lee, T.-H., Iwasaki, S., Gallego-Posada, J., Li, Z., Rangadurai, K., Mudigere, D., and Rabbat, M

    Shi, H.-J. M., Lee, T.-H., Iwasaki, S., Gallego-Posada, J., Li, Z., Rangadurai, K., Mudigere, D., and Rabbat, M. A distributed data-parallel pytorch implementation of the distributed Shampoo optimizer for training neural networks at-scale. arXiv preprint arXiv:2309.06497, 2023

    2023 arXiv

  92. [100]

    A theoretical analysis on feature learning in neural networks: Emergence from inputs and advantage over fixed features

    Shi, Z., Wei, J., and Liang, Y. A theoretical analysis on feature learning in neural networks: Emergence from inputs and advantage over fixed features. In International Conference on Learning Representations, 2022

    2022

  93. [101]

    Silverstein, J. W. and Choi, S.-I. Analysis of the limiting spectral distribution of large dimensional random matrices. Journal of Multivariate Analysis, 54 0 (2): 0 295--309, 1995

    1995

  94. [102]

    Stewart, G. W. and Sun, J.-g. Matrix perturbation theory. Academic press, 1990

    1990

  95. [103]

    K., Jain, P., Netrapalli, P., and Oh, S

    Thekumparampil, K. K., Jain, P., Netrapalli, P., and Oh, S. Sample efficient linear meta-learning by alternating minimization. arXiv preprint arXiv:2105.08306, 2021

    2021 arXiv

  96. [104]

    and Hinton, G

    Tieleman, T. and Hinton, G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. Coursera: Neural Networks for Machine Learning, 4 0 (2): 0 26, 2012

    2012

  97. [105]

    Trefethen, L. N. and Bau, D. Numerical linear algebra, volume 181. SIAM, 2022

    2022

  98. [106]

    On the theory of transfer learning: The importance of task diversity

    Tripuraneni, N., Jordan, M., and Jin, C. On the theory of transfer learning: The importance of task diversity. In Advances in Neural Information Processing Systems, 2020

    2020

  99. [107]

    Overparameterization improves robustness to covariate shift in high dimensions

    Tripuraneni, N., Adlam, B., and Pennington, J. Overparameterization improves robustness to covariate shift in high dimensions. In Advances in Neural Information Processing Systems, 2021 a

    2021

  100. [108]

    Provable meta-learning of linear representations

    Tripuraneni, N., Jin, C., and Jordan, M. Provable meta-learning of linear representations. In International Conference on Machine Learning, 2021 b

    2021

  101. [109]

    Fundamental limits of weak learnability in high-dimensional multi-index models

    Troiani, E., Dandi, Y., Defilippis, L., Zdeborov \'a , L., Loureiro, B., and Krzakala, F. Fundamental limits of weak learnability in high-dimensional multi-index models. arXiv preprint arXiv:2405.15480, 2024

    2024

  102. [110]

    Efficient federated low rank matrix recovery via alternating GD and minimization: A simple proof

    Vaswani, N. Efficient federated low rank matrix recovery via alternating GD and minimization: A simple proof. IEEE Transactions on Information Theory, 2024

    2024

  103. [111]

    Introduction to the non-asymptotic analysis of random matrices

    Vershynin, R. Introduction to the non-asymptotic analysis of random matrices. In Eldar, Y. C. and Kutyniok, G. (eds.), Compressed Sensing: Theory and Applications, pp.\ 210–268. Cambridge University Press, 2012

    2012

  104. [112]

    High-dimensional probability: An introduction with applications in data science, volume 47

    Vershynin, R. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge University Press, 2018

    2018

  105. [113]

    Vyas, N., Morwani, D., Zhao, R., Shapira, I., Brandfonbrener, D., Janson, L., and Kakade, S. M. Soap: Improving and stabilizing Shampoo using Adam . In OPT 2024: Optimization for Machine Learning, 2024

    2024

  106. [114]

    Wainwright, M. J. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019

    2019

  107. [115]

    Wang, Z., Nichani, E., and Lee, J. D. Learning hierarchical polynomials with three-layer neural networks. In International Conference on Learning Representations, 2024

    2024

  108. [116]

    C., Roelofs, R., Stern, M., Srebro, N., and Recht, B

    Wilson, A. C., Roelofs, R., Stern, M., Srebro, N., and Recht, B. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, 2017

    2017

  109. [117]

    Woodbury, M. A. Inverting modified matrices. Department of Statistics, Princeton University, 1950

    1950

  110. [118]

    Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum

    Xie, Z., Wang, X., Zhang, H., Sato, I., and Sugiyama, M. Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum. In International Conference on Machine Learning, 2022

    2022

  111. [119]

    and Hu, E

    Yang, G. and Hu, E. J. Feature learning in infinite-width neural networks. In International Conference on Machine Learning, 2021 a

    2021

  112. [120]

    and Hu, E

    Yang, G. and Hu, E. J. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, 2021 b

    2021

  113. [121]

    Zhang, G., Martens, J., and Grosse, R. B. Fast convergence of natural gradient descent for over-parameterized neural networks. In Advances in Neural Information Processing Systems, 2019

    2019

  114. [122]

    T., Lee, B

    Zhang, T. T., Lee, B. D., Ziemann, I., Pappas, G. J., and Matni, N. Guarantees for nonlinear representation learning: Non-identical covariates, dependent data, fewer samples. In International Conference on Machine Learning, 2024 a

    2024

  115. [123]

    T., Toso, L

    Zhang, T. T., Toso, L. F., Anderson, J., and Matni, N. Sample-efficient linear representation learning from non- IID non-isotropic data. In International Conference on Learning Representations, 2024 b

    2024

  116. [124]

    Ziemann, I., Tsiamis, A., Lee, B., Jedra, Y., Matni, N., and Pappas, G. J. A tutorial on the non-asymptotic theory of system identification. In IEEE Conference on Decision and Control, 2023

    2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.