Pith. sign in

REVIEW 2 major objections 4 minor 58 references

A Sketch-and-Project Analysis of Subsampled Natural Gradient Algorithms

T0 review · 2 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Subsampled natural gradient descent is, on linear least-squares problems, exactly regularized Kaczmarz; its momentum variant SPRING is accelerated Kaczmarz — yielding the first fast convergence rates for both.

desk verdict The LLS equivalences and first SPRING guarantee are solid; the LLQ rate is new but rests on an alignment assumption the authors admit is unrealistic, and the stress-test's alleged λ=0 slip is wrong on reading. read the letter →

arxiv 2508.21022 v3 pith:EWGTRBZA submitted 2025-08-28 cs.LG math.OCstat.ML

Gil Goldshlager , Jiang Hu , Lin Lin This is my paper · ORCID
classification cs.LGmath.OCstat.ML MSC 65F1068W2065K05
keywords subsamplednaturalgradientdescentKaczmarzmethodsketch-and-projectSPRINGmomentumconvergenceratelinearleast-squaresscientificmachinelearningrandomizedalgebra
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Subsampled natural gradient descent (SNGD) and its momentum variant SPRING are behind recent successes in scientific machine learning — finding ground states of quantum systems and solving PDEs with neural networks — yet until now no analysis explained how they converge in the small-mini-batch regime that makes them practical. This paper closes that gap by proving that, for linear least-squares problems, SNGD is exactly a regularized Kaczmarz method and SPRING is exactly its Nesterov-accelerated counterpart, so existing randomized linear algebra theory applies directly. From that identification the paper obtains global convergence rates that hold for any batch size, the first convergence guarantee for SPRING in any setting, and the first proof that SPRING accelerates SNGD, by up to a square-root improvement in the error contraction rate. It also explains why SNGD beats SGD: the rate improves with the rate of spectral decay of the model Jacobian, which SGD cannot exploit. For the more general quadratic losses that model energy-based neural-network training, the paper proves a fast rate under a strong consistency condition, and shows that with too little regularization SNGD can diverge for every step size.

What carries the argument

The central object is the random regularized projector P(S) = J_S^+ J_S, which maps vectors onto the row space of the sampled Jacobian rows. Bundling all stochasticity into P(S) — instead of separating the random gradient from the random preconditioner as earlier analyses did — turns the SNGD update into (I − ηP(S)) acting on the error, a contraction in expectation since P(S)² ⪯ P(S). This lets the paper import two ready-made analyses: regularized Kaczmarz for SNGD, and its accelerated version (ARK) for SPRING. For the LLQ extension, the new identity J_S^+ H J = P(S) J^+ H J, valid when range(HJ) ⊆ range(J), expresses the update as a random projector applied to a deterministic natural-gradie

What would settle it

Compute the spectrum of M = P J^+ H J for a realistic model Jacobian at a chosen regularization λ: if min Re(eig M) < 0, Proposition 3 predicts SNGD diverges for every step size, a prediction testable by running SNGD from a random initial point. Independently, test whether the expected projector P commutes with J^T J on a real neural-network Jacobian from a PINN or wavefunction model; if it does not, Theorem 4's conditions fail and its predicted rate (1 − κ⁻¹(H̃)αγ)^t should be checked empirically to see whether fast convergence persists outside the assumption.

Watch

Extended reading notes

Core claim

On consistent linear least-squares problems (Jθ* = b), SNGD is identical to regularized Kaczmarz iteration, and its momentum variant SPRING is identical to the accelerated regularized Kaczmarz (ARK) method. Using the same mini-batch for gradient and preconditioner bundles all randomness into one regularized projector P(S) = J_S^+ J_S, so the expected error shrinks as (1 − α)^t for SNGD and O((1 − √(α/β))^t) for SPRING — a square-root speed-up. These give the first fast rates for SNGD, the first convergence guarantee for SPRING, and the first proof SPRING accelerates SNGD. SNGD beats SGD by turning Jacobian spectral decay into acceleration. For the general LLQ loss, a similar rate holds only

Load-bearing premise

The linear least-squares results assume the system is consistent (an exact solution exists), and the extension to general quadratic losses additionally assumes the subsampled projector and the model's Gram matrix can be diagonalized using the same eigenvectors — a condition the paper itself says is not expected to hold in practice.

Editorial extensions

If this is right

  • First fast convergence rates for SNGD on linear least-squares: expected squared error decays as (1 − α)^t for any batch size, not only asymptotically large ones.
  • First convergence guarantee for SPRING in any setting, plus the first proof that SPRING accelerates SNGD, by up to a square-root improvement in the rate.
  • Explains the empirical advantage of SNGD over SGD: the rate improves with the spectral decay exponent β of the Jacobian (k^β versus k), which matters for problems like PINNs whose Jacobians have rapidly decaying singular values.
  • Explains why using a single mini-batch for both the stochastic gradient and the stochastic preconditioner is beneficial, and why SPRING's gains are largest for small batch sizes.
  • For the LLQ losses that model energy-based wavefunction training, SNGD can diverge for every step size if the regularization λ is too small; a sufficiently large λ guarantees convergence, giving a new reason why λ > 0 is essential in practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the SNGD–Kaczmarz identity is algebraic, the transfer is not limited to convergence rates — improvements from the Kaczmarz literature, such as non-uniform row sampling or tail averaging, could be ported into SNGD/SPRING implementations; the paper itself only analyzes uniform sampling.
  • Editorial inference: the LLQ theorem's eigen-alignment condition is the point where theory and practice could diverge; checking whether P and J^T J share an eigenbasis on a real network Jacobian from a PINN or wavefunction model would reveal how wide that gap is. The paper's numerical experiments use Gaussian Jacobians, where the alignment holds only asymptotically, so they do not probe it.
  • Editorial inference: the central coupling insight — one mini-batch shared by gradient and curvature estimate collapsing into a single random projector — should carry over to subsampled Newton and Gauss-Newton methods, where the identical two-estimator coupling problem arises and a parallel contraction argument may be constructible.
  • Editorial inference: Proposition 3 yields a practical diagnostic: compute ξ(λ) = min Re eig(P J^+ H J) on a small surrogate model; its sign predicts guaranteed divergence (negative) versus guaranteed convergence (positive) for SNGD, giving a principled rule for choosing λ in energy-based training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper analyzes subsampled natural gradient descent (SNGD) and its momentum variant SPRING on two quadratic model problems: linear least-squares (LLS) and linear least-quadratics (LLQ). For consistent LLS, it proves that SNGD with unit step size is exactly the regularized Kaczmarz method of Goldshlager et al. (2025) and that SPRING is equivalent to the accelerated regularized Kaczmarz (ARK) method of Dereziński et al. (2025), leading to the stated geometric convergence rates. For LLQ, under a strong consistency condition plus an added assumption that the expected regularized projector commutes with JᵀJ, the paper proves a convergence rate for SNGD, and it shows that without the commutation assumption SNGD can diverge for all step sizes when the regularization is small. SPRING for LLQ is left as a conjecture with supporting numerics. The paper frames these results as the first convergence guarantees for SNGD and SPRING and the first rigorous explanation of SPRING's acceleration.

Significance. If the LLS results are correct, they provide a clean and useful bridge between subsampled natural gradient methods and randomized linear algebra: the equivalence proofs are explicit, the rates follow from established Kaczmarz theory, and the single-batch coupling is handled naturally rather than by decoupling gradient and preconditioner samples. The SPRING-ARK equivalence is a genuine new structural insight and gives the first convergence statement for SPRING. The LLQ analysis is more conditional: the main theorem requires an eigenbasis alignment assumption that the authors themselves state is not expected to hold in practice, and Proposition 3 shows that without such an assumption divergence is possible. Nevertheless, the paper is transparent about this limitation and provides concrete special cases (DPP sampling and Gaussian limits) where the alignment holds. Overall, the LLS contribution is solid, while the LLQ contribution is a plausible starting point rather than a general explanation.

major comments (2)
  1. [Section 4.2, Theorem 4] The claimed 'first explanation for SNGD outside least-squares' rests on the assumption that the expected regularized projector P shares an eigenbasis with JᵀJ. The paper explicitly says this assumption 'is not expected to hold in practice,' and Proposition 3 shows that when it fails, SNGD can diverge for every step size at small λ. Thus Theorem 4 establishes convergence only under an idealized alignment condition (or in the DPP/Gaussian special cases of Proposition 9), not for the motivating NNW/PINN settings. The contribution should be reframed accordingly, or the authors should provide evidence or a relaxed condition under which approximate alignment holds in the motivating applications.
  2. [Theorem 1, eq. (16)] The SPRING rate is stated as holding 'for appropriate choices of η and μ,' but the paper never specifies admissible values or ranges for these hyperparameters, nor the corresponding constraints on the ARK step-size transformation \ ilde η = 1-(1-η)/μ given in Appendix A. Since one of the main contributions is the first convergence guarantee for SPRING, the guarantee is not actionable without explicit hyperparameter conditions. Please include the concrete ARK assumptions (e.g., from Dereziński et al. 2025) under which eq. (16) holds, or state a fully self-contained theorem.
minor comments (4)
  1. [Notation, Sections 2-4] The symbol J⁺ is used for both the regularized pseudoinverse J⁺(λ) and the unregularized pseudoinverse; for example, eq. (18) introduces J⁺ without a definition. Please define J⁺ = (JᵀJ)⁻¹Jᵀ explicitly.
  2. [Appendix B, proof of Theorem 4] The proof uses Q for both the deterministic matrix in Theorem 4 and the random matrix P(S)(JᵀJ)⁻¹. This makes equations (38)-(40) hard to read. Use notation such as \bar Q for the deterministic expectation and Q_S for the random quantity. Also note: the apparent replacement of a regularized pseudoinverse by an unregularized one in eqs. (38)-(39) is not an error, because J⁺ there is the full pseudoinverse from eq. (18) and the regularization enters only through P(S); clarifying this would prevent confusion.
  3. [Corollary 5] The asymptotic statements contain typographical issues: 'α = Ω(kβκ−2_dem(J)' is missing a closing parenthesis, and 'γ = Ω(κ2(J))' should presumably be 'γ = Ω(κ⁻²(J))' to be consistent with the rate in eq. (21). Please correct.
  4. [Appendix A, ARK transformation] After deriving \tilde η = 1-(1-η)/μ, the paper does not discuss the admissible sign or magnitude of \tilde η. Since ARK requires certain step-size constraints, state the conditions on η and μ that make \tilde η admissible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SNGD/SPRING equivalences are proven algebraically and the imported rates come from independent, non-fitted prior analyses.

full rationale

The derivation chain is non-circular. For LLS, the paper rewrites the SNGD update (eq. 11) under consistency Jθ*=b as eq. (13)-(14), which is exactly the regularized Kaczmarz update; the expectation bound (15) is then re-derived in Appendix A using P(S)^2 ⪯ P(S), not assumed from the theorem being proved. The SPRING/ARK equivalence is verified by an explicit induction in Appendix A, and the SPRING rate (16) is cited from external ARK analyses (Dereziński et al. 2024, 2025), not from any fit or from a conclusion built into the assumptions. The LLQ analysis is new: Assumption 2 permits the exact rewrite in eq. (18), and Theorem 4's proof is self-contained given the stated commutation assumption. The Appendix B proof's use of the unregularized pseudoinverse of the full J in the deterministic NGD direction is not a silent λ=0 restriction: λ enters through P(S), so eqs. (38)-(39) are exact for λ>0. The paper's own admission that the eigenbasis assumption 'is not expected to hold in practice' is an honest scope limitation rather than a circular move, and Proposition 3 exposes a real fragility at small λ. No fitted parameter is relabeled as a prediction, and the same-author citation (Goldshlager et al. 2025) transfers an independently statable regularized-Kaczmarz theorem after an exact equivalence; it does not make the central claim equivalent to its own input by construction.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central LLS results rest on consistency and on importing the regularized Kaczmarz and ARK rates from prior work. The LLQ results rest on the strong consistency Assumption 2 and an eigenbasis alignment condition that the paper itself calls unrealistic. No parameters are fitted to data; step sizes are given in closed form except for the unspecified appropriate choices in the SPRING statement.

assumptions (6)
  • domain assumption Model is linear, v_theta = J theta, with Hilbert space R^m and J of full column rank (Problem 1).
    All results are for this idealized parametric optimization model, not general neural networks.
  • domain assumption Consistency for LLS: J theta* = b, meaning the linear system is consistent.
    Used to rewrite SNGD as (I - P(S))(theta_t - theta*) in Section 3; an interpolation-style assumption.
  • ad hoc to paper Assumption 2 strong consistency: J theta* = argmin_v L(v) and range(HJ) subset of range(J).
    Introduced to ensure HJ theta + q lies in range(J) so the SNGD update becomes a regularized projector applied to the deterministic NGD direction (Section 4).
  • domain assumption Uniform sampling of mini-batches S without replacement.
    Assumed in Section 2.4; needed to define P = E[P(S)] and to compare with Gaussian sketch results.
  • ad hoc to paper P shares an eigenbasis with J^T J (for Theorem 4).
    The authors state it is not expected to hold in practice, justifying it only via DPP sampling or Gaussian asymptotics (Proposition 9).
  • standard math External convergence analyses of regularized Kaczmarz (Goldshlager et al. 2025) and accelerated regularized Kaczmarz (Dereziński et al. 2024, 2025) are correct.
    The SPRING rate and the SNGD-LSS rate are imported from these prior results after the equivalence proofs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Sketch-and-Project Analysis of Subsampled Natural Gradient Algorithms." pith.science (2026). https://pith.science/paper/EWGTRBZA

@misc{pith2026250821022,
  author       = {Pith},
  title        = {Pith review of: A Sketch-and-Project Analysis of Subsampled Natural Gradient Algorithms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EWGTRBZA}},
  note         = {Machine review of arXiv:2508.21022}
}
read the original abstract

Subsampled natural gradient descent (SNG) has been used to enable high-precision scientific machine learning, but standard analyses based on stochastic preconditioning fail to provide insight into realistic small-sample settings. We overcome this limitation by instead analyzing SNG as a sketch-and-project method. Motivated by this lens, we discard the usual theoretical proxy which decouples gradients and preconditioners using two independent mini-batches, and we replace it with a new proxy based on squared volume sampling. Under this new proxy we show that the expectation of the SNG direction becomes equal to a preconditioned gradient descent step even in the presence of coupling, leading to (i) global convergence guarantees when using a single mini-batch of any size, and (ii) an explicit characterization of the convergence rate in terms of quantities related to the sketch-and-project structure. These findings in turn yield new insights into small-sample settings, for example by suggesting that the advantage of SNG over SGD is that it can more effectively exploit spectral decay in the model Jacobian. We also extend these ideas to explain a popular structured momentum scheme for SNG, known as SPRING, by showing that it arises naturally from accelerated sketch-and-project methods.

Figures

Figures reproduced from arXiv: 2508.21022 by the authors.

Figure 1
Figure 1. SGD, SNGD, and SPRING for two randomly generated difficult instances of Problem 1. [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Eigenvalues of M for randomly generated problems of two sizes. In both cases k = 1 and λ = 0. The imaginary axis is painted in blue in both pictures, eigenvalues with positive real parts are painted in green, and eigenvalues with negative real parts are painted in red. Eigenvalues are displayed for 1, 000 randomly generated problems in each panel [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗
Figure 3
Figure 3. Eigenvalues of M for randomly generated problems with two different values of λ. In both cases (m, n, k) = (4, 2, 1). The imaginary axis is painted in blue in both pictures, eigenvalues with positive real parts are painted in green, and eigenvalues with negative real parts are painted in red. Eigenvalues are displayed for 1, 000 randomly generated problems in each panel. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: SGD, SNGD, and SPRING for a randomly generated instance of Problem 1 with varying [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 31 canonical work pages

  1. [1]

    Convergence of variational M onte C arlo simulation and scale-invariant pre-training

    Nilin Abrahamsen, Zhiyan Ding, Gil Goldshlager, and Lin Lin. Convergence of variational M onte C arlo simulation and scale-invariant pre-training. Journal of Computational Physics, 513: 0 113140, 2024

    work page 2024

  2. [2]

    Natural gradient works efficiently in learning

    Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10 0 (2): 0 251--276, 1998

    work page 1998

  3. [3]

    Functional neural wavefunction optimization

    Victor Armegioiu, Juan Carrasquilla, Siddhartha Mishra, Johannes M \"u ller, Jannes Nys, Marius Zeinhofer, and Hang Zhang. Functional neural wavefunction optimization. arXiv preprint arXiv:2507.10835, 2025

    arXiv 2025

  4. [4]

    Stochastic (approximate) proximal point methods: Convergence, optimality, and adaptivity

    Hilal Asi and John C Duchi. Stochastic (approximate) proximal point methods: Convergence, optimality, and adaptivity. SIAM Journal on Optimization, 29 0 (3): 0 2257--2290, 2019

    work page 2019

  5. [5]

    Incremental proximal methods for large scale convex optimization

    Dimitri P Bertsekas. Incremental proximal methods for large scale convex optimization. Mathematical programming, 129 0 (2): 0 163--195, 2011

    2011

  6. [6]

    Exact and inexact subsampled N ewton methods for optimization

    Raghu Bollapragada, Richard H Byrd, and Jorge Nocedal. Exact and inexact subsampled N ewton methods for optimization. IMA Journal of Numerical Analysis, 39 0 (2): 0 545--578, 2019

    work page 2019

  7. [7]

    Optimization methods for large-scale machine learning

    L \'e on Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60 0 (2): 0 223--311, 2018

    2018

  8. [8]

    Learning the ground state of a non-stoquastic quantum H amiltonian in a rugged neural network landscape

    Marin Bukov, Markus Schmitt, and Maxime Dupont. Learning the ground state of a non-stoquastic quantum H amiltonian in a rugged neural network landscape. SciPost Physics, 10 0 (6): 0 147, 2021

    work page 2021

Show all 58 references
  1. [9]

    Gram- G auss- N ewton method: Learning overparameterized neural networks for regression problems

    Tianle Cai, Ruiqi Gao, Jikai Hou, Siyu Chen, Dong Wang, Di He, Zhihua Zhang, and Liwei Wang. Gram- G auss- N ewton method: Learning overparameterized neural networks for regression problems. arXiv preprint arXiv:1905.11675, 2019

    1905 arXiv

  2. [10]

    Solving the quantum many-body problem with artificial neural networks

    Giuseppe Carleo and Matthias Troyer. Solving the quantum many-body problem with artificial neural networks. Science, 355 0 (6325): 0 602--606, 2017

    2017

  3. [11]

    Empowering deep neural quantum states through efficient optimization

    Ao Chen and Markus Heyl. Empowering deep neural quantum states through efficient optimization. Nature Physics, 20 0 (9): 0 1476--1481, 2024

    2024

  4. [12]

    Kronecker-factored approximate curvature for physics-informed neural networks

    Felix Dangel, Johannes M \"u ller, and Marius Zeinhofer. Kronecker-factored approximate curvature for physics-informed neural networks. Advances in Neural Information Processing Systems, 37: 0 34582--34636, 2024

    2024

  5. [13]

    Stochastic model-based minimization of weakly convex functions

    Damek Davis and Dmitriy Drusvyatskiy. Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization, 29 0 (1): 0 207--239, 2019

    2019

  6. [14]

    Sharp analysis of sketch-and-project methods via a connection to randomized singular value decomposition

    Micha Derezi \'n ski and Elizaveta Rebrova. Sharp analysis of sketch-and-project methods via a connection to randomized singular value decomposition. SIAM Journal on Mathematics of Data Science, 6 0 (1): 0 127--153, 2024

    2024

  7. [15]

    Fine-grained analysis and faster algorithms for iteratively solving linear systems

    Micha Derezi \'n ski, Daniel LeJeune, Deanna Needell, and Elizaveta Rebrova. Fine-grained analysis and faster algorithms for iteratively solving linear systems. arXiv preprint arXiv:2405.05818, 2024

    2024 arXiv

  8. [16]

    Randomized K aczmarz methods with beyond- K rylov convergence

    Micha Derezi \'n ski, Deanna Needell, Elizaveta Rebrova, and Jiaming Yang. Randomized K aczmarz methods with beyond- K rylov convergence. arXiv preprint arXiv:2501.11673, 2025

    2025 arXiv

  9. [17]

    Randomized Kaczmarz with tail averaging

    Ethan N Epperly, Gil Goldshlager, and Robert J Webber. Randomized Kaczmarz with tail averaging. arXiv preprint arXiv:2411.19877, 2024

    2024 arXiv

  10. [18]

    Handbook of convergence theorems for (stochastic) gradient methods

    Guillaume Garrigos and Robert M Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.11235, 2023

    2023 arXiv

  11. [19]

    A K aczmarz-inspired approach to accelerate the optimization of neural network wavefunctions

    Gil Goldshlager, Nilin Abrahamsen, and Lin Lin. A K aczmarz-inspired approach to accelerate the optimization of neural network wavefunctions. Journal of Computational Physics, 516: 0 113351, 2024

    2024

  12. [20]

    Worth their weight: Randomized and regularized block Kaczmarz algorithms without preprocessing

    Gil Goldshlager, Jiang Hu, and Lin Lin. Worth their weight: Randomized and regularized block Kaczmarz algorithms without preprocessing. arXiv preprint arXiv:2502.00882, 2025

    2025

  13. [21]

    Accelerated stochastic matrix inversion: general theory and speeding up BFGS rules for faster second-order optimization

    Robert Gower, Filip Hanzely, Peter Richt \'a rik, and Sebastian U Stich. Accelerated stochastic matrix inversion: general theory and speeding up BFGS rules for faster second-order optimization. Advances in Neural Information Processing Systems, 31, 2018

    2018

  14. [22]

    Randomized iterative methods for linear systems

    Robert M Gower and Peter Richt \'a rik. Randomized iterative methods for linear systems. SIAM Journal on Matrix Analysis and Applications, 36 0 (4): 0 1660--1690, 2015

    2015

  15. [23]

    Solving the H ubbard model with neural quantum states

    Yuntian Gu, Wenrui Li, Heng Lin, Bo Zhan, Ruichen Li, Yifei Huang, Di He, Yantao Wu, Tao Xiang, Mingpu Qin, et al. Solving the H ubbard model with neural quantum states. arXiv preprint arXiv:2507.02644, 2025

    2025 arXiv

  16. [24]

    Improving energy natural gradient descent through woodbury, momentum, and randomization

    Andr \'e s Guzm \'a n-Cordero, Felix Dangel, Gil Goldshlager, and Marius Zeinhofer. Improving energy natural gradient descent through woodbury, momentum, and randomization. arXiv preprint arXiv:2505.12149, 2025

    2025

  17. [25]

    a tzle, and Frank No \'e . Deep-neural-network solution of the electronic S chr \

    Jan Hermann, Zeno Sch \"a tzle, and Frank No \'e . Deep-neural-network solution of the electronic S chr \"o dinger equation. Nature Chemistry, 12 0 (10): 0 891--897, 2020

    2020

  18. [26]

    Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification

    Prateek Jain, Sham M Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification. Journal of machine learning research, 18 0 (223): 0 1--42, 2018

    2018

  19. [27]

    Neural scaling laws surpass chemical accuracy for the many-electron S chr \"o dinger equation

    Du Jiang, Xuelan Wen, Yixiao Chen, Ruichen Li, Weizhong Fu, Hung Q Pham, Ji Chen, Di He, William A Goddard III, Liwei Wang, et al. Neural scaling laws surpass chemical accuracy for the many-electron S chr \"o dinger equation. arXiv preprint arXiv:2508.02570, 2025

    2025 arXiv

  20. [28]

    Characterizing possible failure modes in physics-informed neural networks

    Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in neural information processing systems, 34: 0 26548--26560, 2021

    2021

  21. [29]

    Accelerated natural gradient method for parametric manifold optimization

    Chenyi Li, Shuchen Zhu, Zhonglin Xie, and Zaiwen Wen. Accelerated natural gradient method for parametric manifold optimization. arXiv preprint arXiv:2504.05753, 2025

    2025 arXiv

  22. [30]

    Convergence analysis of stochastic gradient descent with MCMC estimators

    Tianyou Li, Fan Chen, Huajie Chen, and Zaiwen Wen. Convergence analysis of stochastic gradient descent with MCMC estimators. arXiv preprint arXiv:2303.10599, 2023

    2023 arXiv

  23. [31]

    Ab initio calculation of real solids via neural network ansatz

    Xiang Li, Zhe Li, and Ji Chen. Ab initio calculation of real solids via neural network ansatz. Nature Communications, 13 0 (1): 0 7895, 2022

    2022

  24. [32]

    The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning

    Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning, pp.\ 3325--3334. PMLR, 2018

    2018

  25. [33]

    New insights and perspectives on the natural gradient method

    James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 0 (146): 0 1--76, 2020

    2020

  26. [34]

    Fast and furious convergence: Stochastic second order methods under interpolation

    Si Yi Meng, Sharan Vaswani, Issam Hadj Laradji, Mark Schmidt, and Simon Lacoste-Julien. Fast and furious convergence: Stochastic second order methods under interpolation. In International Conference on Artificial Intelligence and Statistics, pp.\ 1375--1386. PMLR, 2020

    2020

  27. [35]

    Achieving high accuracy with PINN s via energy natural gradient descent

    Johannes M \"u ller and Marius Zeinhofer. Achieving high accuracy with PINN s via energy natural gradient descent. In International Conference on Machine Learning, pp.\ 25471--25485. PMLR, 2023

    2023

  28. [36]

    Paved with good intentions: analysis of a randomized block K aczmarz method

    Deanna Needell and Joel A Tropp. Paved with good intentions: analysis of a randomized block K aczmarz method. Linear Algebra and its Applications, 441: 0 199--221, 2014

    2014

  29. [37]

    Stochastic gradient descent, weighted sampling, and the randomized K aczmarz algorithm

    Deanna Needell, Nathan Srebro, and Rachel Ward. Stochastic gradient descent, weighted sampling, and the randomized K aczmarz algorithm. Advances in neural information processing systems, 27, 2014

    2014

  30. [38]

    Geometry of learning neural quantum states

    Chae-Yeun Park and Michael J Kastoryano. Geometry of learning neural quantum states. Physical Review Research, 2 0 (2): 0 023232, 2020

    2020

  31. [39]

    Ab initio solution of the many-electron S chr \"o dinger equation with deep neural networks

    David Pfau, James S Spencer, Alexander GDG Matthews, and W Matthew C Foulkes. Ab initio solution of the many-electron S chr \"o dinger equation with deep neural networks. Physical review research, 2 0 (3): 0 033429, 2020

    2020

  32. [40]

    Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations

    Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378: 0 686--707, 2019

    2019

  33. [41]

    Challenges in training PINN s: A loss landscape perspective

    Pratik Rathore, Weimu Lei, Zachary Frangella, Lu Lu, and Madeleine Udell. Challenges in training PINN s: A loss landscape perspective. arXiv preprint arXiv:2402.01868, 2024

    2024 arXiv

  34. [42]

    Efficient subsampled G auss- N ewton and natural gradient methods for training neural networks

    Yi Ren and Donald Goldfarb. Efficient subsampled G auss- N ewton and natural gradient methods for training neural networks. arXiv preprint arXiv:1906.02353, 2019

    1906 arXiv

  35. [43]

    A simple linear algebra identity to optimize large-scale neural network quantum states

    Riccardo Rende, Luciano Loris Viteritti, Lorenzo Bardone, Federico Becca, and Sebastian Goldt. A simple linear algebra identity to optimize large-scale neural network quantum states. Communications Physics, 7 0 (1): 0 260, 2024

    2024

  36. [44]

    Stochastic reformulations of linear systems: algorithms and convergence theory

    Peter Richt \'a rik and Martin Tak \'a c. Stochastic reformulations of linear systems: algorithms and convergence theory. SIAM Journal on Matrix Analysis and Applications, 41 0 (2): 0 487--524, 2020

    2020

  37. [45]

    Sub-sampled newton methods i: globally convergent algorithms

    Farbod Roosta-Khorasani and Michael W Mahoney. Sub-sampled newton methods i: globally convergent algorithms. arXiv preprint arXiv:1601.04737, 2016 a

    2016 arXiv

  38. [46]

    Sub-sampled newton methods ii: Local convergence rates

    Farbod Roosta-Khorasani and Michael W Mahoney. Sub-sampled newton methods ii: Local convergence rates. arXiv preprint arXiv:1601.04738, 2016 b

    2016 arXiv

  39. [47]

    Accurate ab-initio neural-network solutions to large-scale electronic structure problems

    Michael Scherbela, Nicholas Gao, Philipp Grohs, and Stephan G \"u nnemann. Accurate ab-initio neural-network solutions to large-scale electronic structure problems. arXiv preprint arXiv:2504.06087, 2025

    2025 arXiv

  40. [48]

    Unified variational approach description of ground-state phases of the two-dimensional electron gas

    Conor Smith, Yixiao Chen, Ryan Levy, Yubo Yang, Miguel A Morales, and Shiwei Zhang. Unified variational approach description of ground-state phases of the two-dimensional electron gas. Physical Review Letters, 133 0 (26): 0 266504, 2024

    2024

  41. [49]

    Generalized L anczos algorithm for variational quantum M onte C arlo

    Sandro Sorella. Generalized L anczos algorithm for variational quantum M onte C arlo. Physical Review B, 64 0 (2): 0 024512, 2001

    2001

  42. [50]

    When and why PINN s fail to train: A neural tangent kernel perspective

    Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why PINN s fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449: 0 110768, 2022

    2022

  43. [51]

    Rayleigh- G auss- N ewton optimization with enhanced sampling for variational M onte C arlo

    Robert J Webber and Michael Lindsey. Rayleigh- G auss- N ewton optimization with enhanced sampling for variational M onte C arlo. Physical Review Research, 4 0 (3): 0 033099, 2022

    2022

  44. [52]

    Convergence analysis of an adaptively regularized natural gradient method

    Jiayuan Wu, Jiang Hu, Hongchao Zhang, and Zaiwen Wen. Convergence analysis of an adaptively regularized natural gradient method. IEEE Transactions on Signal Processing, 72: 0 2527--2542, 2024

    2024

  45. [53]

    Convergence analysis of natural gradient descent for over-parameterized physics-informed neural networks

    Xianliang Xu, Ting Du, Wang Kong, Bin Shan, Ye Li, and Zhongyi Huang. Convergence analysis of natural gradient descent for over-parameterized physics-informed neural networks. arXiv preprint arXiv:2408.00573, 2024

    2024 arXiv

  46. [54]

    Sketchy empirical natural gradient methods for deep learning

    Minghan Yang, Dong Xu, Zaiwen Wen, Mengyun Chen, and Pengxiang Xu. Sketchy empirical natural gradient methods for deep learning. arXiv preprint arXiv:2006.05924, 2020

    2006 arXiv

  47. [55]

    Fast convergence of natural gradient descent for over-parameterized neural networks

    Guodong Zhang, James Martens, and Roger B Grosse. Fast convergence of natural gradient descent for over-parameterized neural networks. Advances in Neural Information Processing Systems, 32, 2019

    2019

  48. [56]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  49. [57]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  50. [58]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.