Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Position: Curvature Matrices Should Be Democratized via Linear Operators

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that exposing curvature matrices as linear operators—objects that multiply vectors without ever materializing the matrix—unifies exact and approximate curvature computation, as demonstrated by the curvlinops library.

desk verdict A position paper with a real library behind it: the linear-operator claim holds, but the 'without worrying' framing is oversold and the paper's own Section 5 admits it. read the letter →

arxiv 2501.19183 v1 pith:AXJ2DQ24 submitted 2025-01-31 cs.LG

Felix Dangel , Runa Eschenhagen , Weronika Ormaniec , Andres Fernandez , Lukas Tatzel , Agustinus Kristiadi This is my paper · ORCID
classification cs.LG
keywords linearoperatorscurvaturematricesHessiangeneralizedGauss-NewtonFisherinformationmatrixKFACrandomizedalgebraneuralnetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This position paper argues that curvature matrices—the Hessian of a neural network's empirical risk and its cheaper approximations, such as the generalized Gauss-Newton, Fisher information, and KFAC—should be presented to machine-learning researchers as linear operators, objects that answer matrix-vector products without ever building the full matrix. To back this claim, the paper introduces curvlinops, a library that wraps exact and approximate curvature matrices in one common interface. It argues that this interface delivers three benefits: it hides error-prone implementation details such as automatic differentiation, batch scaling, and stochasticity; it reduces applications like second-order optimization, influence functions, model merging, and pruning to a few lines of code that mirror the underlying mathematics; and it makes existing linear-algebra tooling, including iterative eigensolvers and randomized estimators of trace, diagonal, and spectral density, directly usable. If the position is right, a single abstraction could replace much of the bespoke curvature code in deep learning.

What carries the argument

The load-bearing object is the linear operator: an interface that answers the matrix-vector product $A(v)$ for a matrix $A$ and satisfies the two linearity laws $A(v_1+v_2)=A(v_1)+A(v_2)$ and $A(\alpha v)=\alpha A(v)$. Its work in the argument is to make the representation the same regardless of how the curvature is computed: the Hessian uses nested automatic differentiation, the generalized Gauss-Newton uses vector-Jacobian and Jacobian-vector products, Fisher-vector products are obtained from a Gauss-Newton product on a detached pseudo-loss $f^\top g g^\top f$, and KFAC multiplies vectors through sums of Kronecker products of layer factors. Because every curvature variant lives behind the same multiplication interface, users can compose and invert operators, and swap exact for approximate curvature, without changing their application code.

What would settle it

Run a controlled user study in which practitioners with limited experience in second-order optimization implement a standard task, such as influence-function attribution or Fisher-weighted model merging, given only the linear-operator interface; then compare the curvature matrices they select and the correctness of their results against a reference expert implementation. If a substantial fraction of non-expert users accidentally choose an empirical Fisher where a generalized Gauss-Newton or damped inverse is required, and the library's checks do not catch it, the democratization claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is a design claim, not a new theorem: a single linear-operator interface $v \mapsto A(v)$, where $A$ is a linear map acting like a matrix $A$ but never materializing it, is general enough to cover the curvature matrices that matter in deep learning and convenient enough that application code can treat them like dense matrices. The paper demonstrates this with curvlinops, which provides operators for the Hessian, the generalized Gauss-Newton, the Monte-Carlo and empirical Fisher, and several (E)KFAC variants, and shows that a user can write the same style of code for Newton-CG, influence-function inversion, Fisher-weighted model merging, pruning, and Hessian spectral analysis. It also argues that the interface is extensible, because new curvature approximations are just new operators, and interoperable, because operators can be composed, transformed (for example inverted through iterative solvers), exported to established numerical linear-algebra routines, and connected to randomized linear algebra for estimating matrix properties at scale.

Load-bearing premise

The argument rests on the assumption that a user who cannot see the internal details can still choose the curvature matrix that is mathematically appropriate for their application, so that hiding complexity helps rather than silently misleads.

Editorial extensions

If this is right

  • Second-order optimizers and their approximate variants can be implemented once against the operator interface, with Newton-CG, Neumann-series inversion, and KFAC inverses selected by swapping operators.
  • Influence functions, bi-level optimization, and hyper-gradient methods can use full inverse curvature-vector products rather than being forced onto diagonal approximations.
  • Fisher-weighted model merging can move beyond diagonal Fishers to full or block-structured Fisher operators via conjugate-gradient solves.
  • Randomized linear algebra estimators for trace, diagonal, spectral density, and low-rank sketches become immediately applicable to any curvature matrix, not just one bespoke implementation.
  • Curvature estimators maintained inside optimizers can be exported as operators at any point of training and reused for pruning, uncertainty quantification, or landscape analysis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is to attach a machine-readable 'curvature pedigree' to each operator—specifying whether it is a Hessian, GGN, type-I or type-II Fisher, empirical Fisher, its damping, and its data reduction—so that automated checks can veto an operator that is mathematically inappropriate for the requested application.
  • The same operator abstraction would transfer beyond curvature to any structured matrix used in deep learning, such as attention, kernel, or factorized weight matrices, potentially giving the broader field the same interoperability with randomized linear algebra.
  • If the interface becomes a standard, an empirical prediction follows: the dominant remaining source of user error in curvature-based applications will shift from implementing matrix-vector products to choosing the right curvature variant, which would make documentation and validation layers more important than the operator itself.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This position paper argues that presenting curvature matrices such as the Hessian, GGN, Fisher variants, and KFAC as linear operators—i.e., as black-box matrix-vector product interfaces—provides a general, scalable, and user-friendly abstraction. The argument is supported by curvlinops, a PyTorch library that implements these operators, offers composition and transformation, exports to SciPy, and provides randomized linear algebra estimators. The manuscript illustrates the interface on five applications, reports runtime and memory benchmarks on ResNet50/ImageNet and nanoGPT/Shakespeare, validates estimators on toy problems, and includes a candid alternative-views section that discusses limitations.

Significance. If the position is accepted, a single linear-operator interface could unify exact and approximate curvature matrices across optimization, influence functions, pruning, merging, and spectral analysis, while connecting deep learning to mature randomized linear algebra and SciPy tooling. The manuscript ships a pip-installable library, reproducible code snippets and toy validations, and performance measurements that are broadly consistent with published HVP/GGNVP cost factors; the central claim is independently checkable and is not circular. The main gap is that the 'democratization' and 'user-friendly' wording claims more than the evidence establishes: the interface hides computational mechanics but leaves the consequential choice of curvature object to the user, and the paper's own Section 5 concedes that an inappropriate choice can be made unknowingly. There is also a concrete mathematical slip in the pseudo-loss derivation for Fisher-vector products (Section 2.1) that should be corrected. These issues are fixable, and the manuscript's strengths—transparency, reproducible software, and a clear design argument—make it a valuable position statement.

major comments (3)
  1. [Section 2.1] In the paragraph 'Fisher-vector products (FVPs) via GGNVPs', the pseudo-loss is defined as \ell~(f_n,y_n) = f_n^T g_n g_n^T f_n. Its Hessian with respect to f_n is 2 g_n g_n^T, not g_n g_n^T as claimed immediately after the definition. Consequently, the GGN of the stated pseudo-loss is twice the desired Fisher, MC-Fisher, or empirical Fisher. Please correct the definition to \ell~(f_n,y_n) = (1/2) f_n^T g_n g_n^T f_n, or state explicitly that the implementation rescales by 1/2. This is a load-bearing mathematical error because a user implementing the described recipe would obtain incorrectly scaled Fisher-vector products.
  2. [Section 1 and Section 5] The central claim that users can apply curvature matrices 'without worrying about implementation complexity' and the title's 'Democratized' overreach what the interface actually provides. Section 2.1 and the Step 1 API in Section 3 require users to choose among Hessian, GGN, Fisher, MC-Fisher, empirical Fisher, and KFAC and to set definitional kwargs such as GGN split, Fisher type, and damping. Section 5 explicitly concedes that the user 'might unknowingly use a curvature linear operator that is inappropriate for their application.' The abstraction therefore does not free users from the conceptually consequential choice of curvature matrix, and a wrong choice yields silently wrong results rather than a warning. The paper should either narrow the claim throughout to 'without worrying about implementation complexity' in the narrow computational sense, or add safeguards and guidance—e.g., operator-application compatibility checks and warnings about definitional choices—to make the democratization claim credible.
  3. [Appendix A] The claim that HVP performance 'matches related work' is stated from ranges (4.5–5.5 gradient equivalents observed versus 4–5 reported by Dagréou et al.) that are close but not identical, with no confidence intervals, number of repeats, or matching of setup details such as batch size and model family. Since the scalability argument is part of the paper's support, please report dispersion and repeat counts, and soften the claim to 'is consistent with' rather than 'matches' if the conditions are not directly comparable.
minor comments (4)
  1. [Section 1] The sentence 'In this paper, we argue our position that;' is a grammatical fragment and should be rephrased, e.g., 'In this paper, we argue the following position: ...'
  2. [Section 2.1] The spelling 'Naïve' and 'naive' appear inconsistently; please choose one convention throughout.
  3. [Section 3, Application 3] In the model-merging snippet, the variable 'Fs' is constructed with GGNLinearOperator but described as 'Per-task Fisher matrices'. Please state the conditions under which the GGN equals the Fisher (or rename the variables) to avoid implying that the two are interchangeable unconditionally.
  4. [Figure 4 caption] The top two panels on spectral density do not state the number of Monte-Carlo repetitions used, while the bottom panels report medians over 200 runs; please add this information for reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the paper advocates a design position backed by an installable library; the only mildly self-referential element is that the demonstrations use the authors' own curvlinops, which is independently checkable and not a reduction of the claim to its input.

full rationale

This is a position paper rather than a derivation of quantitative predictions, so the standard circularity patterns do not apply. The mathematical content in Section 2 (Hessian, GGN, type-I/II Fisher, MC/empirical Fisher, KFAC, and the pseudo-loss identity for FVPs) is stated from standard definitions or proven directly, e.g. the pseudo-loss construction yields the desired Fisher because ∇²_f ℓ̃ = g gᵀ, which is a definitional identity rather than a fitted assumption. Section 3's applications are code snippets that follow from the operator abstraction, not a fitted parameter renamed as a prediction. The main self-referential element is that the performance benchmarks in Appendix A and the randomized-linear-algebra sanity checks in Appendix C are produced with the authors' own curvlinops library, which instantiates the very interface whose benefits are claimed. However, the library is installable and the snippets are executable, so the support is independently checkable and does not reduce the position to its own input. Co-authored citations (e.g., Eschenhagen et al. 2023) appear in implementation and testing contexts, not as a load-bearing justification for the central claim, and there is no imported uniqueness theorem or ansatz smuggled in via self-citation. Section 5 explicitly concedes, 'In both cases, the user might unknowingly use a curvature linear operator that is inappropriate for their application,' which weakens the strong 'without worrying' wording but is an acknowledged design tradeoff, not circular reasoning. Overall, no significant circularity is present; the score of 2 reflects only the mildly self-referential demonstration.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted; the paper makes no numerical claims requiring parameter fitting. The axioms listed are background assumptions: the importance of curvature, the sufficiency of matvec queries, the correctness of the library, and the representativeness of the benchmarks. No new theoretical entities are postulated; the library is the paper's artifact, not a hypothesis. Overall, the paper's contribution rests on engineering assumptions rather than on unstated fitted parameters.

assumptions (4)
  • domain assumption Curvature matrices are essential to the cited applications (second-order optimization, uncertainty, pruning, influence, spectra) and challenging to compute.
    Motivates the position in Sections 1 and 2; not proven in the paper but widely accepted in the community.
  • domain assumption Matrix-vector products are sufficient for a broad class of curvature applications.
    Supporting examples in Section 3 all consume matvecs; the paper does not address applications that need submatrices or full factors.
  • domain assumption The curvlinops implementations compute the intended curvature matrices correctly.
    Correctness is backed by sanity checks and equivalence tests for KFAC (Section 2.2), but no formal proof; the benchmarking claims inherit this assumption.
  • ad hoc to paper The two benchmark settings (ResNet50 on ImageNet, nanoGPT on Shakespeare) are representative for the scalability claim.
    Appendix A uses these with chosen batch sizes and excludes some layers from KFAC; generalizing beyond them is an assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Position: Curvature Matrices Should Be Democratized via Linear Operators." pith.science (2026). https://pith.science/paper/AXJ2DQ24

@misc{pith2026250119183,
  author       = {Pith},
  title        = {Pith review of: Position: Curvature Matrices Should Be Democratized via Linear Operators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AXJ2DQ24}},
  note         = {Machine review of arXiv:2501.19183}
}
abstract

Structured large matrices are prevalent in machine learning. A particularly important class is curvature matrices like the Hessian, which are central to understanding the loss landscape of neural nets (NNs), and enable second-order optimization, uncertainty quantification, model pruning, data attribution, and more. However, curvature computations can be challenging due to the complexity of automatic differentiation, and the variety and structural assumptions of curvature proxies, like sparsity and Kronecker factorization. In this position paper, we argue that linear operators -- an interface for performing matrix-vector products -- provide a general, scalable, and user-friendly abstraction to handle curvature matrices. To support this position, we developed $\textit{curvlinops}$, a library that provides curvature matrices through a unified linear operator interface. We demonstrate with $\textit{curvlinops}$ how this interface can hide complexity, simplify applications, be extensible and interoperable with other libraries, and scale to large NNs.

Figures

Figures reproduced from arXiv: 2501.19183 by the authors.

Figure 2
Figure 2. Visual tour of curvature matrices. White lines separate parameters into layers. We consider a synthetic classification task with a small convolutional neural net (three convolutional and one dense layer with ReLU and sigmoid activations, D = 683). We also demonstrate scalability to larger architectures like nanoGPT on Shakespeare and ResNet50 on ImageNet (§A). Related software. Various Python packages provide a line… view at source ↗
Figure 3
Figure 3. Performance analysis: Run time (left column) and peak memory (right column) of linear operators benchmarked on ResNet50 on ImageNet (top row) and nanoGPT on Shakespeare (bottom row) on an A40 GPU with 40 GiB of RAM (the code used to generate these results is here). Details: For ImageNet, we use a batch size of 64 and images of shape (3, 224, 224); for Shakespeare, we use a batch size of 4 and context length 1024. Al… view at source ↗
Figure 4
Figure 4. Estimating linear operator properties with curvlinops. We implement various estimation algorithms from the literature and evaluate them on toy problems. Top: Spectral density estimation with the algorithms and toy matrices from Papyan et al. (2020). The left panel estimates a spectral density, the right panel the spectral density of the matrix logarithm log(|A| + ϵI) with ϵ = 10−5 . Code to reproduce these figures i… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. laplax -- Laplace Approximations with JAX

    cs.LG 2025-07 conditional novelty 6.0 of 10

    The paper presents laplax, a modular JAX library for Laplace approximations that supports multiple curvature estimates, uncertainty pushforwards, calibration, and evaluation routines.

Reference graph

Works this paper leans on

103 extracted references · 66 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Natural gradient works efficiently in learning

    Amari, S.-I. Natural gradient works efficiently in learning. Neural Computation, 2000

    2000

  3. [3]

    An estimator for the diagonal of a matrix

    Bekas, C., Kokiopoulou, E., and Saad, Y. An estimator for the diagonal of a matrix. Applied Numerical Mathematics, 2007

    2007

  4. [4]

    Gradient descent on neurons and its link to approximate second-order optimization

    Benzing, F. Gradient descent on neurons and its link to approximate second-order optimization. In International Conference on Machine Learning (ICML), 2022

    2022

  5. [5]

    Exact natural gradient in deep linear networks and its application to the nonlinear case

    Bernacchia, A., Lengyel, M., and Hennequin, G. Exact natural gradient in deep linear networks and its application to the nonlinear case. In Advances in Neural Information Processing Systems (NeurIPS), 2018

    2018

  6. [6]

    and Bergholm, V

    Biamonte, J. and Bergholm, V. Tensor networks in a nutshell, 2017

    2017

  7. [7]

    Practical G auss- N ewton optimisation for deep learning

    Botev, A., Ritter, H., and Barber, D. Practical G auss- N ewton optimisation for deep learning. In International Conference on Machine Learning (ICML), 2017

    2017

  8. [8]

    J., Leary, C., Maclaurin, D., and Wanderman-Milne, S

    Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., and Wanderman-Milne, S. JAX : composable transformations of P ython+ N um P y programs, 2018. URL https://github.com/jax-ml/jax

    2018

Show all 103 references
  1. [9]

    and van Handel , R

    Brailovskaya, T. and van Handel , R. Universality and sharp matrix concentration inequalities. Geometric and Functional Analysis, 34 0 (6): 0 1734--1838, December 2024. ISSN 1016-443X

    2024

  2. [10]

    Bridgeman, J. C. and Chubb, C. T. Hand-waving and interpretive dance: an introductory course on tensor networks. Journal of Physics A: Mathematical and theoretical, 2017

    2017

  3. [11]

    A., Collin, F.-D., and Durif, G

    Charlier, B., Feydy, J., Glaunes, J. A., Collin, F.-D., and Durif, G. Kernel operations on the gpu, with autodiff, without memory overflows. Journal of Machine Learning Research (JMLR), 2021

    2021

  4. [12]

    Efficient second-order optimization for neural networks with kernel machines

    Chen, Y., Chen, Y., Chen, J., Wen, Z., and Huang, J. Efficient second-order optimization for neural networks with kernel machines. In ACM International Conference on Information & Knowledge Management, 2022

    2022

  5. [13]

    How to compute hessian-vector products? In International Conference on Learning Representations (ICLR) Blogposts, 2024

    Dagréou, M., Ablin, P., Vaiter, S., and Moreau, T. How to compute hessian-vector products? In International Conference on Learning Representations (ICLR) Blogposts, 2024

    2024

  6. [14]

    B ack PACK : Packing more into backprop

    Dangel, F., Kunstner, F., and Hennig, P. B ack PACK : Packing more into backprop. In International Conference on Learning Representations (ICLR), 2020

    2020

  7. [15]

    Vi V i T : Curvature access through the generalized gauss-newton s low-rank structure

    Dangel, F., Tatzel, L., and Hennig, P. Vi V i T : Curvature access through the generalized gauss-newton s low-rank structure. Transactions on Machine Learning Research (TMLR), 2022

    2022

  8. [16]

    Learning fast algorithms for linear transforms using butterfly factorizations

    Dao, T., Gu, A., Eichhorn, M., Rudra, A., and R \'e , C. Learning fast algorithms for linear transforms using butterfly factorizations. In International Conference on Machine Learning (ICML), 2019

    2019

  9. [17]

    S., Desai, A., Poli, M., Grogan, J., Liu, A., Rao, A., Rudra, A., and R \'e , C

    Dao, T., Chen, B., Sohoni, N. S., Desai, A., Poli, M., Grogan, J., Liu, A., Rao, A., Rudra, A., and R \'e , C. Monarch: Expressive structured matrices for efficient and accurate training. In International Conference on Machine Learning (ICML), 2022

    2022

  10. [18]

    Laplace redux - effortless bayesian deep learning

    Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., and Hennig, P. Laplace redux - effortless bayesian deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  11. [19]

    and Mahoney, M

    Derezi \'n ski, M. and Mahoney, M. W. Recent and upcoming developments in randomized numerical linear algebra for machine learning. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024

    2024

  12. [20]

    N., Tropp, J

    Epperly, E. N., Tropp, J. A., and Webber, R. J. Xtrace: Making the most of every sample in stochastic trace estimation. SIAM Journal on Matrix Analysis and Applications (SIMAX), 2024

    2024

  13. [21]

    E., Schneider, F., and Hennig, P

    Eschenhagen, R., Immer, A., Turner, R. E., Schneider, F., and Hennig, P. Kronecker-factored approximate curvature for modern neural network architectures. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  14. [22]

    Evtushenko, Y. G. Iterative methods for solving minimax problems. USSR Computational Mathematics and Mathematical Physics, 1974

    1974

  15. [23]

    S kerch: Sketched matrix decompositions for PyTorch , 2024

    Fernandez, A. S kerch: Sketched matrix decompositions for PyTorch , 2024. URL https://github.com/andres-fr/skerch

    2024

  16. [24]

    and Hodges, J

    Fix, E. and Hodges, J. Discriminatory Analysis: Nonparametric Discrimination: Consistency Properties. USAF School of Aviation Medicine, 1951

    1951

  17. [25]

    and Saunders, M

    Fong, D. and Saunders, M. LSMR : An iterative algorithm for sparse least-squares problems: Systems optimization laboratory. SIAM J. Sci. Comput. , 2010

    2010

  18. [26]

    M-fac: Efficient matrix-free approximations of second-order information

    Frantar, E., Kurtic, E., and Alistarh, D. M-fac: Efficient matrix-free approximations of second-order information. Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  19. [27]

    Q., Bindel, D., and Wilson, A

    Gardner, J., Pleiss, G., Weinberger, K. Q., Bindel, D., and Wilson, A. G. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. Advances in Neural Information Processing Systems (NeurIPS), 2018

    2018

  20. [28]

    Fast approximate natural gradient descent in a kronecker-factored eigenbasis, 2018

    George, T., Laurent, C., Bouthillier, X., Ballas, N., and Vincent, P. Fast approximate natural gradient descent in a kronecker-factored eigenbasis, 2018

    2018

  21. [29]

    Eigenvalue and generalized eigenvalue problems: Tutorial, 2019

    Ghojogh, B., Karray, F., and Crowley, M. Eigenvalue and generalized eigenvalue problems: Tutorial, 2019

    2019

  22. [30]

    Girard, D. A. A fast 'monte-carlo cross-validation' procedure for large least squares problems with noisy data. Numerische Mathematik, 1989

    1989

  23. [31]

    pytorch-hessian-eigenthings: efficient pytorch hessian eigendecomposition, 2018

    Golmant, N., Yao, Z., Gholami, A., Mahoney, M., and Gonzalez, J. pytorch-hessian-eigenthings: efficient pytorch hessian eigendecomposition, 2018

    2018

  24. [32]

    Golub, G. H. and Van Loan, C. F. Matrix Computations. Johns Hopkins University Press, 4 edition, 2013. doi:10.1137/1.9781421407944

    2013 doi

  25. [33]

    Deep curvature suite, 2019

    Granziol, D., Wan, X., and Garipov, T. Deep curvature suite, 2019

    2019

  26. [34]

    and Martens, J

    Grosse, R. and Martens, J. A kronecker-factored approximate F isher matrix for convolution layers. In International Conference on Machine Learning (ICML), 2016

    2016

  27. [35]

    Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., Hubinger, E., Lukošiūtė, K., Nguyen, K., Joseph, N., McCandlish, S., Kaplan, J., and Bowman, S. R. Studying large language model generalization with influence functi...

    2023

  28. [36]

    A., and Dyer, E

    Gur-Ari, G., Roberts, D. A., and Dyer, E. Gradient descent happens in a tiny subspace, 2018

    2018

  29. [37]

    G., and Tropp, J

    Halko, N., Martinsson, P. G., and Tropp, J. A. Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions . SIREV, 2011

    2011

  30. [38]

    Hampel, F. R. The influence curve and its role in robust estimation. Journal of the American Statistical Association, 1974

    1974

  31. [39]

    Ginger: An efficient curvature approximation with linear complexity for general neural networks

    Hao, Y., Cao, Y., and Mou, L. Ginger: An efficient curvature approximation with linear complexity for general neural networks. ar X iv 2402.03295, 2024

    2024 arXiv

  32. [40]

    R., Millman, K

    Harris, C. R., Millman, K. J., Van Der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. Array programming with numpy. Nature, 2020

    2020

  33. [41]

    and Stork, D

    Hassibi, B. and Stork, D. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems (NIPS), 1992

    1992

  34. [42]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition (CVPR), 2016

    2016

  35. [43]

    On ``natural'' learning and pruning in multilayered perceptrons

    Heskes, T. On ``natural'' learning and pruning in multilayered perceptrons. Neural Computation, 12 0 (4), 2000

    2000

  36. [44]

    Hestenes, M. R. and Stiefel, E. Methods of conjugate gradients for solving linear systems. Journal of research of the National Bureau of Standards, 49: 0 409--435, 1952

    1952

  37. [45]

    A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines

    Hutchinson, M. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communication in Statistics---Simulation and Computation, 1989

    1989

  38. [46]

    and Szegedy, C

    Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015

    2015

  39. [47]

    Neural tangent kernel: Convergence and generalization in neural networks

    Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems (NIPS), 2018

    2018

  40. [48]

    The simplest, fastest repository for training/finetuning medium-sized gpts., 2022

    Karpathy, A. The simplest, fastest repository for training/finetuning medium-sized gpts., 2022. URL https://github.com/karpathy/nanoGPT

    2022

  41. [49]

    Kipf, T. N. and Welling, M. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017

    2017

  42. [50]

    Koh, P. W. and Liang, P. Understanding black-box predictions via influence functions. In International Conference on Machine Learning (ICML), 2017

    2017

  43. [51]

    Krishnan, S., Xiao, Y., and Saurous, R. A. Neumann optimizer: A practical optimization algorithm for deep neural networks, 2017

    2017

  44. [52]

    A sober look at LLMs for material discovery: Are they actually good for B ayesian optimization over molecules? In International Conference on Machine Learning (ICML), 2024

    Kristiadi, A., Strieth-Kalthoff, F., Skreta, M., Poupart, P., Aspuru-Guzik, A., and Pleiss, G. A sober look at LLMs for material discovery: Are they actually good for B ayesian optimization over molecules? In International Conference on Machine Learning (ICML), 2024

    2024

  45. [53]

    Limitations of the empirical fisher approximation for natural gradient descent

    Kunstner, F., Hennig, P., and Balles, L. Limitations of the empirical fisher approximation for natural gradient descent. In Advances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  46. [54]

    Optimal brain damage

    LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. In Advances in Neural Information Processing Systems (NIPS), 1989

    1989

  47. [55]

    B., Sorensen, D

    Lehoucq, R. B., Sorensen, D. C., and Yang, C. ARPACK users' guide: solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods. SIAM, 1998

    1998

  48. [56]

    E., and Grosse, R

    Lin, W., Dangel, F., Eschenhagen, R., Bae, J., Turner, R. E., and Grosse, R. B. Fast fractional natural gradient descent using learnable spectral factorizations, 2024 a

    2024

  49. [57]

    E., and Makhzani, A

    Lin, W., Dangel, F., Eschenhagen, R., Bae, J., Turner, R. E., and Makhzani, A. Can we remove the square-root in adaptive gradient methods? a second-order perspective. In International Conference on Machine Learning (ICML), 2024 b

    2024

  50. [58]

    E., and Makhzani, A

    Lin, W., Dangel, F., Eschenhagen, R., Neklyudov, K., Kristiadi, A., Turner, R. E., and Makhzani, A. Structured inverse-free natural gradient descent: Memory-efficient & numerically-stable KFAC . In International Conference on Machine Learning (ICML), 2024 c

    2024

  51. [59]

    Loan, C. F. The ubiquitous K ronecker product. Journal of Computational and Applied Mathematics, 2000

    2000

  52. [60]

    Optimizing millions of hyperparameters by implicit differentiation

    Lorraine, J., Vicol, P., and Duvenaud, D. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020

    2020

  53. [61]

    Indiscriminate data poisoning attacks on neural networks

    Lu, Y., Kamath, G., and Yu, Y. Indiscriminate data poisoning attacks on neural networks. TMLR, 2022

    2022

  54. [62]

    MacKay, D. J. A practical B ayesian framework for backpropagation networks. Neural Computation, 4 0 (3), 1992

    1992

  55. [63]

    Deep learning via H essian-free optimization

    Martens, J. Deep learning via H essian-free optimization. In International Conference on Machine Learning (ICML), 2010

    2010

  56. [64]

    New insights and perspectives on the natural gradient method, 2020

    Martens, J. New insights and perspectives on the natural gradient method, 2020

    2020

  57. [65]

    and Grosse, R

    Martens, J. and Grosse, R. Optimizing neural networks with K ronecker-factored approximate curvature. In International Conference on Machine Learning (ICML), 2015

    2015

  58. [66]

    and Sutskever, I

    Martens, J. and Sutskever, I. Training Deep and Recurrent Networks with Hessian-Free Optimization, pp.\ 479--535. Springer Berlin Heidelberg, 2012

    2012

  59. [67]

    Kronecker-factored curvature approximations for recurrent neural networks

    Martens, J., Ba, J., and Johnson, M. Kronecker-factored curvature approximations for recurrent neural networks. In International Conference on Learning Representations (ICLR), 2018

    2018

  60. [68]

    Matena, M. S. and Raffel, C. A. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  61. [69]

    A., Musco, C., Musco, C., and Woodruff, D

    Meyer, R. A., Musco, C., Musco, C., and Woodruff, D. P. Hutch++: Optimal stochastic trace estimation, 2020

    2020

  62. [70]

    Influence functions for scalable data attribution in diffusion models

    Mlodozeniec, B., Eschenhagen, R., Bae, J., Immer, A., Krueger, D., and Turner, R. Influence functions for scalable data attribution in diffusion models. In International Conference on Learning Representations (ICLR), 2025

    2025

  63. [71]

    Asdl: A unified interface for gradient preconditioning in pytorch, 2023

    Osawa, K., Ishikawa, S., Yokota, R., Li, S., and Hoefler, T. Asdl: A unified interface for gradient preconditioning in pytorch, 2023

    2023

  64. [72]

    Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet H essians

    Papyan, V. Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet H essians. In International Conference on Machine Learning (ICML), 2019 a

    2019

  65. [73]

    The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size, 2019 b

    Papyan, V. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size, 2019 b

    2019

  66. [74]

    Y., and Donoho, D

    Papyan, V., Han, X. Y., and Donoho, D. L. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences (PNAS), 2020

    2020

  67. [75]

    PyTorch : An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch : An imper...

    2019

  68. [76]

    Pearlmutter, B. A. Fast exact multiplication by the H essian. Neural Computation, 1994

    1994

  69. [77]

    Applications of negative dimensional tensors

    Penrose, R. Applications of negative dimensional tensors. Combinatorial Mathematics and its Applications, 1971

    1971

  70. [78]

    ISAAC newton: Input-based approximate curvature for newton's method

    Petersen, F., Sutter, T., Borgelt, C., Huh, D., Kuehne, H., Sun, Y., and Deussen, O. ISAAC newton: Input-based approximate curvature for newton's method. In International Conference on Learning Representations (ICLR), 2023

    2023

  71. [79]

    Potapczynski, A., Finzi, M., Pleiss, G., and Wilson, A. G. CoLA: Exploiting Compositional Structure for Automatic and Efficient Numerical Linear Algebra . In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  72. [80]

    A., Ferri, C., Chen, Z., Goldblum, M., Bruss, C

    Potapczynski, A., Qiu, S., Finzi, M. A., Ferri, C., Chen, Z., Goldblum, M., Bruss, C. B., Sa, C. D., and Wilson, A. G. Searching for efficient linear layers over a continuous space of structured matrices. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  73. [81]

    Lineax: unified linear solves and linear least-squares in jax and equinox

    Rader, J., Lyons, T., and Kidger, P. Lineax: unified linear solves and linear least-squares in jax and equinox. Advances in Neural Information Processing Systems (NeurIPS), Workshop NeuAI for Science, 2023

    2023

  74. [82]

    and Vasconcelos, I

    Ravasi, M. and Vasconcelos, I. Pylops--a linear-operator python library for large scale optimization. ar X iv 1907.12349, 2019

    1907 arXiv

  75. [83]

    and Goldfarb, D

    Ren, Y. and Goldfarb, D. Efficient subsampled gauss-newton and natural gradient methods for training neural networks, 2019

    2019

  76. [84]

    Eigenvalues of the hessian in deep learning: Singularity and beyond, 2017

    Sagun, L., Bottou, L., and LeCun, Y. Eigenvalues of the hessian in deep learning: Singularity and beyond, 2017

    2017

  77. [85]

    U., Dauphin, Y., and Bottou, L

    Sagun, L., Evci, U., Guney, V. U., Dauphin, Y., and Bottou, L. Empirical analysis of the hessian of over-parametrized neural networks, 2018

    2018

  78. [86]

    Cockpit: A practical debugging tool for the training of deep neural networks

    Schneider, F., Dangel, F., and Hennig, P. Cockpit: A practical debugging tool for the training of deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  79. [87]

    Schraudolph, N. N. Fast curvature matrix-vector products for second-order gradient descent. Neural Computation, 2002

    2002

  80. [88]

    Singh, S. P. and Alistarh, D. Woodfisher: Efficient second-order approximation for neural network compression. In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  81. [89]

    P., Bachmann, G., and Hofmann, T

    Singh, S. P., Bachmann, G., and Hofmann, T. Analytic insights into structure and rank of neural network hessian maps. Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  82. [90]

    B., Poggio, T

    Singhal, U., Cheung, B., Chandra, K., Ragan-Kelley, J., Tenenbaum, J. B., Poggio, T. A., and Yu, S. X. How to guess a gradient. ar X iv 2312.04709, 2023

    2023 arXiv

  83. [91]

    and Sun, K

    Soen, A. and Sun, K. Tradeoffs of diagonal fisher information matrix estimators. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  84. [92]

    Dropout: A simple way to prevent neural networks from overfitting

    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR), 2014

    2014

  85. [93]

    Steinhaus, H. et al. Sur la division des corps mat \'e riels en parties. Bulletin of the Polish Academy of Sciences, 1956

    1956

  86. [94]

    Debiasing mini-batch quadratics for applications in deep learning, 2024

    Tatzel, L., Mucsányi, B., Hackel, O., and Hennig, P. Debiasing mini-batch quadratics for applications in deep learning, 2024

    2024

  87. [95]

    Tropp, J. A. An introduction to matrix concentration inequalities. Found. Trends Mach. Learn., 8 0 (1–2): 0 1–230, May 2015. doi:10.1561/2200000048

    2015 doi

  88. [96]

    A., Yurtsever, A., Udell, M., and Cevher, V

    Tropp, J. A., Yurtsever, A., Udell, M., and Cevher, V. Streaming low-rank matrix approximation with an application to scientific simulation. SIAM Journal on Scientific Computing, 2019

    2019

  89. [97]

    van der Ouderaa, T. F. A., Nagel, M., van Baalen, M., Asano, Y. M., and Blankevoort, T. The LLM surgeon. In International Conference on Learning Representations (ICLR), 2024

    2024

  90. [98]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  91. [99]

    E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt , S

    Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt , S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. J...

    2020

  92. [100]

    Williams, C. K. and Rasmussen, C. E. Gaussian processes for machine learning. MIT press Cambridge, MA, 2006

    2006

  93. [101]

    An improved empirical fisher approximation for natural gradient descent

    Wu, X., Yu, W., Zhang, C., and Woodland, P. An improved empirical fisher approximation for natural gradient descent. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  94. [102]

    X., Robeyns, M., Wang, X., and Aitchison, L

    Yang, A. X., Robeyns, M., Wang, X., and Aitchison, L. B ayesian low-rank adaptation for large language models. In International Conference on Learning Representations (ICLR), 2024

    2024

  95. [103]

    Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. W. Py H essian: Neural networks through the lens of the H essian. In IEEE International Conference on Big Data, 2020

    2020

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.