{"id":"7dbbc5b4-c811-44e4-ab71-26d45d6ce750","arxiv_id":"2501.19183","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Linear operators are argued to be the right abstraction for curvature matrices in deep learning, and the curvlinops library shows that one interface can cover exact and approximate curvature across many applications.","lead":"This position paper argues that curvature matrices of neural networks should be exposed through linear operators, an interface that computes matrix-vector products without building the full matrix. It backs the position with curvlinops, a PyTorch library that unifies Hessian, Gauss-Newton, Fisher, and KFAC curvature operators, and demonstrates the interface on applications and large models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The interface hides implementation mechanics but not the consequential choice of which curvature matrix to use; Section 5 concedes users may silently pick an inappropriate operator, undermining the 'without worrying' claim.","rationale":"The reader's weakest assumption is correct and is the most load-bearing issue in the paper. The central claim has two parts: (1) linear operators hide implementation complexity, and (2) this empowers users to apply curvature matrices without worry. Part (1) is well supported by the library's design and benchmarks. Part (2) requires that the remaining user-facing decisions — which curvature matrix, which GGN split, which Fisher type, which damping — are either obvious, safe, or validated. The paper's own Section 5 states that users may unknowingly use an inappropriate operator, and the API examples in Section 3 require explicit selection of operator class and configuration. Therefore the 'without worrying' wording is too strong; the abstraction moves the burden from implementation to definitional choice. This is not a mathematical flaw in the library, but it is a gap between the position's promise and its delivery. A targeted experiment comparing a correct and an incorrect operator in a concrete application (influence functions) and checking for warnings would empirically settle whether the concern lands. Since the reader already identified this and assigned CONDITIONAL, no verdict change is needed.","tokens_in":19491,"tokens_out":5972,"duration_ms":54979,"concrete_test":"Using curvlinops on a small model (e.g., an MLP on MNIST), compute influence-function scores (Section 3, Application 2) two ways: (a) with an EKFACLinearOperator using the damping recommended by Grosse et al. (2023), and (b) with an EmpiricalFisherLinearOperator created via the same constructor with default kwargs, which a non-expert could plausibly choose. Compare the top-ranked influential training examples (e.g., intersection over union or rank correlation). If the two rankings differ substantially and the library emits no warning about the choice of curvature type, then the interface permits exactly the silent misuse that Section 5 warns about, weakening the democratization claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that presenting curvature matrices as linear operators 'empowers users to apply them to various applications without worrying about implementation complexity' (Section 1). For this to hold, the interface must protect users not only from mechanical complexity (HVP/GGNVP/FVP details) but also from the more consequential task of choosing which curvature matrix their application requires. Section 2.1 shows GGN, type-I/II Fisher, MC Fisher, empirical Fisher, and KFAC represent different mathematical objects, and Section 5 concedes: 'In both cases, the user might unknowingly use a curvature linear operator that is inappropriate for their application.' The API in Section 3 requires the user to select an operator class (Hessian, GGN, KFAC, etc.) and pass kwargs (e.g., GGN split, Fisher type, damping) with no validation against the downstream application's requirements. A non-expert user who picks the wrong operator gets silently wrong results, not a warning. Thus 'without worrying' fails precisely at the conceptual level, and 'democratization' becomes a source of silent errors. This is the load-bearing concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper argues that presenting curvature matrices such as the Hessian, GGN, Fisher variants, and KFAC as linear operators—i.e., as black-box matrix-vector product interfaces—provides a general, scalable, and user-friendly abstraction. The argument is supported by curvlinops, a PyTorch library that implements these operators, offers composition and transformation, exports to SciPy, and provides randomized linear algebra estimators. The manuscript illustrates the interface on five applications, reports runtime and memory benchmarks on ResNet50/ImageNet and nanoGPT/Shakespeare, validates estimators on toy problems, and includes a candid alternative-views section that discusses limitations.","tokens_in":19687,"tokens_out":10854,"duration_ms":107964,"significance":"If the position is accepted, a single linear-operator interface could unify exact and approximate curvature matrices across optimization, influence functions, pruning, merging, and spectral analysis, while connecting deep learning to mature randomized linear algebra and SciPy tooling. The manuscript ships a pip-installable library, reproducible code snippets and toy validations, and performance measurements that are broadly consistent with published HVP/GGNVP cost factors; the central claim is independently checkable and is not circular. The main gap is that the 'democratization' and 'user-friendly' wording claims more than the evidence establishes: the interface hides computational mechanics but leaves the consequential choice of curvature object to the user, and the paper's own Section 5 concedes that an inappropriate choice can be made unknowingly. There is also a concrete mathematical slip in the pseudo-loss derivation for Fisher-vector products (Section 2.1) that should be corrected. These issues are fixable, and the manuscript's strengths—transparency, reproducible software, and a clear design argument—make it a valuable position statement.","major_comments":[{"comment":"In the paragraph 'Fisher-vector products (FVPs) via GGNVPs', the pseudo-loss is defined as \\ell~(f_n,y_n) = f_n^T g_n g_n^T f_n. Its Hessian with respect to f_n is 2 g_n g_n^T, not g_n g_n^T as claimed immediately after the definition. Consequently, the GGN of the stated pseudo-loss is twice the desired Fisher, MC-Fisher, or empirical Fisher. Please correct the definition to \\ell~(f_n,y_n) = (1/2) f_n^T g_n g_n^T f_n, or state explicitly that the implementation rescales by 1/2. This is a load-bearing mathematical error because a user implementing the described recipe would obtain incorrectly scaled Fisher-vector products.","section":"Section 2.1"},{"comment":"The central claim that users can apply curvature matrices 'without worrying about implementation complexity' and the title's 'Democratized' overreach what the interface actually provides. Section 2.1 and the Step 1 API in Section 3 require users to choose among Hessian, GGN, Fisher, MC-Fisher, empirical Fisher, and KFAC and to set definitional kwargs such as GGN split, Fisher type, and damping. Section 5 explicitly concedes that the user 'might unknowingly use a curvature linear operator that is inappropriate for their application.' The abstraction therefore does not free users from the conceptually consequential choice of curvature matrix, and a wrong choice yields silently wrong results rather than a warning. The paper should either narrow the claim throughout to 'without worrying about implementation complexity' in the narrow computational sense, or add safeguards and guidance—e.g., operator-application compatibility checks and warnings about definitional choices—to make the democratization claim credible.","section":"Section 1 and Section 5"},{"comment":"The claim that HVP performance 'matches related work' is stated from ranges (4.5–5.5 gradient equivalents observed versus 4–5 reported by Dagréou et al.) that are close but not identical, with no confidence intervals, number of repeats, or matching of setup details such as batch size and model family. Since the scalability argument is part of the paper's support, please report dispersion and repeat counts, and soften the claim to 'is consistent with' rather than 'matches' if the conditions are not directly comparable.","section":"Appendix A"}],"minor_comments":[{"comment":"The sentence 'In this paper, we argue our position that;' is a grammatical fragment and should be rephrased, e.g., 'In this paper, we argue the following position: ...'","section":"Section 1"},{"comment":"The spelling 'Naïve' and 'naive' appear inconsistently; please choose one convention throughout.","section":"Section 2.1"},{"comment":"In the model-merging snippet, the variable 'Fs' is constructed with GGNLinearOperator but described as 'Per-task Fisher matrices'. Please state the conditions under which the GGN equals the Fisher (or rename the variables) to avoid implying that the two are interchangeable unconditionally.","section":"Section 3, Application 3"},{"comment":"The top two panels on spectral density do not state the number of Monte-Carlo repetitions used, while the bottom panels report medians over 200 runs; please add this information for reproducibility.","section":"Figure 4 caption"}],"recommendation":"major_revision","confidential_remarks":"This is a legitimate position-plus-software paper, and the library appears to be a real contribution. The main revision should address the pseudo-loss scaling error in Section 2.1 and the overstatement of the democratization claim in light of Section 5's own admission about silent misuse. The stress-test concern about non-expert users selecting the wrong curvature matrix does land, but it can be answered by carefully scoping the claim and adding guidance or safeguards rather than by rejecting the paper. I have no concerns about novelty, citation patterns, or author conduct."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Felix — you'll want to know this one because it's a position paper that actually ships code. The contribution is curvlinops, a PyTorch library that exposes Hessian, GGN, MC/empirical Fisher, and (E)KFAC as one linear-operator interface, with deterministic checks and export to SciPy. The abstraction itself is old news — SciPy, PyLops, CoLA, Lineax, GPyTorch — and previous curvature packages cover pieces. What's new is the breadth under one interface and the honest measurements. Figure 3's HVP costs match the published numbers from Dagréou et al., and the KFAC overhead of 1.5–2.5 gradients is useful information. Figure 4 reproduces known toy results for trace/diagonal/spectral-density estimators. That's genuine community service.\n\nThe paper also does something rare: the Alternative Views section (Section 5) is not a footnote. It concedes that encapsulating curvature can duplicate computation, that the interface may be too minimal, and — the important one — that hiding complexity may promote incorrect usage. The stress-test note you sent is right about this: the interface hides the mechanics of HVP/GGNVP/FVP but not the consequential choice of which curvature matrix your application needs. A user who picks the wrong operator gets silently wrong numbers. The caveat is that the paper itself says this, in Section 5, in almost those words. So it's not a hidden flaw; it's a limitation the authors know about and leave unresolved. For a position paper, that's tolerable, but it does punch a hole in the 'without worrying' phrase. The claim should really be 'without worrying about implementation mechanics, provided you know which curvature you need.' If the authors want the democratization claim to be stronger, they'd need validation in the interface or a decision guide.\n\nMinor soft spots: 'user-friendly' and 'democratized' are not measured — no user study, just code snippets. The library is PyTorch-only; forward-mode AD is absent, and nanoGPT's attention had to be switched to a non-efficient path for double backprop. No commit-level artifact is pinned in the preprint, though figures have code links. None of these are fatal.\n\nWho is this for? Practitioners who want curvature without reimplementing it, and researchers in second-order optimization, influence functions, pruning, or loss-landscape analysis. It deserves a serious referee. I'd send it to review with a request to soften the 'without worrying' language and to add a short section on choosing the right operator. My own verdict on the position: it holds, with the misuse caveat named.","headline":"A position paper with a real library behind it: the linear-operator claim holds, but the 'without worrying' framing is oversold and the paper's own Section 5 admits it.","tokens_in":20254,"tokens_out":2479,"would_cite":true,"duration_ms":22423,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that exposing curvature matrices as linear operators—objects that multiply vectors without ever materializing the matrix—unifies exact and approximate curvature computation, as demonstrated by the curvlinops library.","keywords":["linear operators","curvature matrices","Hessian","generalized Gauss-Newton","Fisher information matrix","KFAC","randomized linear algebra","neural networks"],"falsifier":"Run a controlled user study in which practitioners with limited experience in second-order optimization implement a standard task, such as influence-function attribution or Fisher-weighted model merging, given only the linear-operator interface; then compare the curvature matrices they select and the correctness of their results against a reference expert implementation. If a substantial fraction of non-expert users accidentally choose an empirical Fisher where a generalized Gauss-Newton or damped inverse is required, and the library's checks do not catch it, the democratization claim is falsified.","tokens_in":19285,"feed_emoji":"🧮","tokens_out":8361,"duration_ms":78961,"temperature":0.7,"pith_summary":"This position paper argues that curvature matrices—the Hessian of a neural network's empirical risk and its cheaper approximations, such as the generalized Gauss-Newton, Fisher information, and KFAC—should be presented to machine-learning researchers as linear operators, objects that answer matrix-vector products without ever building the full matrix. To back this claim, the paper introduces curvlinops, a library that wraps exact and approximate curvature matrices in one common interface. It argues that this interface delivers three benefits: it hides error-prone implementation details such as automatic differentiation, batch scaling, and stochasticity; it reduces applications like second-order optimization, influence functions, model merging, and pruning to a few lines of code that mirror the underlying mathematics; and it makes existing linear-algebra tooling, including iterative eigensolvers and randomized estimators of trace, diagonal, and spectral density, directly usable. If the position is right, a single abstraction could replace much of the bespoke curvature code in deep learning.","feed_headline":"Make every curvature matrix a matrix-vector product","feed_subtitle":"A unified linear-operator abstraction could let AI developers swap exact and approximate Hessians without reimplementation.","key_machinery":"The load-bearing object is the linear operator: an interface that answers the matrix-vector product $A(v)$ for a matrix $A$ and satisfies the two linearity laws $A(v_1+v_2)=A(v_1)+A(v_2)$ and $A(\\alpha v)=\\alpha A(v)$. Its work in the argument is to make the representation the same regardless of how the curvature is computed: the Hessian uses nested automatic differentiation, the generalized Gauss-Newton uses vector-Jacobian and Jacobian-vector products, Fisher-vector products are obtained from a Gauss-Newton product on a detached pseudo-loss $f^\\top g g^\\top f$, and KFAC multiplies vectors through sums of Kronecker products of layer factors. Because every curvature variant lives behind the same multiplication interface, users can compose and invert operators, and swap exact for approximate curvature, without changing their application code.","core_discovery":"The central claim is a design claim, not a new theorem: a single linear-operator interface $v \\mapsto A(v)$, where $A$ is a linear map acting like a matrix $A$ but never materializing it, is general enough to cover the curvature matrices that matter in deep learning and convenient enough that application code can treat them like dense matrices. The paper demonstrates this with curvlinops, which provides operators for the Hessian, the generalized Gauss-Newton, the Monte-Carlo and empirical Fisher, and several (E)KFAC variants, and shows that a user can write the same style of code for Newton-CG, influence-function inversion, Fisher-weighted model merging, pruning, and Hessian spectral analysis. It also argues that the interface is extensible, because new curvature approximations are just new operators, and interoperable, because operators can be composed, transformed (for example inverted through iterative solvers), exported to established numerical linear-algebra routines, and connected to randomized linear algebra for estimating matrix properties at scale.","pith_inferences":["A natural extension the paper leaves implicit is to attach a machine-readable 'curvature pedigree' to each operator—specifying whether it is a Hessian, GGN, type-I or type-II Fisher, empirical Fisher, its damping, and its data reduction—so that automated checks can veto an operator that is mathematically inappropriate for the requested application.","The same operator abstraction would transfer beyond curvature to any structured matrix used in deep learning, such as attention, kernel, or factorized weight matrices, potentially giving the broader field the same interoperability with randomized linear algebra.","If the interface becomes a standard, an empirical prediction follows: the dominant remaining source of user error in curvature-based applications will shift from implementing matrix-vector products to choosing the right curvature variant, which would make documentation and validation layers more important than the operator itself."],"forward_implications":["Second-order optimizers and their approximate variants can be implemented once against the operator interface, with Newton-CG, Neumann-series inversion, and KFAC inverses selected by swapping operators.","Influence functions, bi-level optimization, and hyper-gradient methods can use full inverse curvature-vector products rather than being forced onto diagonal approximations.","Fisher-weighted model merging can move beyond diagonal Fishers to full or block-structured Fisher operators via conjugate-gradient solves.","Randomized linear algebra estimators for trace, diagonal, spectral density, and low-rank sketches become immediately applicable to any curvature matrix, not just one bespoke implementation.","Curvature estimators maintained inside optimizers can be exported as operators at any point of training and reused for pruning, uncertainty quantification, or landscape analysis."],"supporting_citations":[{"why":"Supplies the nested-autodiff Hessian-vector product that the Hessian operator is built on.","marker":"Pearlmutter (1994)"},{"why":"Establishes Hessian-free Newton-CG and the cost comparison of GGN-vector products, backing the optimization and performance claims.","marker":"Martens (2010)"},{"why":"Provides fast GGN and Fisher matrix-vector products via vector-Jacobian and Jacobian-vector products, the base of the Fisher-vector product implementation.","marker":"Schraudolph (2002)"},{"why":"Introduces KFAC and its damping heuristic, one of the library's central approximate curvature operators.","marker":"Martens & Grosse (2015)"},{"why":"Documents the difference between the empirical Fisher and the Fisher, the main correctness caveat for the interface.","marker":"Kunstner et al. (2019)"},{"why":"Explains the GGN-Fisher equivalence and the split choices that make curvature definitions non-obvious.","marker":"Martens (2020)"},{"why":"Defines influence functions through inverse Hessian-vector products, a flagship application of the operator interface.","marker":"Koh & Liang (2017)"},{"why":"Introduces Fisher-weighted model merging, an application the paper scales beyond diagonal approximations.","marker":"Matena & Raffel (2022)"},{"why":"Provides the empirical cost measurements for Hessian-vector products that the scalability discussion relies on.","marker":"Dagréou et al. (2024)"},{"why":"Introduces the stochastic trace estimator that the library's trace and diagonal estimation routines build on.","marker":"Hutchinson (1989)"}],"fun_headline_variants":["One operator interface to rule all curvature matrices","Unify Hessians with linear operators","Curvature matrices via linear operators, no materialization","Democratize curvature: just matrix-vector products"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on the assumption that a user who cannot see the internal details can still choose the curvature matrix that is mathematically appropriate for their application, so that hiding complexity helps rather than silently misleads.","fun_headline_variants_meta":{"raw":{"variants":["One operator interface to rule all curvature matrices","Unify Hessians with linear operators","Curvature matrices via linear operators, no materialization","Democratize curvature: just matrix-vector products"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1283,"prompt_tokens":905,"completion_tokens":378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":319}},"tokens_in":521,"tokens_out":378,"duration_ms":4192,"temperature":1.0,"reasoning_tokens":319,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T21:00:37.778843+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled user study in which practitioners with limited experience in second-order optimization implement a standard task, such as influence-function attribution or Fisher-weighted model merging, given only the linear-operator interface; then compare the curvature matrices they select and the correctness of their results against a reference expert implementation. If a substantial fraction of non-expert users accidentally choose an empirical Fisher where a generalized Gauss-Newton or damped inverse is required, and the library's checks do not catch it, the democratization claim is falsified.","supporting_citations":[{"cited_title":"Deep learning via H essian-free optimization","cited_arxiv_id":null,"evidence_quote":"Establishes Hessian-free Newton-CG and the cost comparison of GGN-vector products, backing the optimization and performance claims."},{"cited_title":"Limitations of the empirical fisher approximation for natural gradient descent","cited_arxiv_id":null,"evidence_quote":"Documents the difference between the empirical Fisher and the Fisher, the main correctness caveat for the interface."},{"cited_title":"New insights and perspectives on the natural gradient method, 2020","cited_arxiv_id":null,"evidence_quote":"Explains the GGN-Fisher equivalence and the split choices that make curvature definitions non-obvious."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines influence functions through inverse Hessian-vector products, a flagship application of the operator interface."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Fisher-weighted model merging, an application the paper scales beyond diagonal approximations."},{"cited_title":"A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines","cited_arxiv_id":null,"evidence_quote":"Introduces the stochastic trace estimator that the library's trace and diagonal estimation routines build on."}],"review_version":1}