REVIEW 4 major objections 6 minor 25 references
Neural Preconditioning via Krylov Subspace Geometry
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A neural preconditioner trained through differentiable FGMRES reduces average Krylov iterations on 3D-1D PDE systems from 147.5 to 12.86.
desk verdict Plausible two-stage neural preconditioner whose headline dynamic-tuning gain is confounded by a missing control arm, but the core idea is worth a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key object is the dynamic loss functional $L_{\mathrm{dynamic}}^{(M)}(\theta) = \frac{1}{M}\sum_{j=1}^M \langle |s^\mu_{j,\theta}| \rangle_\mu$, where $s^\mu_{j,\theta}$ are the sines of the angles between the residual and the Krylov subspace generated by FGMRES with the neural preconditioner $N_\theta$ on system $(A_\mu, b_\mu)$. This loss is evaluated by a differentiable implementation of the Arnoldi-Givens process, so gradients flow through the solver iterations; the recurrence $\|r_j\| = s_j \|r_{j-1}\|$ is what ties these angles to actual convergence, and the window parameter $M$ controls how much of the early solver dynamics the training supervises.
What would settle it
Train two versions of the same preconditioner on the same problem family, one with the average-sine loss and one with the log-product loss over the same window, and compare validation FGMRES iteration counts. If any checkpoint shows the average sine decreasing while the product $\prod_{j=1}^M |s_{j,\theta}|$ or the measured iteration count increases, the surrogate link is broken.
Extended reading notes
Core claim
The central claim is that minimizing the arithmetic mean of the first-M sine angles $|s_{j,\theta}|$ between residuals and Krylov subspaces, evaluated on the fly during a differentiable Flexible GMRES run, is an effective training signal for neural preconditioners. The paper shows that these sine values are available through the Arnoldi process and Givens rotations, that they are differentiable with respect to the preconditioner parameters, and that they connect directly to the residual recurrence $\|r_j\| = |s_j| \,\cdots\, |s_1|\,\|r_0\|$. The two-stage strategy produces a preconditioner that improves early-stage convergence, reduces iteration counts by more than an order of magnitude on the tested problem family, and generalizes to previously unseen geometric configurations of the embedded 1D domain.
Load-bearing premise
The method's success rests on the assumption that reducing the arithmetic mean of the first $M$ sine angles, which is what the loss directly minimizes, also reduces the product of those sines, which is what actually controls GMRES convergence; the paper's one-sided inequality does not guarantee that.
Editorial extensions
If this is right
- FGMRES iteration counts on the tested 3D-1D problem family fall from an average of 147.5 to 26.71 with static pretraining and to 12.86 after dynamic fine-tuning, a more than tenfold reduction.
- The dynamic loss concentrates improvement in the first $M$ iterations: the mean relative residual at iteration 10 drops from $1.12 \times 10^{-3}$ at epoch 0 to $1.3 \times 10^{-5}$ at the final epoch.
- The estimated convergence factor $\left(\frac{1}{M}\sum_{j=1}^M \langle |s^\mu_{j,\theta}| \rangle_\mu\right)^M$ decreases from $5.36 \times 10^{-3}$ to $1.0 \times 10^{-5}$ during fine-tuning, matching the observed residual-decay acceleration.
- The window size $M$ gives a tunable trade-off: small $M$ focuses on early convergence where computational savings are largest, while larger $M$ would supervise longer-term solver behavior at higher memory cost.
- Because the loss is defined through the FGMRES iteration itself, the trained preconditioner specializes in manipulating Krylov subspace geometry rather than merely reducing the residual norm in one step.
Reading between the lines
- A testable extension is to replace the arithmetic-mean sine loss with the log-product loss over the same window; the paper's own inequality only bounds the product by a power of the mean, so a product-based loss may be more directly aligned with the actual contraction factor.
- The differentiable FGMRES machinery could be reused to train other solver components, such as initial guesses, deflation spaces, or inner-outer iteration policies, not only preconditioners.
- The reported gains may depend on the spectral and geometric regularity of this specific 3D-1D problem family; on matrices with non-monotonic sine profiles, minimizing the mean could in principle raise the product and slow convergence.
- The convolutional U-Net restricts the method to structured tensor-product meshes; extending the same loss to graph-based or mesh-informed architectures, as the paper suggests, is the natural path toward unstructured domains.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-stage training strategy for neural preconditioners for parametrized linear systems from 3D-1D mixed-dimensional PDEs. Stage one minimizes a static residual loss; stage two fine-tunes the network with a dynamic loss defined as the average of the sines of the principal angles between residuals and Krylov subspaces, computed through a differentiable Flexible GMRES implementation. On a test family of 100 graph geometries, the authors report mean FGMRES iteration counts dropping from 147.5 unpreconditioned to 26.71 after static pretraining and to 12.86 after dynamic fine-tuning. The central claim is that the dynamic Krylov-geometry loss is responsible for the additional improvement.
Significance. If the dynamic fine-tuning effect is real and correctly attributed, the paper offers a principled, solver-aware loss that connects neural network training to Krylov subspace geometry, and the differentiable FGMRES formulation is a useful technical contribution. The problem family is relevant and the reported iteration-count reductions are large. The paper also benefits from a clear separation of the static and dynamic phases and from reporting generalization over unseen graph instances. However, the causal attribution of the final improvement is currently underdetermined by the experimental design, and the theoretical link between the loss and the convergence factor is only one-sided.
major comments (4)
- [Sections 6.2-6.3, Tables 2-3, Section 7.4] The paper attributes the iteration-count reduction from 26.71 to 12.86 to the dynamic loss L_dynamic, but there is no control arm that continues static training for the same 150 additional epochs on the same fine-tuning data. The two phases differ in three ways at once: loss functional, training dataset size (480 augmented graphs vs 100 unaugmented graphs), and number of epochs. Any of these differences, not only the loss, could produce the reported gain. A control experiment with L_static on the same 100-graph fine-tuning set for 150 epochs is necessary to support the causal claim stated in Section 8.
- [Section 5, Eqs. (4)-(5)] The dynamic loss is the arithmetic mean of the sine angles, while the FGMRES convergence factor is the product in Eq. (3). The inequality ∏⟨|s_j|⟩ ≤ (L_dynamic)^M is a bound, not an equivalence; two profiles with the same mean can have very different products. Equation (5) then estimates the product from the same fitted angle averages that define the loss, so the reported two-order-of-magnitude drop in this estimate cannot serve as independent evidence of improved convergence. Please provide a direct empirical relation between L_dynamic and actual FGMRES iterations, or optimize a product-based loss instead of the arithmetic mean.
- [Section 7.2, Figs. 3-5, Table 4] Part of the evidence for the dynamic phase is that the average sine angles and the dynamic loss decrease during fine-tuning. Since the loss is exactly the arithmetic mean of those sine angles, this decrease is expected by construction and is not independent evidence that the solver improves. The independent quantity, the iteration count, is currently confounded with the missing control described above. Please separate the optimization history from the evaluation metric when reporting the dynamic-phase results.
- [Section 7.4] The evaluation reports only mean iteration counts and ranges over 100 test graphs. There are no standard deviations, confidence intervals, or wall-clock timings, and there is no comparison with classical preconditioners (e.g., the block or algebraic multigrid methods cited in Refs. [10-12]). Because a learned preconditioner adds a forward pass per FGMRES iteration, a reduction in iteration count does not automatically imply a reduction in time to solution; the practical claim needs timing data. The absence of uncertainty measures also weakens the stated robustness and generalization claims.
minor comments (6)
- [Section 4] There is a typo: "define the the principal angle" should read "define the principal angle".
- [Section 5, Algorithm 1] The text refers to "line 11 in Algorithm 5", but only Algorithm 1 exists; this should be corrected.
- [Section 7.1 vs Table 2] Section 7.1 states that static training ran for 100 epochs, while Table 2 reports Epochs = 250; please reconcile this discrepancy.
- [Tables 4 and 5] The quantities ∆+ and ∆− are reported in both tables but are never defined in the text, and the in-text references to "Table 7.2.2" and "Table 7.2.1" do not match the actual table labels.
- [Eq. (5)] The notation in Eq. (5) is ambiguous: the product over j of the averaged sines is not the same as the average of the product, and the order of averaging should be stated explicitly.
- [Section 6.3] Please report the wall-clock time and hardware used for the fine-tuning phase, as well as optimizer details such as Adam betas, weight decay, and the learning-rate schedule, to improve reproducibility.
Circularity Check
Partial circularity: the reported convergence-factor drop in Eq. (5) is just the dynamic training loss raised to the M-th power; the independent FGMRES iteration counts keep the central claim from being fully circular.
-
self definitional
[Section 7.2.1, Eq. (5) and Table 4; compare Eq. (4)]
"Assuming that a is negligible, we may estimate the product of sines governing the convergence factor via (1/M Σ_j ⟨|s_µ_j,θ|⟩_µ)^M, which approximates the asymptotic bound for the Krylov Subspace method convergence. By looking at the estimate above, we can observe a substantial drop (two orders of magnitude) of its value, from 5.3 × 10−3 to 1.0 × 10−5, experienced during the dynamic training stage."
Eq. (4) defines L_dynamic(M)(θ) = (1/M) Σ_j ⟨|s_µ_j,θ|⟩_µ. The right-hand side of Eq. (5) is therefore exactly L_dynamic(M)(θ)^M. Since the dynamic fine-tuning phase minimizes L_dynamic by gradient descent, this 'estimate' of the convergence factor is a monotone transform of the training loss; its drop is forced by the optimization and cannot serve as independent evidence that the product of sines improved. The paper's independent evidence is the external FGMRES iteration-count comparison (147.5 to 26.71 to 12.86) over unseen test graphs, which does not reduce to the loss definition. Hence the circularity is partial, confined to the sine-based convergence estimate.
full rationale
The derivation chain has one degenerate link: the convergence-factor estimate in Eq. (5) is literally the dynamic loss of Eq. (4) raised to the M-th power, so its decrease (Table 4) is a restatement of the training objective and cannot fail as evidence. However, the paper's central practical claim is the FGMRES iteration count on 100 unseen test graph configurations (Section 7.4), which is not a transform of the loss and does not reduce to the training objective. The static-pretraining component is taken from the authors' prior work [17], but it is re-trained and benchmarked within this paper, so that self-citation is not load-bearing. The missing control for 'extra training only' is a genuine experimental confound, but it is not a logical circularity under the enumerated patterns; the surrogate gap between the arithmetic mean and the product is a correctness and robustness concern, not a circular reduction. The one-sided inequality offered as a 'theoretical guarantee' does not by itself prove that lowering the arithmetic mean lowers the product, but that is an over-claim rather than a circularity. Overall: partial circularity in the sine-based evidence, independent content in the iteration-count evidence.
Assumptions & free parameters
free parameters (3)
- neural network weights theta =
not provided (U-Net with roughly 1e6 parameters)
- dynamic loss horizon M =
10
- static residual scaling alpha =
not reported numerically
assumptions (4)
- standard math GMRES residual norm recurrence ||r_j|| = s_j ... s_1 ||r_0|| for the FGMRES process with the learned preconditioner.
- ad hoc to paper The arithmetic-mean sine loss is a reliable proxy for the product-of-sines convergence factor.
- domain assumption The training right-hand sides and random vascular graphs are representative of the whole parameter family P.
- domain assumption Backpropagation through the unrolled Arnoldi-Givens computation yields usable gradients for the U-Net parameters.
Cite this review
Pith. "Pith review of Neural Preconditioning via Krylov Subspace Geometry." pith.science (2026). https://pith.science/paper/LBG52634
@misc{pith2026250715452,
author = {Pith},
title = {Pith review of: Neural Preconditioning via Krylov Subspace Geometry},
year = {2026},
howpublished = {\url{https://pith.science/paper/LBG52634}},
note = {Machine review of arXiv:2507.15452}
}
read the original abstract
We propose a geometry-aware strategy for training neural preconditioners tailored to parametrized linear systems arising from the discretization of mixed-dimensional partial differential equations (PDEs). These systems are typically ill-conditioned because of the presence of embedded lower-dimensional structures and are solved using Krylov subspace methods. Our approach yields an approximation of the inverse operator employing a learning algorithm consisting of a two-stage training framework: an initial static pre-training phase, based on residual minimization, followed by a dynamic fine-tuning phase that incorporates solver convergence dynamics into training via a novel loss functional. This dynamic loss is defined by the principal angles between the residuals and the Krylov subspaces. It is evaluated using a differentiable implementation of the Flexible GMRES algorithm, which enables backpropagation through both the Arnoldi process and Givens rotations. The resulting neural preconditioner is explicitly optimized to improve early-stage convergence and reduce iteration counts in a family of 3D-1D mixed-dimensional problems with geometric variability of the 1D domain. Numerical experiments show that our solver-aligned approach significantly improves convergence rate, robustness, and generalization.
Reference graph
Works this paper leans on
-
[1]
application to tissue perfusion problems
D’Angelo, C., Quarteroni, A.: On the coupling of 1d and 3d diffusion-reaction equations. application to tissue perfusion problems. Mathematical Models and Methods in Applied Sciences 18(8), 1481–1504 (2008)
work page 2008
-
[2]
Boon, W.M., Nordbotten, J.M., Vatne, J.E.: Functional analysis and exterior calculus on mixed-dimensional geometries. Ann. Mat. Pura Appl.200(2), 757–789 (2021)
work page 2021
-
[3]
SIAM Journal on Numerical Analysis 59(1), 558–582 (2021)
Kuchta, M., Laurino, F., Mardal, K.-A., Zunino, P.: Analysis and approximation of mixed-dimensional pdes on 3d-1d domains coupled with lagrange multipliers. SIAM Journal on Numerical Analysis 59(1), 558–582 (2021)
work page 2021
-
[4]
Mathematical Models and Methods in Applied Sciences 33(12), 2425–2462 (2023)
Heltai, L., Zunino, P.: Reduced lagrange multiplier approach for non-matching coupling of mixed-dimensional domains. Mathematical Models and Methods in Applied Sciences 33(12), 2425–2462 (2023)
work page 2023
-
[5]
Saad, Y.: Iterative Methods for Sparse Linear Systems. Applied Mathematics. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA (2003)
work page 2003
-
[6]
Numerical Linear Algebra with Applications 18(1), 1–40 (2011)
Mardal, K.-A., Winther, R.: Preconditioning discretizations of systems of partial differential equations. Numerical Linear Algebra with Applications 18(1), 1–40 (2011)
work page 2011
-
[7]
Chen, K.: Matrix Preconditioning Techniques and Applications vol. 19. Cam- bridge University Press, Cambridge, UK (2005)
work page 2005
-
[8]
Elsevier, Amsterdam, The Netherlands (2000)
Trottenberg, U., Oosterlee, C.W., Schuller, A.: Multigrid. Elsevier, Amsterdam, The Netherlands (2000)
work page 2000
Show all 25 references
-
[9]
Oxford University Press, Oxford, UK (1999)
Quarteroni, A., Valli, A.: Domain Decomposition Methods for Partial Differential Equations. Oxford University Press, Oxford, UK (1999)
1999
-
[10]
SIAM Journal on Scientific Computing 38(6), 962–987 (2016)
Kuchta, M., Nordaas, M., Verschaeve, J.C.G., Mortensen, M., Mardal, K.-A.: Preconditioners for saddle point systems with trace constraints coupling 2d and 1d domains. SIAM Journal on Scientific Computing 38(6), 962–987 (2016)
2016
-
[11]
SIAM Journal on Scientific Computing 46(3), 1461–1486 (2024) 22
Budiˇ sa, A., Hu, X., Kuchta, M., Mardal, K.-A., Zikatanov, L.: Algebraic multi- grid methods for metric-perturbed coupled problems. SIAM Journal on Scientific Computing 46(3), 1461–1486 (2024) 22
2024
-
[12]
Computer Methods in Applied Mechanics and Engineering 431, 117256 (2024)
Firmbach, M., Steinbrecher, I., Popp, A., Mayr, M.: An approximate block factor- ization preconditioner for mixed-dimensional beam-solid interaction. Computer Methods in Applied Mechanics and Engineering 431, 117256 (2024)
2024
-
[13]
SIAM Journal on Scientific Computing 45(3), 127–151 (2023)
Azulay, Y., Treister, E.: Multigrid-augmented deep learning preconditioners for the helmholtz equation. SIAM Journal on Scientific Computing 45(3), 127–151 (2023)
2023
-
[14]
SIAM Journal on Scientific Computing 47(1), 151–181 (2025)
Kopaniˇ c´ akov´ a, A., Karniadakis, G.E.: Deeponet based preconditioning strategies for solving parametric linear systems of equations. SIAM Journal on Scientific Computing 47(1), 151–181 (2025)
2025
-
[15]
Nature Machine Intelligence 3, 218–229 (2021)
Lu, L., Jin, P., Pang, G., Zhang, Z., Karniadakis, G.: Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence 3, 218–229 (2021)
2021
-
[16]
Communications on Applied Mathematics and Computation, 1–38 (2024)
Xu, Z.-Q.J., Zhang, Y., Luo, T.: Overview frequency principle/spectral bias in deep learning. Communications on Applied Mathematics and Computation, 1–38 (2024)
2024
-
[17]
https://arxiv.org/abs/2505.08491
Dimola, N., Franco, N.R., Zunino, P.: Numerical Solution of Mixed-Dimensional PDEs Using a Neural Preconditioner (2025). https://arxiv.org/abs/2505.08491
2025 arXiv
-
[18]
arXiv stat.ML(2305.19638) (2024)
Williams, C., Falck, F., Deligiannidis, G., Holmes, C., Doucet, A., Syed, S.: A unified framework for u-net design and analysis. arXiv stat.ML(2305.19638) (2024)
2024 arXiv
-
[19]
arXiv e-prints, 1502 (2015)
Gunes Baydin, A., Pearlmutter, B.A., Andreyevich Radul, A., Siskind, J.M.: Automatic differentiation in machine learning: a survey. arXiv e-prints, 1502 (2015)
2015
-
[20]
SIAM Journal on Scientific Computing 14(2), 461–469 (1993)
Saad, Y.: A flexible inner-outer preconditioned gmres algorithm. SIAM Journal on Scientific Computing 14(2), 461–469 (1993)
1993
-
[21]
ESAIM: Mathematical Modelling and Numerical Analysis 53(6), 2047–2080 (2019)
Laurino, F., Zunino, P.: Derivation and analysis of coupled pdes on manifolds with high dimensionality gap arising from topological model reduction. ESAIM: Mathematical Modelling and Numerical Analysis 53(6), 2047–2080 (2019)
2019
-
[22]
Acta Numerica 10, 251–312 (2001)
Eiermann, M., Ernst, O.G.: Geometric aspects of the theory of krylov subspace methods. Acta Numerica 10, 251–312 (2001)
2001
-
[23]
arXiv preprint arXiv:1912.01703 (2019)
Paszke, A.: Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 (2019)
2019 arXiv
-
[24]
Journal of Scientific Computing 97(35) (2023) 23
Franco, N.R., Manzoni, A., Zunino, P.: Mesh-informed neural networks for oper- ator learning in finite element spaces. Journal of Scientific Computing 97(35) (2023) 23
2023
-
[25]
arXiv (2018) 24
Battaglia, P., Hamrick, J.B.C., Bapst, V., Sanchez, A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., Gulcehre, C., Song, F., Ballard, A., Gilmer, J., Dahl, G.E., Vaswani, A., Allen, K., Nash, C., Langston, V.J., Dyer, C., Heess, N., Wiers...
2018
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.