pith. sign in

arxiv: 2606.18200 · v1 · pith:JTPFXVCYnew · submitted 2026-06-16 · 💻 cs.MS · cs.NA· math.NA· physics.comp-ph

A Diagnostic Software Suite for Auditing Learned PDE Simulators

Lennon J. Shikhman This is my paper

Pith reviewed 2026-06-26 21:27 UTC · model grok-4.3

classification 💻 cs.MS cs.NAmath.NAphysics.comp-ph
keywords learned PDE simulatorsdiagnostic suitesemigroup consistencyfinite-difference generatorL2 errorNavier-Stokesshallow-water dynamicsmagnetohydrodynamics
0
0 comments X

The pith

Relative L2 error can stay moderate while structural diagnostics for learned PDE simulators deteriorate substantially.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents a software suite that audits learned PDE simulators by checking whether they function as coherent numerical time propagators rather than merely matching states at isolated times. Standard relative L2 error is shown to be inadequate because it can remain acceptable even as properties like semigroup consistency and energy behavior break down. The suite runs architecture-independent post-hoc checks using reference trajectories, model predictions, and equation metadata, with a configurable selection of diagnostics relevant to each problem. Validation across five benchmark tasks and multiple model architectures, including controlled underfit variants, demonstrates cases where L2 error does not flag these structural failures. This matters for users who need reliable long-term evolution from learned replacements of traditional solvers.

Core claim

The validation study establishes that relative L2 error alone does not determine whether a learned model behaves as a coherent numerical time propagator, since the error can remain moderate or even improve while diagnostics for semigroup consistency, finite-difference generator discrepancy, energy behavior, integral balance, admissibility constraints, perturbation response, and scaling-law consistency deteriorate substantially on the benchmark tasks.

What carries the argument

The diagnostic software suite, which enforces a minimal contract of reference trajectories, learned propagator outputs, equation metadata, and a configuration to compute an interpretable panel of time-propagation checks.

If this is right

  • Learned models can achieve acceptable relative L2 error while violating semigroup consistency or energy behavior.
  • Auditing requires reporting the full diagnostic panel instead of reducing behavior to a single state-error score.
  • The suite applies post hoc to any saved predictions from FNO, DeepONet, U-Net, or ResNet-style models on the listed PDE tasks.
  • Controlled underfit and oversmoothed variants exhibit substantially worse structural diagnostics even when L2 error is managed.
  • Deployment decisions for learned simulators should incorporate these checks to avoid incoherent time propagation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Training procedures could add terms that penalize failures on these diagnostics to improve coherence directly.
  • The same diagnostic approach might extend to learned simulators for other evolution problems beyond the five PDE benchmarks.
  • Problem-specific configuration of the suite becomes essential when moving to new equations where only a subset of checks apply.
  • Widespread adoption would shift evaluation practice away from single-metric leaderboards toward multi-panel reports.

Load-bearing premise

The chosen diagnostics correctly identify coherent numerical time propagation and the five benchmark tasks represent the settings where learned simulators would be used.

What would settle it

Finding a learned simulator on one of the benchmark tasks that passes every structural diagnostic yet produces visibly unphysical long-time trajectories under the same initial conditions used in validation.

read the original abstract

Learned PDE simulators are increasingly used as low-cost replacements for expensive numerical solvers, but standard relative $L^2$ error does not determine whether a learned model behaves as a coherent numerical time propagator. This paper presents a diagnostic software suite for auditing learned PDE simulators as approximate evolution operators. The suite provides architecture-independent, post hoc diagnostics for relative state error, semigroup consistency, finite-difference generator discrepancy, energy behavior, integral balance, admissibility constraints, perturbation response, and scaling-law consistency. The software is designed around a minimal contract: reference trajectories, a learned propagator or saved predictions, equation metadata, and a diagnostic configuration specifying which structures are meaningful for the problem under study. We validate the suite on five benchmark PDE tasks: two-dimensional incompressible Navier-Stokes, shallow-water dynamics, active matter, three-dimensional compressible Navier-Stokes, and three-dimensional magnetohydrodynamics, using FNO, DeepONet, U-Net, and ResNet-style surrogate models together with controlled underfit and oversmoothed variants. The validation study shows that relative $L^2$ error can remain moderate, or even improve, while structural diagnostics deteriorate substantially. The package therefore supports software-level auditing of learned PDE simulators by reporting an interpretable diagnostic panel rather than collapsing model behavior into a single state-error score.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript presents a diagnostic software suite for auditing learned PDE simulators as approximate evolution operators. It supplies architecture-independent, post-hoc checks (relative state error, semigroup consistency, finite-difference generator discrepancy, energy behavior, integral balance, admissibility constraints, perturbation response, scaling-law consistency) under a minimal contract requiring only reference trajectories, predictions, equation metadata, and a configuration file. Validation is performed on five benchmark tasks (2D incompressible Navier-Stokes, shallow-water, active matter, 3D compressible Navier-Stokes, 3D MHD) using FNO, DeepONet, U-Net, and ResNet surrogates plus controlled underfit/oversmoothed variants; the central empirical observation is that relative L2 error can remain moderate or improve while the structural diagnostics degrade substantially.

Significance. If the reported empirical observation holds, the suite supplies a practical, reusable auditing tool that prevents over-reliance on a single scalar error metric for scientific machine-learning surrogates. The minimal-contract design and explicit separation of diagnostics from any particular architecture are strengths that lower the barrier to adoption. The work also supplies reproducible code and a falsifiable panel of tests rather than a new theoretical claim.

major comments (1)
  1. [Validation study (§4–5)] Validation study (described in abstract and §4–5): the central claim that relative L2 error can remain moderate while structural diagnostics deteriorate is load-bearing for the paper’s contribution, yet the manuscript supplies no quantitative tables, specific error values, or statistical error analysis across the five tasks and model variants; without these numbers the strength of the supporting evidence cannot be assessed.
minor comments (2)
  1. [Software design] The description of the minimal contract (reference trajectories + predictions + metadata + config) is clear, but an explicit listing of which diagnostics are activated by default for each of the five PDEs would improve reproducibility.
  2. [Figures] Figure captions for the diagnostic panels should state the exact model variant and task shown so that readers can map visuals directly to the quantitative claims.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive review and the recommendation for minor revision. We address the single major comment below.

read point-by-point responses

  1. Referee: [Validation study (§4–5)] Validation study (described in abstract and §4–5): the central claim that relative L2 error can remain moderate while structural diagnostics deteriorate is load-bearing for the paper’s contribution, yet the manuscript supplies no quantitative tables, specific error values, or statistical error analysis across the five tasks and model variants; without these numbers the strength of the supporting evidence cannot be assessed.

    Authors: We agree that the central empirical claim requires explicit quantitative support for readers to assess its strength. The current manuscript presents the phenomenon through figures in §§4–5 that compare L² error against the structural diagnostics on the five tasks and the controlled model variants, but does not include accompanying tables of numerical values or statistical summaries. In the revision we will add tables that report the key metrics (relative L² error, semigroup consistency error, finite-difference generator discrepancy, energy drift, etc.) for each task–model pair, together with means and standard deviations where multiple runs are available. This addition will make the evidence fully quantitative while preserving the existing figures. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper introduces an independent diagnostic software suite with explicitly defined procedures (semigroup consistency, finite-difference generator discrepancy, energy behavior, etc.) that are applied post-hoc to external reference trajectories and benchmark models. No derivation chain, parameter fitting, or prediction step reduces by construction to quantities defined within the paper itself. The central empirical observation—that L2 error can decouple from structural diagnostics—is a direct measurement on held-out benchmarks rather than a self-referential claim. No self-citation load-bearing steps or ansatz smuggling appear in the provided text.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper is a software and benchmarking contribution rather than a mathematical derivation; it relies on the existence of reference PDE solvers but introduces no fitted parameters, new axioms, or postulated entities.

axioms (1)
  • domain assumption Reference trajectories from traditional numerical PDE solvers are available and serve as ground truth.
    The entire validation and diagnostic contract depends on having such trajectories.

pith-pipeline@v0.9.1-grok · 5759 in / 1211 out tokens · 29452 ms · 2026-06-26T21:27:11.801327+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 9 canonical work pages

  1. [1]

    L. Lu, P. Jin, G. Pang, Z. Zhang, G. E. Karniadakis, Learning nonlinear operators via deeponet based on the universal approximation theorem of operators, Nature Machine Intelligence 3 (3) (03 2021).doi:10.1038/ s42256-021-00302-5. URLhttps://www.osti.gov/biblio/2281727

    arXiv 2021

  2. [2]

    Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stu- art, A. Anandkumar, Fourier neural operator for parametric partial dif- ferential equations, International Conference on Learning Representa- tions (2021). URLhttps://openreview.net/forum?id=c8P9NQVtmnO

    2021

  3. [3]

    Kovachki, Z

    N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stu- art, A. Anandkumar, Neural operator: Learning maps between function spaces with applications to pdes, Journal of Machine Learning Research 24 (89) (2023) 1–97. URLhttp://jmlr.org/papers/v24/21-1524.html

    2023

  4. [4]

    Takamoto, T

    M. Takamoto, T. Praditia, R. Leiteritz, D. MacKinlay, F. Alesiani, D. Pflüger, M. Niepert, Pdebench: An extensive benchmark for scientific machine learning, in: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh (Eds.), Advances in Neural Information Processing Systems, Vol. 35, Curran Associates, Inc., 2022, pp. 1596–1611. URLhttps://proceedin...

    2022

  5. [5]

    Koehler, S

    F. Koehler, S. Niedermayr, R. Westermann, N. Thuerey, Apebench: A benchmark for autoregressive neural emulators of pdes, in: A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, C. Zhang (Eds.), Advances in Neural Information Processing Sys- tems, Vol. 37, Curran Associates, Inc., 2024, pp. 120252–120310. doi:10.52202/079017-3822. URLhtt...

    work page doi:10.52202/079017-3822 2024

  6. [6]

    Agocs, Miguel Beneitez, Marsha Berger, Blakesley Burkhart, Stuart B

    R. Ohana, M. McCabe, L. Meyer, R. Morel, F. J. Agocs, M. Beneitez, M. Berger, B. Burkhart, S. B. Dalziel, D. B. Fielding, D. Fortunato, J. A. Goldberg, K. Hirashima, Y.-F. Jiang, R. R. Kerswell, S. Maddu, J. Miller, P. Mukhopadhyay, S. S. Nixon, J. Shen, R. Watteaux, B. R.-S. Blancard, F. Rozet, L. Parker, M. Cranmer, S. Ho, The well: a large-scale collec...

    work page doi:10.52202/079017-1430 2024

  7. [7]

    R. J. LeVeque, Finite Volume Methods for Hyperbolic Problems, Cam- bridge Texts in Applied Mathematics, Cambridge University Press, Cambridge, 2002

    2002

  8. [8]

    Hairer, C

    E. Hairer, C. Lubich, G. Wanner, Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations, 2nd Edition, Vol. 31 of Springer Series in Computational Mathemat- ics, Springer, Berlin, Heidelberg, 2006.doi:10.1007/3-540-30666-8

    work page doi:10.1007/3-540-30666-8 2006

  9. [9]

    Pazy, Semigroups of Linear Operators and Applications to Par- tial Differential Equations, 1st Edition, Vol

    A. Pazy, Semigroups of Linear Operators and Applications to Par- tial Differential Equations, 1st Edition, Vol. 44 of Applied Mathemat- ical Sciences, Springer-Verlag, New York, NY, 1983.doi:10.1007/ 978-1-4612-5561-1

    1983

  10. [10]

    Engel, R

    K.-J. Engel, R. Nagel, One-Parameter Semigroups for Linear Evolution Equations, 1st Edition, Vol. 194 of Graduate Texts in Mathematics, Springer, New York, NY, 2000.doi:10.1007/b97696

    work page doi:10.1007/b97696 2000

  11. [11]

    Helwig, X

    J. Helwig, X. Zhang, C. Fu, J. Kurtin, S. Wojtowytsch, S. Ji, Group equivariant fourier neural operators for partial differential equations, in: Proceedings of the 40th International Conference on Machine Learning, ICML’23, JMLR.org, 2023

    2023

  12. [12]

    Z. Li, H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizzade- nesheli, A. Anandkumar, Physics-informed neural operator for learning partial differential equations, ACM / IMS J. Data Sci. 1 (3) (May 2024). 31 doi:10.1145/3648506. URLhttps://doi.org/10.1145/3648506

    work page doi:10.1145/3648506 2024

  13. [13]

    Goswami, A

    S. Goswami, A. Bora, Y. Yu, G. E. Karniadakis, Physics-informed deep neural operator networks, in: Machine Learning in Modeling and Sim- ulation: Methods and Applications, Springer International Publishing, Cham, 2023, pp. 219–254.doi:10.1007/978-3-031-36644-4\_6. URLhttps://doi.org/10.1007/978-3-031-36644-4_6

    work page doi:10.1007/978-3-031-36644-4 2023

  14. [14]

    Kar- niadakis

    M. Raissi, P. Perdikaris, G. Karniadakis, Physics-informed neu- ral networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equa- tions, Journal of Computational Physics 378 (2019) 686–707. doi:https://doi.org/10.1016/j.jcp.2018.10.045. URLhttps://www.sciencedirect.com/science/article/pii/ S002...

    work page doi:10.1016/j.jcp.2018.10.045 2019

  15. [15]

    G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, L. Yang, Physics-informed machine learning, Nature Reviews Physics 3 (6) (2021) 422–440.doi:10.1038/s42254-021-00314-5. URLhttps://doi.org/10.1038/s42254-021-00314-5

    work page doi:10.1038/s42254-021-00314-5 2021

  16. [16]

    Krishnapriyan, A

    A. Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, M. Mahoney, Char- acterizing possible failure modes in physics-informed neural networks, in: M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems, Vol. 34, Curran Associates, Inc., 2021, pp. 26548–26560. URLhttps://proceedings.neurips.cc/pa...

    2021

  17. [17]

    Rahaman, A

    N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio, A. Courville, On the spectral bias of neural networks, in: K. Chaudhuri, R. Salakhutdinov (Eds.), Proceedings of the 36th In- ternational Conference on Machine Learning, Vol. 97 of Proceedings of Machine Learning Research, PMLR, 2019, pp. 5301–5310. URLhttps://proceedings.mlr.p...

    2019

  18. [18]

    URLhttps://openreview.net/forum?id=0S1LWZHQYn 32

    L.Shikhman, Diagnosingfailuremodesofneuraloperatorsacrossdiverse PDE families, Transactions on Machine Learning Research (2026). URLhttps://openreview.net/forum?id=0S1LWZHQYn 32

    2026

  19. [19]

    L. J. Shikhman, Semigroup consistency as a diagnostic for learned physics simulators, in: ICML 2026 Workshop on AI for Physics, 2026. URLhttps://openreview.net/forum?id=MeAFOZnrvM

    2026

  20. [20]

    Z. Gao, L. Yang, G. E. Karniadakis, Spectral audit of in-context oper- ator networks (2026).arXiv:2606.02427. URLhttps://arxiv.org/abs/2606.02427

    Pith/arXiv arXiv 2026

  21. [21]

    Lanthaler, A

    S. Lanthaler, A. M. Stuart, M. Trautner, Discretization error of fourier neural operators (2025).arXiv:2405.02221. URLhttps://arxiv.org/abs/2405.02221

    arXiv 2025

  22. [22]

    Classics in Mathematics

    T. Kato, Perturbation Theory for Linear Operators, 2nd Edition, Vol. 132 of Classics in Mathematics, Springer-Verlag, Berlin, Heidelberg, 1995, originally published as volume 132 in Grundlehren der mathe- matischen Wissenschaften.doi:10.1007/978-3-642-66282-9

    work page doi:10.1007/978-3-642-66282-9 1995

  23. [23]

    G. I. Barenblatt, Introduction, Cambridge Texts in Applied Mathemat- ics, Cambridge University Press, Cambridge, 1996, pp. 1–27. 33

    1996