TRIE: An Evaluation Framework for Stochastic PDE Surrogates
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-02 19:36 UTCgrok-4.3pith:FFPUNLFYrecord.jsonopen to challenge →
The pith
Generative models best capture invariant statistics and uncertainty in stochastic PDE forecasts.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Across the two stationary chaotic SPDEs and eleven parameter values, generative models provide the most consistent performance by accurately capturing invariant measure statistics and recording the lowest CRPS in every probabilistic setting examined; standard pointwise-trained neural surrogates produce plausible short rollouts yet fail to match long-time statistical structure, and approximate uncertainty methods yield stochastic forecasts that are frequently miscalibrated.
What carries the argument
TRIE evaluation framework, which diagnoses surrogates on reproduction of invariant measures, trustworthiness of predictive uncertainty, and efficiency of probabilistic generation.
If this is right
- Pointwise neural surrogates should not be relied upon for long-term distributional forecasting in systems with stochastic forcing.
- Generative models should be the default choice when a surrogate must reproduce invariant statistics and calibrated uncertainty.
- Latent generative models with automatic dimension reduction can deliver comparable statistical fidelity at substantially lower inference cost.
- Evaluation protocols for stochastic surrogates must include explicit checks on invariant measures and temporal-spatial calibration rather than short-rollout error alone.
Where Pith is reading between the lines
- The same evaluation criteria could be used to compare surrogates for other uncertain physical systems such as turbulent flows or climate models.
- Training objectives that directly penalize mismatch in invariant measures might improve performance of non-generative models.
- The observed speed-accuracy trade-off in latent models suggests similar dimension-discovery techniques could be tested on higher-resolution or three-dimensional SPDEs.
Load-bearing premise
The two chosen stationary chaotic SPDEs and the eleven parameter values are representative enough for the framework's diagnostics to generalize to other stochastic PDE systems.
What would settle it
Apply the same TRIE diagnostics to a third stochastic PDE with different forcing or non-stationary statistics and check whether generative models still record the lowest CRPS and closest match to invariant measures.
Figures
read the original abstract
Many scientific systems exhibit uncertainty from stochastic forcing, unresolved degrees of freedom, or imperfect observations, making reliable surrogate forecasting fundamentally distributional rather than pointwise. For such systems, deterministic neural surrogates fail to capture statistical measures and forecast uncertainty. We introduce TRIE, an evaluation framework for stochastic PDE surrogates that asks whether models reproduce invariant measures, provide trustworthy predictive uncertainty, and scale to efficient probabilistic generation. We demonstrate TRIE on two stationary chaotic spatially extended SPDEs, stochastic Kuramoto--Sivashinsky and stochastic Kolmogorov flow, across 11 parameter values. Our evaluation shows that standard pointwise-trained neural surrogates can produce plausible short rollouts while failing to match long-time statistical structure. Approximate uncertainty methods such as Monte Carlo dropout and heteroscedastic Gaussian likelihoods produce stochastic forecasts, but are often miscalibrated and overconfident under temporal and spatial uncertainty diagnostics. Across these criteria, generative models provide the most consistent performance, accurately capturing invariant measure statistics and achieving the lowest CRPS in all reported probabilistic settings. Finally, we show that latent generative models with automatic dimension discovery retain much of this statistical fidelity while reducing Kolmogorov inference time by roughly $12\times$. We release our code and data at https://github.com/scailab/TRIE-SPDE-Bench to support reproducible evaluation of stochastic PDE forecasting models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces TRIE, an evaluation framework for stochastic PDE surrogates that tests reproduction of invariant measures, calibration of predictive uncertainty under temporal/spatial diagnostics, and efficiency of probabilistic generation. Demonstrated on the stochastic Kuramoto-Sivashinsky equation and stochastic Kolmogorov flow across 11 parameter values, the work reports that pointwise neural surrogates produce plausible short rollouts but fail to match long-time statistics, that Monte Carlo dropout and heteroscedastic likelihoods yield miscalibrated forecasts, and that generative models achieve the most consistent performance with lowest CRPS while latent variants with automatic dimension discovery preserve fidelity at roughly 12× lower Kolmogorov inference time. Code and data are released for reproducibility.
Significance. If the central findings hold, TRIE supplies a needed standardized benchmark for distributional surrogates in scientific machine learning, where pointwise metrics are insufficient. The explicit comparison of model classes on invariant-measure fidelity and CRPS, together with the public code release, strengthens the contribution by enabling direct community verification and extension. The efficiency result for latent generative models is a concrete, actionable observation for practitioners working with high-dimensional SPDEs.
major comments (2)
- [Abstract] Abstract and evaluation sections: the claim that generative models provide the most consistent performance and lowest CRPS rests on only two stationary chaotic SPDEs (stochastic Kuramoto-Sivashinsky and stochastic Kolmogorov flow) across 11 parameter values; because the TRIE diagnostics could rank model classes differently on systems whose correlation lengths, forcing statistics, or attractor structure differ materially, the representativeness assumption is load-bearing for any general statement about model-class superiority.
- [Methods] Methods and results sections: the abstract and reported performance differences provide no explicit description of data splits, number of trajectories used for invariant-measure estimation, or statistical significance tests applied to CRPS and calibration scores; without these details the robustness of the ranking between generative models and approximate uncertainty methods cannot be verified from the text alone.
minor comments (1)
- The GitHub link is given but the manuscript does not state the exact commit or tag used for the released code; adding this would improve reproducibility.
Simulated Author's Rebuttal
We thank the referee for their constructive comments on our manuscript introducing the TRIE framework. We address each of the major comments below.
read point-by-point responses
-
Referee: [Abstract] Abstract and evaluation sections: the claim that generative models provide the most consistent performance and lowest CRPS rests on only two stationary chaotic SPDEs (stochastic Kuramoto-Sivashinsky and stochastic Kolmogorov flow) across 11 parameter values; because the TRIE diagnostics could rank model classes differently on systems whose correlation lengths, forcing statistics, or attractor structure differ materially, the representativeness assumption is load-bearing for any general statement about model-class superiority.
Authors: We agree that the evaluation is limited to two specific SPDEs and that general statements about model superiority should be made cautiously. In the revised version, we will modify the abstract and relevant sections to clarify that the superior performance of generative models is observed on the tested systems (stochastic Kuramoto-Sivashinsky and stochastic Kolmogorov flow across 11 parameter values) and to highlight the potential for different rankings on other SPDEs with varying characteristics. We will also add a discussion on the representativeness of these test cases. revision: yes
-
Referee: [Methods] Methods and results sections: the abstract and reported performance differences provide no explicit description of data splits, number of trajectories used for invariant-measure estimation, or statistical significance tests applied to CRPS and calibration scores; without these details the robustness of the ranking between generative models and approximate uncertainty methods cannot be verified from the text alone.
Authors: The current manuscript text does not include these explicit details, which is an oversight. We will revise the methods and results sections to provide a clear description of the data splits, the number of trajectories used for estimating invariant measures, and any statistical significance tests performed on the CRPS and calibration metrics. This will enhance the verifiability of our findings. revision: yes
Circularity Check
No circularity: empirical evaluation framework with external benchmarks
full rationale
The paper introduces TRIE as an evaluation framework and applies it empirically to two SPDEs across 11 parameter values, comparing neural surrogates, uncertainty methods, and generative models against invariant measures, CRPS, and calibration diagnostics. No equations, derivations, or fitted parameters are presented that reduce to the inputs by construction. All performance claims rest on direct statistical comparisons to external benchmarks rather than self-referential definitions or self-citation chains. The analysis is self-contained against those benchmarks.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Stationary chaotic SPDEs possess well-defined invariant measures that can be estimated from long trajectories.
- standard math CRPS and related probabilistic scores are appropriate for assessing forecast quality in spatially extended stochastic systems.
Reference graph
Works this paper leans on
-
[1]
Michael Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions.Journal of Machine Learning Research, 26(209): 1–80, 2025
work page 2025
-
[2]
Stochastic interpolants with data-dependent couplings.arXiv preprint arXiv:2310.03725, 2023
Michael S Albergo, Mark Goldstein, Nicholas M Boffi, Rajesh Ranganath, and Eric Vanden-Eijnden. Stochastic interpolants with data-dependent couplings.arXiv preprint arXiv:2310.03725, 2023
-
[3]
Stochastic model for power grid dy- namics
Marian Anghel, Kenneth A Werley, and Adilson E Motter. Stochastic model for power grid dy- namics. In2007 40th Annual Hawaii International Conference on System Sciences (HICSS’07), pages 113–113. IEEE, 2007
work page 2007
-
[4]
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks.Advances in neural information processing systems, 28, 2015
work page 2015
-
[5]
Probabilistic forecasting with stochastic interpolants and f\" ollmer processes
Yifan Chen, Mark Goldstein, Mengjian Hua, Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Probabilistic forecasting with stochastic interpolants and f\" ollmer processes. arXiv preprint arXiv:2403.13724, 2024. 10
-
[6]
Random attractors.Journal of Dynamics and Differential Equations, 9(2):307–341, 1997
Hans Crauel, Arnaud Debussche, and Franco Flandoli. Random attractors.Journal of Dynamics and Differential Equations, 9(2):307–341, 1997
work page 1997
-
[7]
Pattern formation outside of equilibrium.Reviews of modern physics, 65(3):851, 1993
Mark C Cross and Pierre C Hohenberg. Pattern formation outside of equilibrium.Reviews of modern physics, 65(3):851, 1993
work page 1993
-
[8]
Unreliable uncertainty estimates with monte carlo dropout.arXiv preprint arXiv:2512.14851, 2025
Aslak Djupskås, Alexander Johannes Stasik, and Signe Riemer-Sørensen. Unreliable uncertainty estimates with monte carlo dropout.arXiv preprint arXiv:2512.14851, 2025
-
[9]
Gideon Dresdner, Dmitrii Kochkov, Peter Christian Norgaard, Leonardo Zepeda-Nunez, Jamie Smith, Michael Brenner, and Stephan Hoyer. Learning to correct spectral methods for simulating turbulent flows.Transactions on Machine Learning Research, 2023
work page 2023
-
[10]
Computational galerkin methods
Clive AJ Fletcher. Computational galerkin methods. InComputational galerkin methods, pages 72–85. Springer, 1984
work page 1984
-
[11]
Is mc dropout bayesian? arXiv preprint arXiv:2110.04286, 2021
Loic Le Folgoc, Vasileios Baltatzis, Sujal Desai, Anand Devaraj, Sam Ellis, Octavio E Martinez Manzanera, Arjun Nair, Huaqi Qiu, Julia Schnabel, and Ben Glocker. Is mc dropout bayesian? arXiv preprint arXiv:2110.04286, 2021
-
[12]
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. InInternational Conference on Machine Learning, pages 1050–1059. PMLR, 2016
work page 2016
-
[13]
Han Gao, Sebastian Kaltenbach, and Petros Koumoutsakos. Generative learning for forecasting the dynamics of high-dimensional complex systems.Nature Communications, 15(1):8904, 2024
work page 2024
-
[14]
Hiroshi Gotoda, Marc Pradas, and Serafim Kalliadasis. Chaotic versus stochastic behavior in active-dissipative nonlinear systems.Physical Review Fluids, 2(12):124401, 2017
work page 2017
-
[15]
Celso Grebogi, Edward Ott, and James A Yorke. Crises, sudden changes in chaotic attractors, and transient chaos.Physica D: Nonlinear Phenomena, 7(1-3):181–200, 1983
work page 1983
-
[16]
Martin Hairer and Jonathan C Mattingly. Ergodicity of the 2d navier-stokes equations with degenerate stochastic forcing.Annals of Mathematics, pages 993–1032, 2006
work page 2006
-
[17]
Ensemble kalman filtering without a model
Franz Hamilton, Tyrus Berry, and Timothy Sauer. Ensemble kalman filtering without a model. Physical Review X, 6(1):011021, 2016
work page 2016
-
[18]
Benjamin Holzschuh, Qiang Liu, Georg Kohl, and Nils Thuerey. Pde-transformer: Efficient and versatile transformers for physics simulations.arXiv preprint arXiv:2505.24717, 2025
-
[19]
Werner Horsthemke and René Lefever.Noise-induced transitions: theory and applications in physics, chemistry, and biology. Springer, 1984
work page 1984
-
[20]
Li Jing, Jure Zbontar, et al. Implicit rank-minimizing autoencoder.Advances in Neural Information Processing Systems, 33:14736–14746, 2020
work page 2020
-
[21]
Repellers, semi-attractors, and long-lived chaotic transients
Holger Kantz and Peter Grassberger. Repellers, semi-attractors, and long-lived chaotic transients. Physica D: Nonlinear Phenomena, 17(1):75–86, 1985
work page 1985
-
[22]
Eleni D Koronaki, Nikolaos Evangelou, Cristina P Martin-Linares, Edriss S Titi, and Ioannis G Kevrekidis. Nonlinear dimensionality reduction then and now: Aims for dissipative pdes in the ml era.Journal of Computational Physics, 506:112910, 2024
work page 2024
-
[23]
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles.Advances in neural information processing systems, 30, 2017
work page 2017
-
[24]
Ensemble forecasting.Journal of computational physics, 227(7):3515–3539, 2008
Martin Leutbecher and Tim N Palmer. Ensemble forecasting.Journal of computational physics, 227(7):3515–3539, 2008. 11
work page 2008
-
[25]
SPDEBench: An Extensive Benchmark for Learning Stochastic PDEs
Zheyan Li, Yuantu Zhu, Hao Ni, Siran Li, Bingguang Chen, and Qi Meng. Spdebench: An extensive benchmark for learning regular and singular stochastic pdes.arXiv preprint arXiv:2505.18511, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[26]
Alec J Linot, Joshua W Burby, Qi Tang, Prasanna Balaprakash, Michael D Graham, and Romit Maulik. Stabilized neural ordinary differential equations for long-time forecasting of dynamical systems.Journal of Computational Physics, 474:111838, 2023
work page 2023
-
[27]
Flow match- ing for generative modeling
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow match- ing for generative modeling. In11th International Conference on Learning Representations, ICLR 2023, 2023
work page 2023
-
[28]
Phillip Lippe, Bas Veeling, Paris Perdikaris, Richard Turner, and Johannes Brandstetter. Pde- refiner: Achieving accurate long rollouts with neural pde solvers.Advances in Neural Informa- tion Processing Systems, 36:67398–67433, 2023
work page 2023
-
[29]
Chuizheng Meng, Sam Griesemer, Defu Cao, Sungyong Seo, and Yan Liu. When physics meets machine learning: A survey of physics-informed machine learning.Machine Learning for Computational Science and Engineering, 1(1):20, 2025
work page 2025
-
[30]
Estimating the mean and variance of the target proba- bility distribution
David A Nix and Andreas S Weigend. Estimating the mean and variance of the target proba- bility distribution. InProceedings of 1994 ieee international conference on neural networks (ICNN’94), volume 1, pages 55–60. IEEE, 1994
work page 1994
-
[31]
Jaideep Pathak, Brian Hunt, Michelle Girvan, Zhixin Lu, and Edward Ott. Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach.Physical review letters, 120(2):024102, 2018
work page 2018
-
[32]
Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaos
Christian Pedersen, Laure Zanna, and Joan Bruna. Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaos. InInternational Conference on Machine Learning, pages 48635–48661. PMLR, 2025
work page 2025
-
[33]
Expanding maps on sets which are almost invariant
Giulio Pianigiani and James A Yorke. Expanding maps on sets which are almost invariant. decay and chaos.Transactions of the American Mathematical Society, 252:351–366, 1979
work page 1979
-
[34]
Gencast: Diffusion-based ensemble forecasting for medium-range weather
Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Tom R Andersson, Andrew El-Kadi, Dominic Masters, Timo Ewalds, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, et al. Gencast: Diffusion-based ensemble forecasting for medium-range weather. In105th AMS Annual Meeting. AMS, 2025
work page 2025
-
[35]
JC Robinson and C Pierre. Infinite-dimensional dynamical systems: An introduction to dissipa- tive parabolic pdes and the theory of global attractors. cambridge texts in applied mathematics, 2003
work page 2003
-
[36]
Lost in latent space: An empirical study of latent diffusion models for physics emulation
François Rozet, Ruben Ohana, Michael McCabe, Gilles Louppe, François Lanusse, and Shirley Ho. Lost in latent space: An empirical study of latent diffusion models for physics emulation. arXiv preprint arXiv:2507.02608, 2025
-
[37]
Salva Rühling Cachay, Bo Zhao, Hailey Joren, and Rose Yu. Dyffusion: A dynamics-informed diffusion model for spatiotemporal forecasting.Advances in neural information processing systems, 36:45259–45287, 2023
work page 2023
-
[38]
Cristopher Salvi, Maud Lemercier, and Andris Gerasimovics. Neural stochastic pdes: Resolution-invariant learning of continuous spatiotemporal dynamics.Advances in Neural Information Processing Systems, 35:1333–1344, 2022
work page 2022
-
[39]
Steven H Strogatz.Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. Chapman and Hall/CRC, 2024
work page 2024
-
[40]
Nicolaas Godfried Van Kampen.Stochastic processes in physics and chemistry, volume 1. Elsevier, 1992
work page 1992
-
[41]
Johannes V oit.The statistical mechanics of financial markets. Springer, 2005. 12
work page 2005
-
[42]
Diffcast: A unified framework via residual diffusion for precipitation nowcasting
Demin Yu, Xutao Li, Yunming Ye, Baoquan Zhang, Chuyao Luo, Kuai Dai, Rui Wang, and Xunlai Chen. Diffcast: A unified framework via residual diffusion for precipitation nowcasting. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27758–27767, 2024
work page 2024
-
[43]
Kevin Zeng, Carlos E Perez De Jesus, Andrew J Fox, and Michael D Graham. Autoencoders for discovering manifold dimension and coordinates in data from complex dynamical systems. Machine Learning: Science and Technology, 5(2):025053, 2024. A Technical appendices and supplementary material A.1 SPDEs and Simulation A.1.1 Stochastic Kuramoto-Sivashinsky We mod...
work page 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.