REVIEW 2 major objections 5 minor 37 references
This paper claims that machine learning weather models can be tested for physical consistency by rotating or reversing the planet's coordinates, and that GraphCast and NeuralGCM fail these tests while a physics-based general circulation mod
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 09:33 UTC pith:WPONKNA6
load-bearing objection Solid, novel testing framework; the Reverse Longitude result for GraphCast is weakened by inconsistent inputs, so the headline claim overreaches. the 2 major comments →
Spatial Generalization Tests for Machine Learning-based Weather Models to Assess Physical Consistency
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that spatial generalization can be tested cleanly by exploiting the invariance of the atmosphere's governing equations under three involutive coordinate transformations. Once the required sign changes are applied to winds, forcing, and where possible rotation and time, a physically consistent weather model should produce forecasts that, after reverting the transformation, match the baseline. The paper reports that GraphCast violates this: its generalization error is comparable to or larger than its forecast error, and it maps the daily cycle partly from date-time and coordinate information rather than from incident solar radiation. NeuralGCM's encoder-decoder also fails
What carries the argument
The machinery is a set of three spatial involutions—half-turn in longitude, reflection in latitude, and reflection in longitude—applied to every initial and boundary field, together with the sign changes they induce in wind components, Coriolis or planetary rotation, and, where the model allows, time and solar forcing. The test metric is the generalization error: the difference between a baseline forecast and the forecast on the transformed planet after reversing the transformation. A model passes if this error is negligible relative to its own forecast error; a physics-based general circulation model demonstrates passability with errors at rounding level.
Load-bearing premise
The test cases are treated as physically self-consistent, so that a model failure indicates a learned spatial non-invariance rather than an inconsistent input set; this is violated for GraphCast's Reverse Longitude case, where time cannot be made local or backward and the Coriolis parameter is learned and cannot be changed, and it remains unproven for NeuralGCM, where only the encoder-decoder is tested without full time integration.
What would settle it
Run the Reverse Longitude test on an ML weather model that accepts local, coordinate-dependent time and allows the planetary rotation or Coriolis parameter to be negated. If the generalization error drops to rounding-error level, the paper's attribution of failure to coordinate- and time-dependent shortcuts is supported; if errors persist, the failure may instead stem from the test's internal inconsistency. Alternatively, run NeuralGCM's full model with a manual pressure-to-sigma conversion; if ghost continents disappear during time integration, the claim that the encoder-decoder itself breaks
If this is right
- Forecast-skill metrics alone are insufficient to certify physical consistency in ML weather models; spatial generalization tests can expose nonphysical shortcuts.
- ML weather models that fail these tests cannot be used for scientific experiments such as idealized solar forcing, land-use change, or melting sea ice.
- A model that genuinely generalizes spatially should be more robust to shifting climate conditions and may require less frequent retraining.
- The proposed test protocol can be extended beyond the three base cases to seven combined transformations and other rotation angles depending on grid symmetry.
- If the daily-cycle finding is correct, removing date-time inputs and relying on physical radiative forcing could eliminate one identified source of nonphysical mapping.
Where Pith is reading between the lines
- If temporal generalization depends on spatial generalization, then models failing these tests are unlikely to be reliable climate projection tools, regardless of their weather forecast skill.
- The same protocol could be turned into a training signal by augmenting training data with spatially transformed fields, potentially forcing architectures to learn invariant physics.
- The identified dependence on coordinates and time suggests a concrete architectural change—replace coordinate/time features with physically meaningful boundary and forcing fields—that could be tested by retraining and re-running the tests.
- The test framework could be extended to probabilistic and ensemble forecast models, checking whether their internal distributions remain invariant under the same transformations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes three spatial generalization test cases for ML-based weather models: Rotate Longitude, Reverse Latitude, and Reverse Longitude. The tests transform all input and boundary fields and appropriate wind components (and, where possible, planetary rotation and time information) so that a spatially invariant model should reproduce the baseline forecast after the transformation is reverted. The authors apply the tests to GraphCast small and to NeuralGCM's encoder-decoder, with SpeedyWeather as a physics-based control. They report that GraphCast fails all three tests, that NeuralGCM's encoder-decoder produces 'ghost continents,' and that SpeedyWeather reproduces the baseline to rounding-error precision. They further diagnose GraphCast's daily-cycle dependence on date-time and coordinate information. The central claim is that current ML weather models rely on location/time shortcuts and lack the spatial invariance required for climate-change generalization.
Significance. The proposed tests are simple, parameter-free, and directly address an important gap: forecast skill metrics do not measure physical consistency. The paper ships reproducible code and uses a 365-member ensemble, and the SpeedyWeather control plus the Appendix A invariance derivation are useful reference points. If the central claim holds, the work provides a practical diagnostic that could guide future architecture design. However, the strongest claims currently outrun the evidence: the Reverse Longitude test for GraphCast is admitted to be physically inconsistent (time and Coriolis are not adapted), and the NeuralGCM evaluation covers only the encoder-decoder, not the full forecast model. These issues do not invalidate the overall approach, but they require substantial revision before the stated conclusions are supported.
major comments (2)
- [Sec. 3.2, Sec. 5.1 (Table 2), Fig. 3] The Reverse Longitude test for GraphCast is not physically self-consistent. The authors state that time would need to run backwards or be formulated as local time, and that the Coriolis parameter is learned and cannot be changed. Therefore the model receives a reversed planet with an unreversed time input and no Ω→−Ω, so the reported 1.5–3× forecast-error generalization error may be dominated by this known input inconsistency rather than by learned spatial non-invariance. This undermines the statement in Sec. 3.1 that 'GraphCast fails each generalization test' and the abstract's claim that all boundary conditions and forcings were adapted accordingly. Please either drop Reverse Longitude from the GraphCast failure claims, provide a controlled experiment with consistent time/Coriolis handling, or explicitly re-label it as a physical-consistency test rather than a spatial-invariance test.
- [Sec. 3.3] The NeuralGCM evaluation is limited to the encoder-decoder module without any time integration. The assertion that 'errors will only be further amplified' is not demonstrated and is not logically guaranteed: the physics-based dynamical core could damp or correct initial-condition errors introduced by the encoder-decoder. Hence the abstract's claim that 'GraphCast and NeuralGCM fail these tests' overstates the NeuralGCM evidence. Please either run the full forecast loop (e.g., with a manual sigma-coordinate conversion and an explicit statement of what is excluded) or restrict the claim to 'NeuralGCM's encoder-decoder component fails to generalize spatially.'
minor comments (5)
- [Abstract and Sec. 1] The statement 'Physics-based general circulation models simulate a rotated/reversed planet with only rounding errors' is supported only by SpeedyWeather and by a 2D incompressible Navier-Stokes proof. Full GCMs include moist processes, radiation, and land-surface schemes not covered by the proof. Please qualify the claim as applying to the tested model or add a limitation statement.
- [Sec. 3.3] The phrase 'errors will only be further amplified' is an unsupported assertion. Consider replacing it with a statement of intent to test the full model in future work.
- [Sec. 5.5, Eq. (1)] The climatological skill score is computed from MSE and can be negative; this is not discussed in the text. Please add a sentence explaining the interpretation and acceptable range.
- [Fig. 4] Panel (j) is not defined in the caption. Please specify what correlation is plotted and how it is computed.
- [Appendix A] The notation R_x, R_y, R_o and the derivation leading to Eqs. (4)–(5) are difficult to follow. Suggest rewriting with explicit coordinate arguments or adding a table of transformation rules for clarity.
Circularity Check
No significant circularity: the invariance derivation is self-contained and the empirical test protocol compares models with themselves; the Reverse Longitude inconsistency is a validity caveat, not a circular derivation.
full rationale
The paper's formal derivation is Appendix A, which is self-contained and parameter-free: it starts from the 2D incompressible Navier-Stokes equations with Coriolis forcing and shows that applying reversal/rotation operators together with the tilde transforms (Eqs. 7-9) recovers the original equations. No fitted constant or empirical target is imported. The empirical test protocol is also non-circular: the generalization error is computed as the difference between a model's own baseline forecast and its forecast on rotated/reversed inputs (Sec. 5.5), and the pass/fail criterion is a comparison with the same model's forecast error, not with any quantity fitted to the conclusion. The passages that could look circular are the admissions in Sec. 3.2 and Sec. 5.1 that Reverse Longitude is not fully physically consistent for GraphCast: time cannot be run backwards or made local, and the Coriolis parameter is learned and cannot be changed. These are genuine validity caveats — they mean the Reverse Longitude failure may be partly caused by the input inconsistency itself — but they are not circularity, because the paper does not define failure as the inconsistency; it argues the inconsistency exposes the model's time and coordinate dependence. Similarly, the use of SpeedyWeather, co-developed by an author, is supplemented by the independent analytic invariance proof, so the self-citation is not load-bearing. The NeuralGCM result is explicitly scoped to the encoder-decoder module (Sec. 3.3) and is not used to claim a full-model forecast failure. Overall, no step reduces by construction to its own inputs.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption The atmospheric governing equations are invariant under the three spatial transformations when the specified sign changes, time shifts, and solar-forcing rotations are applied.
- ad hoc to paper The transformed input fields are physically self-consistent and complete for each ML model.
- domain assumption Leaving coordinate inputs and embeddings unchanged while transforming fields is the correct way to expose learned spatial shortcuts rather than architecture artifacts.
- domain assumption 2m-temperature is a sufficient diagnostic for the failure/success classification.
Cite this review
Pith. "Pith review of Spatial Generalization Tests for Machine Learning-based Weather Models to Assess Physical Consistency." pith.science (2026). https://pith.science/paper/WPONKNA6
@misc{pith2026260720716,
author = {Pith},
title = {Pith review of: Spatial Generalization Tests for Machine Learning-based Weather Models to Assess Physical Consistency},
year = {2026},
howpublished = {\url{https://pith.science/paper/WPONKNA6}},
note = {Machine review of arXiv:2607.20716}
}
read the original abstract
Machine learning-based weather prediction is revolutionizing weather forecasting by learning from weather data in present-day climate. However, generalization to other climates remains a major challenge. With melting sea ice, land-use change, and increasing ocean temperatures, boundary conditions are changing. Therefore, generalization in time depends on generalization in space. Here, we present three test cases to evaluate whether machine learning-based weather and climate models generalize in space and apply them to GraphCast and NeuralGCM. We reverse or rotate the planet in longitude or latitude under the model's coordinate system and adapt all boundary conditions and forcings accordingly. Physics-based general circulation models simulate a rotated/reversed planet with only rounding errors, but GraphCast and NeuralGCM fail these tests. The analyses furthermore revealed unphysical variable mappings based on correlation rather than causation. We argue that machine learning-based climate models should be designed to pass generalization tests to prevent overfitting on present-day regional climate.
Figures
Reference graph
Works this paper leans on
-
[1]
Forecasting Global Weather with Graph Neural Networks (2022)
Keisler, R. Forecasting Global Weather with Graph Neural Networks (2022). URL http://arxiv.org/abs/2202.07575. ArXiv:2202.07575 [physics]
Pith/arXiv arXiv 2022
-
[2]
URL https://www.science.org/doi/10.1126/science.adi2336
Lam, R.et al.Learning skillful medium-range global weather forecasting.Science (2023). URL https://www.science.org/doi/10.1126/science.adi2336
-
[3]
URL https://www.nature.com/articles/ s41586-023-06185-3
Bi, K.et al.Accurate medium-range global weather forecasting with 3D neural networks.Nature619, 533–538 (2023). URL https://www.nature.com/articles/ s41586-023-06185-3
2023
-
[4]
URL https://www.nature.com/articles/s41586-024-08252-9
Price, I.et al.Probabilistic weather forecasting with machine learning.Nature 637, 84–90 (2025). URL https://www.nature.com/articles/s41586-024-08252-9
2025
-
[5]
URL https://www.nature.com/articles/ s41586-024-07744-y
Kochkov, D.et al.Neural general circulation models for weather and cli- mate.Nature632, 1060–1066 (2024). URL https://www.nature.com/articles/ s41586-024-07744-y
2024
-
[6]
Rasp, S., Pritchard, M. S. & Gentine, P. Weatherbench: A benchmark dataset for data-driven weather forecasting.Journal of Advances in Modeling Earth Systems 12, e2020MS002109 (2020)
2020
-
[7]
URL http://arxiv.org/ abs/2202.11214
Pathak, J.et al.FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators (2022). URL http://arxiv.org/ abs/2202.11214. 2202.11214. 33
Pith/arXiv arXiv 2022
-
[8]
Mathieu, M., Couprie, C. & LeCun, Y. Deep multi-scale video prediction beyond mean square error (2016). URL http://arxiv.org/abs/1511.05440. 1511.05440
Pith/arXiv arXiv 2016
-
[9]
URL https://onlinelibrary.wiley.com/doi/abs/10.1029/ 2023MS004021
Karlbauer, M.et al.Advancing Parsimonious Deep Learning Weather Prediction Using the HEALPix Mesh.Journal of Advances in Modeling Earth Systems16, e2023MS004021 (2024). URL https://onlinelibrary.wiley.com/doi/abs/10.1029/ 2023MS004021
2024
-
[10]
URL https://gmd.copernicus.org/articles/18/5575/2025/
Mahesh, A.et al.Huge ensembles – Part 1: Design of ensemble weather forecasts using spherical Fourier neural operators.Geoscientific Model Development18, 5575–5603 (2025). URL https://gmd.copernicus.org/articles/18/5575/2025/
2025
-
[11]
URL http://arxiv.org/abs/2310.02074
Watt-Meyer, O.et al.ACE: A fast, skillful learned global atmospheric model for climate prediction (2023). URL http://arxiv.org/abs/2310.02074
Pith/arXiv arXiv 2023
-
[12]
Ullrich, P. A.et al.Recommendations for comprehensive and indepen- dent evaluation of machine learning-based earth system models.Jour- nal of Geophysical Research: Machine Learning and Computation2, e2024JH000496 (2025). URL https://agupubs.onlinelibrary.wiley.com/doi/abs/ 10.1029/2024JH000496. E2024JH000496 2024JH000496
-
[13]
URL https://arxiv.org/abs/2605.06944
Henn, B.et al.Aimip phase 1: systematic evaluations of ai weather and climate models (2026). URL https://arxiv.org/abs/2605.06944. arXiv:2605.06944
Pith/arXiv arXiv 2026
-
[14]
URL https://rmets.onlinelibrary
Hersbach, H.et al.The ERA5 global reanalysis.Quarterly Journal of the Royal Meteorological Society146, 1999–2049 (2020). URL https://rmets.onlinelibrary. wiley.com/doi/10.1002/qj.3803
doi:10.1002/qj.3803 1999
-
[15]
URL http://arxiv.org/abs/2409.18529
Rackow, T.et al.Robustness of AI-based weather forecasts in a changing climate (2024). URL http://arxiv.org/abs/2409.18529. 34
Pith/arXiv arXiv 2024
-
[16]
URL https://onlinelibrary.wiley.com/doi/abs/10
Rasp, S.et al.WeatherBench 2: A Benchmark for the Next Generation of Data- Driven Global Weather Models.Journal of Advances in Modeling Earth Systems 16, e2023MS004019 (2024). URL https://onlinelibrary.wiley.com/doi/abs/10. 1029/2023MS004019
2024
-
[17]
Jalan, S. & Sukhatme, J. Intraseasonal Equatorial Kelvin and Rossby Waves in Modern AI-ML Models (2025). URL http://arxiv.org/abs/2507.07952
Pith/arXiv arXiv 2025
-
[18]
& Barnes, E
Diao, C. & Barnes, E. A. Assessing MJO Tropical-Extratropical Teleconnections in Deep Learning Weather Prediction Mod- els (2025). URL https://essopenarchive.org/users/631510/articles/ 1296512-assessing-mjo-tropical-extratropical-teleconnections-in-deep-learning-weather-prediction-models
2025
-
[19]
URL https://www.nature.com/articles/s41612-025-00949-6
Ba˜ no-Medina, J.et al.Are AI weather models learning atmospheric physics? A sensitivity analysis of cyclone Xynthia.Climate and Atmospheric Science8, 92 (2025). URL https://www.nature.com/articles/s41612-025-00949-6
2025
-
[20]
& Messori, G
Olivetti, L. & Messori, G. Do data-driven models beat numerical models in forecasting weather extremes? A comparison of IFS HRES, Pangu-Weather, and GraphCast.Geoscientific Model Development17, 7915–7962 (2024). URL https: //gmd.copernicus.org/articles/17/7915/2024/
2024
-
[21]
Meng, Z., Hakim, G. J., Yang, W. & Vecchi, G. A. Deep Learning Atmospheric Models Reliably Simulate Out-of-Sample Land Heat and Cold Wave Frequencies (2025). URL http://arxiv.org/abs/2507.03176
arXiv 2025
-
[22]
Zhang, Z., Fischer, E., Zscheischler, J. & Engelke, S. Physics-based models out- perform AI weather forecasts of record-breaking extremes.Science Advances12, eaec1433 (2026). URL https://www.science.org/doi/10.1126/sciadv.aec1433. 35
-
[23]
Bonavita, M. On Some Limitations of Current Machine Learning Weather Pre- diction Models.Geophysical Research Letters51, e2023GL107377 (2024). URL https://onlinelibrary.wiley.com/doi/abs/10.1029/2023GL107377
-
[24]
Selz, T. & Craig, G. C. Can Artificial Intelligence-Based Weather Prediction Models Simulate the Butterfly Effect?Geophysical Research Letters50(2023). URL https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2023GL105747
-
[25]
Hakim, G. J. & Masanam, S. Dynamical Tests of a Deep Learning Weather Prediction Model (2024). URL https://journals.ametsoc.org/view/journals/aies/ 3/3/AIES-D-23-0090.1.xml
2024
-
[26]
URL https://joss.theoj.org/papers/10.21105/joss.06323
Kl¨ ower, M.et al.SpeedyWeather.jl: Reinventing atmospheric generalcirculation models towards interactivity and extensibility.Journal of Open Source Software 9, 6323 (2024). URL https://joss.theoj.org/papers/10.21105/joss.06323
-
[27]
D¨ uben, P. D. & Palmer, T. N. Benchmark Tests for Numerical Weather Forecasts on Inexact Hardware.Monthly Weather Review142, 3809– 3829 (2014). URL https://journals.ametsoc.org/view/journals/mwre/142/10/ mwr-d-14-00110.1.xml
2014
-
[28]
& D¨ uben, P
Chantry, M., Thornes, T., Palmer, T. & D¨ uben, P. Scale-Selective Preci- sion for Weather and Climate Forecasting.Monthly Weather Review147, 645–655 (2019). URL https://journals.ametsoc.org/view/journals/mwre/147/2/ mwr-d-18-0308.1.xml
2019
-
[29]
G´ orski, K. M.et al.HEALPix: A Framework for High-Resolution Discretization and Fast Analysis of Data Distributed on the Sphere.The Astrophysical Journal 622, 759 (2005). URL https://iopscience.iop.org/article/10.1086/427976. 36
doi:10.1086/427976 2005
-
[30]
URL https://ams.confex.com/ams/103ANNUAL/meetingapp.cgi/Paper/415842
ARCO-ERA5: An Analysis-Ready Cloud-Optimized Reanalysis Dataset. URL https://ams.confex.com/ams/103ANNUAL/meetingapp.cgi/Paper/415842
-
[31]
Hoyer, S. & Hamman, J. xarray: N-D labeled Arrays and Datasets in Python. Journal of Open Research Software5(2017). URL https://openresearchsoftware. metajnl.com/articles/10.5334/jors.148
doi:10.5334/jors.148 2017
-
[32]
R.et al.Array programming with NumPy.Nature585, 357–362 (2020)
Harris, C. R.et al.Array programming with NumPy.Nature585, 357–362 (2020). URL https://www.nature.com/articles/s41586-020-2649-2
2020
-
[33]
Hunter, J. D. Matplotlib: A 2D Graphics Environment.Computing in Sci- ence & Engineering9, 90–95 (2007). URL http://ieeexplore.ieee.org/document/ 4160265/
2007
-
[34]
Cartopy: a cartographic{Python}library with a{Matplotlib}interface (2010)
Office, M. Cartopy: a cartographic{Python}library with a{Matplotlib}interface (2010). URL https://cartopy.readthedocs.io
2010
-
[35]
URL https://github.com/MakieOrg/GeoMakie
MakieOrg/GeoMakie.jl (2026). URL https://github.com/MakieOrg/GeoMakie. jl. Original-date: 2019-08-04T11:06:51Z
2026
-
[36]
N.et al.The JASMIN super-data-cluster (2012)
Lawrence, B. N.et al.The JASMIN super-data-cluster (2012). URL http:// arxiv.org/abs/1204.3553. ArXiv:1204.3553 [cs.DC]
Pith/arXiv arXiv 2012
-
[37]
URL https://www.jasmin.ac.uk/about/
JASMIN Site. URL https://www.jasmin.ac.uk/about/. 37
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.