{"id":"9c3432cd-c80f-4902-ba5c-50bc65c4444c","arxiv_id":"2505.02506","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Subject to strong caveats, stable 10-year rollouts are achievable for FourCastNet, SFNO, and ClimaX with multi-step training and moderate capacity, while SFNO is the least sensitive to hyperparameters.","lead":"Three popular deep learning weather models were trained and rolled forward for ten years to see which design choices keep them realistic. The result is that multi-step training and moderate model size help stability, but random seeds and variable choices can still break any of them.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Statistical-fidelity claim is under-supported: only time-mean and std RMSE over five variables from one 10-year rollout are checked; spatial, spectral, and joint-distribution statistics are never assessed.","rationale":"The reader's verdict is CONDITIONAL, and my stress-test did not find a reason to move away from that. The paper is transparent about the evaluation protocol, and the existence results (stable configurations exist for each architecture; some seeds diverge) are supported by the figures. The weakest point is the gap between the abstract's 'preserving the statistical properties' and the actual metrics: time-mean and temporal-std RMSE over five variables from one rollout. I considered whether the different learning rate used for SFNO (Sec. A: 1e-3 vs 4e-3) confounds the 'SFNO is most robust' claim; that is a legitimate secondary concern, but the stability counts in Figs. 1-3 are strikingly architecture-dependent even within the chosen grid, so I do not think the LR difference alone overturns the robustness comparison. The single most load-bearing concern remains the statistical-fidelity metric. The proposed test is deliberately cheap relative to the full grid: one stable configuration and three diagnostics would show whether the current claim generalizes beyond first and second marginal moments. Since the concern narrows rather than falsifies, and the reader already conditioned on it, I recommend UNCHANGED.","tokens_in":14687,"tokens_out":8666,"duration_ms":115681,"concrete_test":"Select the representative stable SFNO model (33 variables, 4 autoregressive steps, L=4, D=512, seed 597; Appendix Figs. 8-9) and compute three additional diagnostics for the 10-year rollout: (a) zonal-wavenumber power spectra of 500 hPa geopotential, (b) the leading EOF of daily 500 hPa geopotential (or gridpoint tas-zg500 correlations), and (c) a second 10-year rollout initialized from 1 July 2009. Compare each to the corresponding ERA5 diagnostic with sampling uncertainty. If the spectral slope, EOF pattern, or second-rollout temporal-mean/std RMSE is far from ERA5 while the published mean/std RMSE remains low, the statistical-fidelity and single-initial-condition justifications fail; if all three match within sampling error, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that stable 10-year rollouts 'preserv[e] the statistical properties of the reference dataset' (Abstract). The evaluation section (Sec. 2) defines stability as RMSE between the temporal mean of the rollout and the temporal mean of the target over the same 10-year window, supplemented in Appendix Fig. 4 by the RMSE of the temporal standard deviation. These are first and second marginal moments, averaged over five variables. A model with correct means and variances can still have wrong spatial correlations, no storm-track activity, a flat power spectrum, or incorrect joint dependence between variables (e.g., temperature versus geopotential height), so the 'statistical properties' claim does not follow from the evidence. The reliance on a single initial condition (1 Jan 2009; Sec. 2, 'We refrain to rollout all configurations from multiple initial conditions since we observed in preliminary experiments...') further leaves open the possibility that the reported low-RMSE rollouts are a property of that one trajectory rather than of the learned attractor; the supporting preliminary experiment is not documented. These are scope limitations rather than demonstrated failures, but they are load-bearing because the headline claim is about statistical fidelity, not merely non-divergence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a systematic empirical study of three autoregressive deep-learning weather/climate models (FourCastNet, SFNO, ClimaX) trained on WeatherBench1 ERA5 data at 5.625 degrees resolution. The authors grid-search the number of autoregressive training steps (1, 2, 4), layer count (4, 6, 8), hidden dimension (128, 256, 512), and two prognostic-variable sets (8 and 33 variables), with 10 random seeds per configuration, and evaluate 10-year rollouts initialized on 1 Jan 2009. The primary metric is the area-weighted normalized RMSE between the rollout's temporal mean and the ERA5 temporal mean over 2009-2018, averaged over five variables, with a climatology baseline; Appendix Fig. 4 reports the same comparison for temporal standard deviation. The main empirical findings are that all three architectures can produce rollouts with RMSE near climatology, that multi-step training strongly improves stability, that FCN and ClimaX require moderate hidden dimensions while SFNO benefits from larger capacity, and that SFNO is the most robust across hyperparameters, although every architecture exhibits severe seed-dependent divergence for some configurations.","tokens_in":14866,"tokens_out":3962,"duration_ms":48244,"significance":"If the results hold, the paper provides a useful and unusually broad empirical map of design choices for stable long-horizon autoregressive atmospheric models, complementing prior architecture-specific demonstrations. Strengths include the systematic grid over architecture, training depth, capacity, variable set, and seeds; the use of held-out ERA5 data with a clear climatology baseline; and public code. The paper also honestly documents seed-induced instability across all architectures, which is a valuable caution for practitioners. The main limitation is that 'preserving statistical properties' is operationalized only through first and second marginal temporal moments of five variables from a single initial condition; spatial, spectral, and multivariate statistics are not checked, so the headline claim is stronger than the evidence.","major_comments":[{"comment":"The claim that stable rollouts 'preserv[e] the statistical properties of the reference dataset' (Abstract) is under-supported by the evaluation, which compares only the temporal mean (and, in Appendix Fig. 4, the temporal standard deviation) of five variables via area-weighted RMSE. Models matching these marginal moments can still have incorrect spatial correlations, spectral energy distributions, storm-track variability, or cross-variable joint distributions; I ask the authors to add diagnostics such as spatial pattern correlations, wavenumber spectra, or joint statistics, or to explicitly reframe the claim as stability of the first two marginal moments.","section":"Section 2, Evaluation; Section 3"},{"comment":"The evaluation relies on a single 10-year rollout from 1 Jan 2009, and the statement that multiple initial conditions 'does not effect the long-term behavior' is not supported by any documented experiment. Because the paper's conclusion is about the learned attractor rather than one trajectory, the preliminary experiment should be reported, or rollouts from several initial conditions should be included.","section":"Section 2, Evaluation"},{"comment":"The robustness ranking (for example, 'substantially higher' number of SFNO runs with RMSE near climatology) is based on visual thresholds in Figures 1-3, and the notion of an RMSE score 'close to climatology' is never given a quantitative threshold or uncertainty estimate. Please specify the threshold and, ideally, report confidence intervals or a simple statistical comparison across seeds, since the central ranking claims rest on these counts.","section":"Section 3, Figures 1-3"}],"minor_comments":[{"comment":"The text calls the metric 'mean squared error (RMSE)'; this is inconsistent, and the paper should state whether the plotted values are root mean square errors or mean squared errors.","section":"Section 2"},{"comment":"The figures say 'Traininig Variables' where 'prognostic variables' is meant; this typo appears in several captions.","section":"Appendix Figures 1-5"},{"comment":"The paper should state whether the temporal standard deviation is computed from 6-hourly values or from daily means, and over which period.","section":"Section 2, Evaluation"},{"comment":"The phrase 'does not effect the long-term behavior' should be 'does not affect the long-term behavior.'","section":"Section 2, Evaluation"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a workshop-style empirical study with a broad but relatively shallow evaluation. My main concern is the gap between the abstract's statistical-fidelity claim and the mean/std RMSE evidence; this is fixable by adding diagnostics or softening the claims. I do not see a fatal flaw, and the systematic grid with ten seeds is a genuine strength. I would ask the editor to require the multi-initial-condition evidence and a quantitative threshold for 'close to climatology' before considering the paper for archival publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely useful empirical sweep: three architectures, two variable sets, 1/2/4-step training, three depths and widths, ten seeds, and 10-year rollouts at 5.625 degrees, with failure counts reported for each configuration. Second, the headline claim outruns the evidence: \"preserving statistical properties\" is supported only by time-mean and temporal-standard-deviation RMSE over five variables from a single 10-year rollout. That overreach is correctable and does not sink the practical conclusions.\n\nWhat is actually new: the paper systematically varies autoregressive training steps, capacity, and prognostic variables, and shows which configurations keep 10-year rollouts near ERA5 climatology. Multi-step training helps across architectures; moderate capacity matters for FourCastNet and ClimaX while SFNO tolerates larger hidden dimensions; seed sensitivity is real and can make any architecture diverge; and variable-set choice can flip stability. These are concrete, actionable findings that go beyond Karlbauer et al. and are useful to anyone building coarse-resolution climate emulators. The code is public, the protocol is clearly documented, and the comparisons against climatology as a lower bound are sensible.\n\nThe soft spots are proportionate. The statistical-properties language in the abstract is too strong. What is verified is first and second marginal moments, averaged over five variables, for one initial condition. Spatial correlations, spectra, joint distributions between variables, and multiple-start statistics are not assessed. The authors state that multiple initial conditions do not affect long-term behavior based on unpublished preliminary experiments; that is an assertion, not evidence, though it is not a wild one for an attractor-like rollout. The summary plots average over seeds with finite RMSE, which could flatter configurations where most seeds diverge, but the paper does display the run counts, so a careful reader is not misled. The polar-overfitting explanation for capacity effects is a hypothesis, not a verified mechanism.\n\nNone of these problems break the main practical message. The paper is honest about resolution limits and about stability being only a necessary condition. Citation practice is appropriate and the related work is engaged seriously.\n\nWho is this for? Anyone deciding how to train or evaluate autoregressive DL climate models at coarse resolution. It deserves a serious referee: the experiment is broad, reproducible in principle, and the findings are practically useful. A revision should soften the statistical-fidelity claim, add multi-initial-condition evidence or clearly label the single-trajectory scope, and ideally check at least one spatial or spectral diagnostic. I would take it to peer review and would cite it as an empirical reference point.","headline":"A useful empirical sweep of autoregressive training choices for stable climate rollouts, though the statistical-fidelity claim outruns the metrics used to support it.","tokens_in":15407,"tokens_out":2141,"would_cite":true,"duration_ms":30883,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes that stable 10-year climate rollouts are achievable with all three tested deep-learning weather architectures when training uses multiple autoregressive steps and moderate model capacity, and that SFNO is the most…","keywords":["autoregressive climate models","long-term rollout stability","deep learning weather prediction","ERA5","FourCastNet","SFNO","ClimaX","multi-step training"],"falsifier":"Take a configuration the paper reports as stable, such as SFNO with 4 training steps, 33 prognostic variables, hidden dimension 512, and seed 597, initialize the same trained model on several different dates (for instance 1 January 1979, 1 January 1990, and 1 July 2009), and recompute the area-weighted normalized RMSE of the temporal mean and temporal standard deviation for the five evaluated variables; if any initialization gives an RMSE clearly above the climatology baseline or produces diverging values for any variable, the claim that this configuration preserves reference climate statistics fails.","tokens_in":14463,"feed_emoji":"🌍","tokens_out":10484,"duration_ms":118041,"temperature":0.7,"pith_summary":"Deep-learning weather models trained on reanalysis data typically lose physical consistency after about two weeks, and it has been unclear whether any of them can be trusted for climate-length simulations. This paper asks which design choices decide that outcome, testing three prominent architectures—FourCastNet, SFNO, and ClimaX—on ERA5 at 5.625° resolution with a grid search over autoregressive training steps, layer count, hidden dimension, and the set of prognostic variables, using ten random seeds per configuration. The central finding is that every architecture has configurations whose 10-year rollouts stay close to the ERA5 climatology in both temporal mean and temporal standard deviation, so stability is not reserved for geometry-aware models. The recipe that produces such rollouts is multi-step autoregressive training combined with moderate hidden dimension, while SFNO tolerates a wider range of settings. The paper also reports that all three models can completely diverge for some random seeds, so stability is real but not guaranteed.","feed_headline":"With the right settings, all three weather models run 10 stable years","feed_subtitle":"Multi-step training and moderate size keep rollouts close to ERA5 climatology, though seeds still cause blow-ups.","key_machinery":"The load-bearing mechanism is the multi-step autoregressive training loss, defined as the summed area-weighted mean squared error over M consecutive 6-hour forward steps, with gradients computed only after summing all M steps. This objective forces the model to remain stable under its own errors rather than only one step ahead. Around it, the evaluation uses a single 10-year rollout initialized on 1 January 2009 and compares the rollout's temporal mean and temporal standard deviation to ERA5 via area-weighted normalized RMSE on five variables (tas, uas, vas, ta850, zg500). The three architectures embody different inductive biases: SFNO applies spherical harmonics transforms on the sphere, FourCastNet uses adaptive Fourier neural operators on the latitude–longitude grid, and ClimaX uses a vision transformer with variable tokenization; the paper's grid search over steps, layers, hidden dimension, and variable count is what connects these biases to observed stability.","core_discovery":"On the paper's own terms, the discovery is that long-run stability is a property of the training objective and capacity, not of the architecture's spherical geometry. Ten-year rollouts with area-weighted normalized RMSE close to the climatology baseline are achieved by FourCastNet, SFNO, and ClimaX alike, as long as training unrolls over multiple autoregressive steps (M=2 or 4) and uses a moderate hidden dimension for the non-geometry models (D=128 or 256). SFNO tolerates larger capacity and more variable sets; for SFNO the trend is that bigger hidden dimensions improve the rollout, while FourCastNet and ClimaX overfit near the poles when capacity grows. The paper verifies that stable rollouts also match the temporal standard deviation of the reference period, and it treats climatology as the lower bound because the models are not driven by greenhouse-gas or ocean forcings. At the same time, every architecture contains configurations where some random seeds cause the 10-year rollout to blow up, and stability for one set of prognostic variables does not imply stability for the other.","pith_inferences":["If the same stability plateau holds at higher resolution, the paper's recipe—multi-step training, moderate hidden dimension, and a small prognostic set for dense models—would give a concrete starting point for kilometer-scale learned climate models; the paper explicitly leaves this untested.","The strong seed dependence suggests that averaging several seed-trained models, or adding spectral or noise regularization, might reduce divergence rates; that is a testable extension of the paper's own finding.","Because only one initialization date was used, an immediate stress test is to roll out the same stable configurations from different starting dates and seasons and compare the spread of the 10-year means; preliminary tests by the authors suggest no effect, but the evidence is not shown.","The evaluation metric is a necessary condition rather than a full certificate: checking spatial patterns, spectral energy, and teleconnections would reveal whether low RMSE masks physically wrong variability."],"forward_implications":["Multi-step autoregressive training (two or four 6-hour steps per loss term) should become the default for climate-length rollouts: one-step training rarely reaches RMSE close to climatology.","Non-spherical architectures are viable climate emulators if capacity is kept moderate (hidden dimension 128 or 256), which expands the set of usable backbones beyond those with spherical geometry.","Stability results should be reported per seed: a configuration can look stable on average while individual seeds blow up, so seed counts and failure rates are necessary parts of any long-term stability claim.","A configuration that is stable for one set of prognostic variables can be unstable for another, so the variable set is a design choice that must be tuned together with the architecture.","At this coarse resolution, the identified stable configurations provide a reproducible baseline for studying data-driven climate emulation at decade scales, since they reproduce the reference mean and variability."],"supporting_citations":[{"why":"Supplies the training hyperparameters, the 8-variable prognostic set, and the prior observation that multiple backbones can produce stable long rollouts, which this paper quantitatively extends.","marker":"Karlbauer et al. (2024b)"},{"why":"Provides the WeatherBench1 dataset at 5.625° resolution used for training, validation, and testing.","marker":"Rasp et al. (2020)"},{"why":"Defines the ERA5 reanalysis that is the target dataset and the climatology baseline for the RMSE comparison.","marker":"Hersbach et al. (2020)"},{"why":"Defines the ClimaX architecture and the 33-prognostic-variable selection (minus relative humidity and the 50 hPa level) used in the grid search.","marker":"Nguyen et al. (2023)"},{"why":"Defines the FourCastNet architecture, one of the three backbones whose stability is compared.","marker":"Pathak et al. (2022)"},{"why":"Defines the SFNO architecture whose spherical-harmonics geometry the paper identifies as the most robust.","marker":"Bonev et al. (2023)"},{"why":"Provides the CMIP6 solar-forcing dataset used to compute the top-of-atmosphere incident solar radiation forcing variable.","marker":"Matthes et al. (2017)"},{"why":"Evidence cited for the claim that multi-step training mitigates autoregressive error growth, which the paper confirms for decade-scale rollouts.","marker":"Lam et al. (2023)"}],"fun_headline_variants":["Stable weather AI comes from training design, not geometry","Multi-step training keeps AI weather models stable for a decade","Random seeds break even tuned 10-year weather forecast models","SFNO is robust, but every weather AI model has blow-up seeds","Long-term weather stability is about objective, not architecture"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion assumes that a single 10-year rollout that matches ERA5's temporal mean and standard deviation for five variables is enough to certify that a model has stable, physically consistent climate statistics.","fun_headline_variants_meta":{"raw":{"variants":["Stable weather AI comes from training design, not geometry","Multi-step training keeps AI weather models stable for a decade","Random seeds break even tuned 10-year weather forecast models","SFNO is robust, but every weather AI model has blow-up seeds","Long-term weather stability is about objective, not architecture"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001153,"raw_usage":{"total_tokens":4761,"prompt_tokens":909,"completion_tokens":3852,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":3769}},"tokens_in":525,"tokens_out":3852,"duration_ms":32488,"temperature":1.0,"reasoning_tokens":3769,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:49:50.126074+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a configuration the paper reports as stable, such as SFNO with 4 training steps, 33 prognostic variables, hidden dimension 512, and seed 597, initialize the same trained model on several different dates (for instance 1 January 1979, 1 January 1990, and 1 July 2009), and recompute the area-weighted normalized RMSE of the temporal mean and temporal standard deviation for the five evaluated variables; if any initialization gives an RMSE clearly above the climatology baseline or produces diverging values for any variable, the claim that this configuration preserves reference climate statistics fails.","supporting_citations":[],"review_version":1}