{"id":"4ce24b5a-3df5-45e5-bd1f-572073d7d603","arxiv_id":"2507.18378","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Stretched-grid and limited-area machine-learning weather models are competitive for Europe, with stretched-grid models showing better generalization to unseen forecast times and limited-area models benefiting from external boundary forcing.","lead":"This paper compares two machine-learning designs for regional weather forecasting: limited-area models and stretched-grid models, trained on the same data and architecture. It finds both are accurate, but stretched-grid models generalize better to unseen forecast times, while limited-area models exploit high-quality boundary forecasts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal-generalisability claim is not isolated: the extended-times experiment varies global-domain exposure together with normalization, hidden-grid resolution, and loss weighting, so SGM's advantage cannot be uniquely attributed to global training data.","rationale":"The reader's conditional verdict is appropriate, and my stress-test identifies the same weakest point: Section 4.3's mechanism for SGM's temporal-generalisation advantage is an inference, not a demonstrated result. The paper is otherwise careful: it uses a shared Anemoi framework, near-identical training, multiple forcing experiments (ideal, operational, pragmatic, mixed), and openly lists limitations in Section 4.2. The computational-efficiency and boundary-quality results are supported by direct measurements. The main risk is that the abstract's 'significantly surpasses ... temporal generalisability' overstates a single-variable, single-lead-time, confounded comparison. I keep the conditional verdict: acceptance should require either softening the causal language or running the normalization-matched LAM ablation described above, ideally with additional attention to the other structural differences.","tokens_in":21411,"tokens_out":8440,"duration_ms":88413,"concrete_test":"Train a LAM-512 variant using the SGM's per-variable normalization statistics (global ERA5 mean/std) while keeping the LAM graph, boundary size, and regional-only loss unchanged, and repeat the extended-times experiment at 03/09/15/21 UTC. If the SGM advantage in 2m-temperature MSE over Northern Africa disappears or shrinks materially, the temporal-generalisability claim cannot be attributed to global training data; if it persists, one known confound is ruled out and the global-data explanation gains support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim that SGM 'significantly surpasses LAM in terms of (temporal) generalisability' rests on the extended-times experiment (Sections 2.4.4 and 3.3.5). In that experiment, SGM achieves lower 2m-temperature MSE than LAM at shifted initialization times (03/09/15/21 UTC), and Section 4.3 attributes this to SGM's access to a global domain with a full diurnal cycle across time zones. That attribution is not demonstrated, because LAM and SGM differ simultaneously in at least three relevant ways besides the presence of the global domain: (i) per-variable normalization uses CERRA regional statistics for LAM and ERA5 global statistics for SGM (Section 2.3); (ii) the LAM hidden grid has fixed refinement nine over the 600-km boundary extension, giving higher boundary resolution than the SGM's mixed-resolution global/regional hidden grid (Section 2.2); and (iii) the SGM loss is a 25/75 regional/global weighted sum, while the LAM loss is regional-only (Section 2.3). The extended-times experiment varies all of these at once, so the observed SGM advantage over Northern Africa cannot be uniquely attributed to global training data. The supplementary LAM retrained on shifted times (Fig. 14c-d) changes the training distribution rather than adding global coverage, so it does not control for these confounds. The causal mechanism in Section 4.3 is therefore inferred, not established, and the abstract overstates the finding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a controlled comparison of two graph-neural-network approaches to regional machine-learning weather prediction over Europe: a limited-area model (LAM), which forecasts only on the regional domain and receives lateral boundary forcings from an external global model, and a stretched-grid model (SGM), which forecasts on a global domain with a refined regional subdomain. Both are built in the Anemoi framework with a shared architecture, training data, and training procedure, then evaluated on 2020 test data through several inference experiments: ideal (reanalysis initialization/forcing), operational-like (IFS-HRES analysis/forecasts), pragmatic and mixed LAM forcing choices, and an extended-times experiment initialized at times of day not seen during training. The main findings are that the two approaches are broadly competitive on the regional domain; LAM benefits from high-quality boundary forcings, especially at later lead times; SGM is fully self-contained, is computationally comparable, and reportedly shows superior temporal generalisability, attributed to its access to global training data. The paper concludes with practical recommendations for meteorological institutes choosing between the two designs.","tokens_in":21710,"tokens_out":4215,"duration_ms":45802,"significance":"If the results hold, this is a useful and timely benchmark for the MLWP community and for national meteorological services considering operational regional forecasting. The study's strengths are its unusually controlled setup: the same architecture family, data sources, training schedule, rollout procedure, and verification code are used for both model types, and the authors explicitly report computational costs and memory footprints. The use of multiple inference experiments, including replacing reanalysis with operational IFS data and swapping LAM boundary forcings, is informative and goes beyond a single scoreboard comparison. The bootstrap-based significance testing for RMSE skill scores in Section B.2 is a welcome methodological element. However, the headline claim of SGM's superior temporal generalisability is considerably narrower than the abstract suggests, and the causal attribution to global training data is not isolated from other design differences. The paper is therefore valuable as a comparative engineering study, but the strongest advertised conclusion needs either additional controlled experiments or substantial qualification.","major_comments":[{"comment":"The claim that SGM 'significantly surpasses LAM in terms of (temporal) generalisability' is stated as a general property, but the supporting evidence in Section 3.3.5 and Figure 14 is limited to 2-meter temperature at a single +6 h lead, for 512-channel models, with the advantage localized mainly over Northern Africa. No confidence intervals or effect sizes are reported for the MSE differences in Figure 14, and the text only states that results are 'similar' for the 1024-channel models without showing them. The word 'significantly' is therefore unsupported in its present form. Please either restrict the claim to the specific variable, lead time, model size, and region, or provide a multi-variable, multi-lead analysis with significance testing and with the 1024-channel results shown.","section":"Abstract, Section 3.3.5, Section 5"},{"comment":"The attribution of SGM's temporal-generalisability advantage to its global training data is not demonstrated by the extended-times experiment. The experiment varies the presence of the global domain together with at least three other differences that are known to affect model behaviour: per-variable normalization uses regional (CERRA) statistics for LAM and global (ERA5) statistics for SGM (Section 2.3); the LAM hidden grid has a fixed refinement-nine mesh extending 600 km into the boundary zone, giving higher boundary resolution than the SGM's mixed-resolution hidden grid (Section 2.2); and the SGM loss is a 25/75 regional/global weighted sum while the LAM loss is regional-only (Section 2.3). The supplementary LAM retrained on shifted times of day (Figure 14c-d) changes the training distribution but does not add global coverage, so it does not control for these confounds. The causal mechanism in Section 4.3 is therefore inferred rather than established. Please add ablation or control experiments, such as LAM trained with ERA5 normalization, SGM trained with CERRA normalization, a LAM with a coarser boundary hidden grid, or a SGM with different regional loss weights, or substantially soften the causal interpretation.","section":"Sections 2.4.4, 3.3.5, and 4.3"},{"comment":"The operational-like inference experiments are performed only with models without rollout training, while the main ideal-inference comparisons use rollout-trained models. The paper acknowledges this in Section 4.1 and notes that fine-tuning or rollout training could change the relative performance at later lead times. However, the abstract and conclusion present LAM's later-lead advantage under boundary forcing as a robust strength without carrying this caveat. Since the operational setup is the one most relevant to the recommended use cases, the conclusions about LAM's advantage in operational-like settings should either be supported by experiments with rollout-trained, fine-tuned models or be explicitly framed as applying to the non-rollout configuration used here.","section":"Sections 3.3.3, 3.3.4, and 4.1"}],"minor_comments":[{"comment":"The CERRA available period is listed as 'Sep. 1984 - 2020'; '2020' should be 'December 2020' to be unambiguous.","section":"Table 1"},{"comment":"The units for surface pressure and mean sea level pressure are listed as 'm s^-1'; these appear to be a copy-paste error and should be pressure units (Pa or hPa).","section":"Table A.3"},{"comment":"The caption and text should state the lead time, model size, and whether the displayed MSE differences were tested for significance; the current caption only identifies the model pair.","section":"Figure 14 and Section 3.3.5"},{"comment":"Reference [33] is cited as 'Private discussions'; this is not verifiable by readers and should be replaced with a public source or removed.","section":"Reference list"},{"comment":"The extended-times experiment is described as using models without rollout training, but this is not restated in Section 3.3.5 when the results are interpreted; a brief reminder would help readers connect the experiment design to the interpretation.","section":"Section 2.4.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid, well-executed comparison study, and the shared Anemoi framework gives it a clear advantage over earlier independent LAM/SGM comparisons. The main issue is not the quality of the experiments as a benchmark, but the gap between the strong causal claim in the abstract and the confounded evidence behind it. I see no circularity and no reason to doubt the integrity of the experiments. The revision should focus on either adding targeted ablation/control experiments for the temporal-generalisability attribution or substantially rewording the abstract and conclusions to describe a narrower, observationally grounded finding. The paper is within scope for the journal and, with this revision, would be a useful contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is the first direct, matched comparison of limited-area and stretched-grid models for regional machine-learning weather prediction, and it earns a careful read. Using a shared Anemoi framework, the authors isolate the structural difference reasonably well and show that both approaches are competitive over Europe. That is genuinely useful for operational choices. The mixed boundary-forcing experiment — feeding LAM with SGM's own global forecasts — is a nice touch, and the extended-times experiment is new, even if not fully controlled.\n\nWhat the paper does well: a near-identical training setup, bootstrap-tested skill scores, two model sizes, and an honest limitations section. The computational comparison is also informative: the traditional NWP advantage of limited-area domains does not translate directly to the ML setting. The main comparative results — LAM benefiting from high-quality boundary forcing, SGM being self-contained slightly faster to train — are well supported.\n\nNow the soft spots. The abstract says SGM \"significantly surpasses LAM in terms of (temporal) generalisability.\" That overstates what the experiment shows. The support is 2-metre temperature over Northern Africa at a single 6-hour lead, and the comparison is confounded: LAM and SGM differ in per-variable normalization statistics, hidden-grid resolution over the boundary, and loss weighting, all at once. The stress-test note is right that the advantage cannot be uniquely attributed to the global domain. The supplementary LAM retrained on shifted times changes the training distribution but does not control for these factors, so the mechanism in Section 4.3 is inferred, not established. The operational comparison is also explicitly preliminary: models without rollout or fine-tuning, initialised from interpolated IFS analysis. The authors flag this, but it limits the operational conclusions. A minor concern: hand-tuned hyperparameters, like the 25% regional loss weight, could shift the relative ranking; that is acceptable for a first comparison but worth noting.\n\nThe central comparison holds up, and the paper's own caveats are honest. The temporal-generalisability claim needs to be softened or backed by an ablation that varies global exposure while holding normalization and loss weighting fixed.\n\nWho is this for? Anyone at a meteorological institute choosing between LAM and SGM for operational regional MLWP, and researchers working on regional data-driven forecasting. It deserves a serious referee, and I would recommend acceptance after revision. The revision should reword the abstract and Section 4.3 to match the evidence and acknowledge the confounds in the extended-times experiment.","headline":"First fair head-to-head of LAM and SGM for regional MLWP; solid comparison, but the headline claim about temporal generalisability outruns the evidence.","tokens_in":22279,"tokens_out":1781,"would_cite":true,"duration_ms":19917,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Limited-area and stretched-grid machine-learning weather models are competitive over Europe, but the stretched-grid model generalises better to forecast times it has never seen.","keywords":["machine learning weather prediction","graph neural network","limited-area model","stretched-grid model","regional weather forecasting","temporal generalisability","regional reanalysis","boundary forcing"],"falsifier":"Retrain the limited-area model with the same per-variable normalisation statistics and boundary hidden-grid resolution as the stretched-grid model, then re-run the shifted-times-of-day forecasts. If the limited-area model still loses on those unseen hours, the global-time-zone explanation stands; if the gap closes, the difference is due to the setup rather than to global data exposure.","tokens_in":1540,"feed_emoji":"🌦️","tokens_out":1644,"duration_ms":107557,"temperature":0.7,"pith_summary":"This paper compares two ways of building a regional machine-learning weather forecaster. The limited-area model (LAM) predicts only over Europe and, at every step, receives new boundary conditions from an external global model; the stretched-grid model (SGM) is one graph-neural-network model that forecasts the whole globe at coarse resolution and Europe at high resolution. It claims the two designs are competitive over Europe, with the LAM exploiting high-quality boundary forcings to stay accurate at later lead times, while the SGM is fully self-contained, benefits from more training data, and significantly outperforms the LAM when forecasting at times of day not seen during training. If true, a meteorological institute can choose between the two based on its data access and operational needs rather than on a large intrinsic accuracy gap.","feed_headline":"Generalizes better: stretched-grid weather AI beats limited-area rival","feed_subtitle":"A shared-architecture test over Europe finds both designs skilful; the globe-trained stretched grid adapts best to unseen hours.","key_machinery":"The machinery is the pair of graph constructions. Both models use the same encoder-processor-decoder graph-neural-network design on a high-resolution regional grid over Europe, with a coarse grid outside the region. The SGM connects the regional and global nodes into one graph whose hidden processor mesh is refined only near Europe, so one model predicts the entire globe and can be rolled out with no external input. The LAM replaces the global domain with a relatively small boundary zone, gives the hidden mesh a uniformly high resolution that extends deep into that zone, and at each rollout step replaces boundary fields with values from an external global model, a process called boundary forcing. These graph differences define data exposure and coupling to external forecast information, and they are the mechanisms through which the paper explains the performance split: SGM's global exposure aids temporal generalisation and synoptic-scale variables, while LAM's forcing flexibility aids later-lead-time accuracy.","core_discovery":"The central discovery is that, using a shared graph-neural-network architecture and near-identical training, LAM and SGM are comparable deterministic regional forecasters, but their differences are application-dependent. SGM beats LAM for synoptic-scale fields and generalises better to shifted initialisation times; LAM matches or beats SGM for some small-scale surface variables and benefits from superior boundary forcings at later lead times, though this advantage depends on the quality of the external forcing. The paper attributes SGM's temporal generalisability to its exposure to a full 24-hour daily cycle across global time zones, which gives it training signal for local solar cycles that the region-only LAM lacks.","pith_inferences":["The paper's attribution implies a concrete experiment: retrain a LAM with the SGM's global normalisation statistics and boundary hidden-grid resolution; if it still loses on shifted forecast times, the global-time-zone training signal is the cause, and if the gap closes, the design differences share responsibility.","The same logic suggests that a regional model covering multiple climate zones or continental dry regions could partially recover the SGM's temporal-generalisation advantage without needing a global domain.","Because the SGM's global structure is reusable for transfer learning, the same architecture could serve as a shared base for many regional reanalyses, something the paper mentions as a promising direction.","The LAM's later-lead-time advantage under ideal reanalysis forcing may overstate its operational edge; the paper's operational-like experiment suggests that fine-tuning on operational data would be needed to see the true ranking."],"forward_implications":["National weather services with limited access to global training data can realistically build a regional machine-learning forecasting system around the LAM design without sacrificing short-range regional skill.","Choosing the SGM design removes the need for real-time external boundary forecasts, simplifying operational deployment and technical maintenance.","A LAM's later-lead-time skill is only as good as its boundary source; pairing it with a strong global model gives it an edge that disappears when it is forced by a weaker one.","Both designs gain from more parameters, and the SGM's larger global training set gives it extra headroom when scaling up; the paper estimates that moving from the coarse test grid to a standard finer global grid would increase its additional training data from 3% to 44%.","Deterministic MSE-trained versions of both designs smooth out forecast variability and underrepresent extremes, so applications sensitive to extremes should plan for that limitation."],"supporting_citations":[{"why":"introduced graph-based machine-learning limited-area models and the multi-scale mesh graph on which the LAM here is built.","marker":"[30]"},{"why":"introduced the stretched-grid approach for regional data-driven weather modelling that the SGM here adapts.","marker":"[28]"},{"why":"supplies the graph-neural-network architecture shared by both models.","marker":"[20]"},{"why":"supplies the loss weighting, rollout training, and model configuration that make the LAM/SGM comparison near-identical.","marker":"[21]"},{"why":"describes the European regional reanalysis that defines the high-resolution regional domain and the training and verification target.","marker":"[35]"},{"why":"describes the global reanalysis used for the SGM's global domain and for LAM boundary forcing.","marker":"[17]"},{"why":"documents the shared open-source machine-learning weather prediction platform that allowed both model types to be trained and evaluated in a common setup.","marker":"[11]"}],"fun_headline_variants":["Stretched-grid AI generalizes better than limited-area in regional test","Regional weather AI: both designs skilful, stretched-grid adapts to new times","Global training helps stretched-grid AI generalize in regional forecast test","Test: stretched-grid AI has edge in temporal generalization over limited-area"],"cache_read_input_tokens":24320,"weakest_assumption_plain":"The stretched-grid model's better handling of forecast times never seen in training is credited to its access to the whole globe's daily cycle, but the limited-area model also differs in how variables are normalised and in how finely its boundary is resolved, so the cause is inferred, not shown.","fun_headline_variants_meta":{"raw":{"variants":["Stretched-grid AI generalizes better than limited-area in regional test","Regional weather AI: both designs skilful, stretched-grid adapts to new times","Global training helps stretched-grid AI generalize in regional forecast test","Test: stretched-grid AI has edge in temporal generalization over limited-area"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000621,"raw_usage":{"total_tokens":2895,"prompt_tokens":977,"completion_tokens":1918,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":1841}},"tokens_in":593,"tokens_out":1918,"duration_ms":15106,"temperature":1.0,"reasoning_tokens":1841,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:12:51.116667+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the limited-area model with the same per-variable normalisation statistics and boundary hidden-grid resolution as the stretched-grid model, then re-run the shifted-times-of-day forecasts. If the limited-area model still loses on those unseen hours, the global-time-zone explanation stands; if the gap closes, the difference is due to the setup rather than to global data exposure.","supporting_citations":[{"cited_title":"The ERA5 global reanalysis","cited_arxiv_id":null,"evidence_quote":"describes the global reanalysis used for the SGM's global domain and for LAM boundary forcing."},{"cited_title":"Dramsch et al.Introducing Anemoi: a new collaborative framework for ML weather forecasting","cited_arxiv_id":null,"evidence_quote":"documents the shared open-source machine-learning weather prediction platform that allowed both model types to be trained and evaluated in a common setup."}],"review_version":2}