{"id":"34284d55-0a0c-4c3f-953c-619eec34b71e","arxiv_id":"2505.13873","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Baguan shows that a two-frame Siamese masked autoencoder pre-training step reduces overfitting in a plain ViT weather model, improving medium-range forecasts over Pangu-Weather and IFS on the 2018 ERA5 test set.","lead":"A new self-supervised pre-training scheme for weather AI, called Baguan, feeds the model two slightly different time snapshots so it learns local, physically grounded patterns instead of memorizing global noise. The authors report that this reduces overfitting and beats the Pangu-Weather and IFS baselines on 2018 forecasts, with the same backbone then fine-tuned for monthly and regional outlooks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 4.7%/23.2% gains rest on a single test year (2018) and published Pangu/IFS numbers never rerun on Baguan's exact protocol, so a baseline mismatch or a lucky year could erase the claimed edge.","rationale":"The paper has two intertwined contributions: an internal causal claim that Siamese MAE pre-training helps a plain ViT weather model, and an external comparative claim that the resulting Baguan beats Pangu-Weather and IFS by specific margins. The internal claim has independent support from the ablation in Section 4.2.1 and the masking-ratio sweep in Section 4.2.4, although these are run at 1.40625 degree resolution and without error bars. The external claim is where the load-bearing risk sits: the headline percentages in Section 4.3 are computed against published Pangu-Weather and IFS curves on a single year, 2018, with no rerun of either baseline and no demonstration that the comparison protocols match exactly. This is the same weakest assumption the reader identified, so I agree with the reader's analysis. The concern is not internally fatal: a protocol mismatch or an unusual year would undermine the specific quantitative headline, not the architectural or pre-training contributions. That is why the appropriate verdict remains CONDITIONAL rather than REJECT. One concrete check can settle the concern: rerun Pangu-Weather on Baguan's exact evaluation protocol and extend the evaluation to additional years. If the 4.7% advantage persists across years, the conditional can be upgraded; if not, the headline numbers need to be revised or re-scoped. The absence of released code or checkpoints, despite the stated intention to open-source, makes this external verification the only practical path to confidence in the comparative claim.","tokens_in":24011,"tokens_out":7745,"duration_ms":74687,"concrete_test":"Rerun the official Pangu-Weather checkpoint (or its WeatherBench2 reproduction) on exactly Baguan's evaluation protocol: 2018, 00/12 UTC initializations, the same variables and units as Table 3, the same latitude-weighted RMSE and ACC definitions, and lead times from 1 hour to 7 days; then repeat the same protocol on 2019-2021. If the average per-variable RMSE improvement over Pangu-Weather is no longer about +4.7% (or is negative in a second year), the headline claim is not robust to baseline protocol or year choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 states the test set is 2018 at 00/12 UTC and reports Baguan's average 4.7% RMSE improvement over Pangu-Weather and 23.2% over IFS, but it does not recompute either baseline. Section 4.1.1 says Baguan follows Pangu-Weather's training/validation/test split, yet nothing verifies that the published Pangu and IFS curves use Baguan's variable set (Table 3), pressure levels, units, normalization, latitude weighting, or the same climatology period for ACC; any mismatch changes the per-variable RMSE deltas directly. The single test year also makes the reported averages sensitive to circulation regime: an anomalous 2018 could inflate the apparent advantage, and no error bars or per-initialization spread are provided. The paper itself concedes in Section 4.3 that current SOTA models (FengWu, FuXi, GraphCast) are excluded because training codes are lacking, so the baseline set is not the strongest available. The internal pre-training ablation in Section 4.2.1 supports the claim that Siamese MAE helps relative to no pre-training, but it does not validate the external quantitative comparison on which the headline numbers rest. The load-bearing assumption is therefore that the published baseline curves plus the single year 2018 constitute a fair and representative measurement of Baguan's skill advantage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies self-supervised pre-training for data-driven global weather forecasting. It proposes Baguan, a ViT-based model pre-trained with a Siamese masked autoencoder in which the first frame is unmasked and the second frame is masked at ratio r, then fine-tuned for fixed lead times and iteratively fine-tuned for autoregressive rollout. The main claims are: (i) pre-training with a task of intermediate difficulty introduces a locality bias that regularizes attention and reduces overfitting, supported by a controlled comparison of Siamese MAE, MAE, and no pre-training; (ii) Baguan outperforms Pangu-Weather and IFS on 0.25-degree ERA5 forecasts; and (iii) the pre-trained model transfers to S2S and regional forecasting. The paper also includes a linear-regression analysis intended to explain pre-training as a spectral regularizer.","tokens_in":24296,"tokens_out":7973,"duration_ms":74883,"significance":"If substantiated, the core idea that a model-agnostic pre-training stage can inject locality bias and reduce overfitting in a plain ViT weather model is genuinely useful, and it addresses an under-explored axis relative to architectural locality in Swin/GNN designs. The controlled Siamese-MAE versus MAE versus no-pre-training ablation in Section 4.2.1, the masking-ratio sweep in Section 4.2.4, and the model/data scaling analyses in Section 4.5 are valuable and go beyond a single benchmark table. The theoretical regularization analysis in Section 3.5 and Appendix A.12 is a concrete attempt at a mechanism, and the paper states it will release code and checkpoints, which would help reproducibility. The main weakness is that the headline comparisons against Pangu-Weather and IFS rest on a single test year and on published baseline numbers that are not re-run under the paper's protocol; the overfitting evidence is also thinner than the claims require.","major_comments":[{"comment":"The headline comparison against Pangu-Weather and IFS is not a controlled evaluation. The test set is a single year (2018, at 00 and 12 UTC), and neither baseline is re-run under the paper's protocol; there is no evidence that the published Pangu/IFS values use the same variable list (Table 3), pressure levels, normalization, latitude weighting, or ACC climatology as Eqs. (9)-(12). Any mismatch changes per-variable RMSE directly, and a single test year cannot protect against an atypical circulation regime. Please re-run Pangu (and, if possible, IFS/AIFS) under the Baguan protocol or report results through a maintained benchmark such as WeatherBench2, evaluate over multiple years, and report per-initialization spreads or confidence intervals before claiming the 4.7% and 23.2% improvements.","section":"Section 4.3 / Fig. 5"},{"comment":"The paper explicitly excludes FengWu, FuXi, and GraphCast, saying 'we lack training codes for these algorithms'; this weakens the benchmark claim because the strongest available reference results are omitted, and for GraphCast public code exists. If the goal is to demonstrate the value of pre-training rather than to claim a competitive SOTA position, the claims should be reworded to 'compared with published Pangu/IFS numbers' rather than implying a current SOTA ranking, or the missing baselines should be added.","section":"Section 4.3"},{"comment":"The evidence for the core overfitting claim is thin. The reported train-validation loss gap after pre-training (3.6e-4) versus without (4.0e-4) is computed over the last 50 training steps with no variance estimate, no number of seeds, and no significance test; the absolute difference is small relative to the quantities involved. Please report full learning curves, multiple seeds, and a statistical comparison. The attention-map and spectrum analyses in Sections 4.2.2-4.2.3 are qualitative and depend on the stated assumption that attention maps behave like covariance matrices; they should be presented as supporting intuition rather than as direct evidence.","section":"Section 4.2.1"},{"comment":"The S2S result is claimed as a contribution ('achieves SOTA performance in S2S forecasting, surpassing ECMWF-S2S'), but the full experiment is not in this manuscript; it refers to companion paper [10] and to Fig. 6 whose 'detailed results' are also deferred to [10]. A reader cannot verify the data, baselines, metrics, or statistical significance from this paper. Either include the complete S2S protocol and results here, or clearly mark the result as external work and remove it from the contribution list.","section":"Section 4.4.1"},{"comment":"The ACC formula as written is not the standard latitude-weighted ACC: the denominator is sqrt(Σ L f'^2 g'^2), whereas the correct definition is sqrt((Σ L f'^2)(Σ L g'^2)). If Eq. (11) is what was implemented, the ACC curves in Figs. 5 and 10 are not standard ACC; if not, the equation must be corrected. Since ACC is one of the two headline metrics, this needs to be fixed and clarified.","section":"Appendix A.2, Eq. (11)"}],"minor_comments":[{"comment":"Please fix typos: 'exmaple' in Section 3.4, 'flexibity' in the introduction, 'Reigional' in the Table 2 caption, and 'Groud Truth' in Figs. 8 and 12.","section":"Throughout"},{"comment":"The quantities λ_t and λ_{t-1} in Eq. (5) are not defined, and since the relation is only a proportionality, the 'formally defined' task difficulty cannot be computed from the text; please define the λs or state explicitly that Eq. (5) is a schematic.","section":"Section 3.3.1, Eq. (5)"},{"comment":"The right panel caption says the RMSE is 'relative to mask_ratio=1.0', while the text says the results are presented relative to the w/o pre-train version; please make the normalization consistent.","section":"Fig. 3"},{"comment":"The 'Impact of Pre-training' column in Table 1 uses vague categories such as Positive, Negative, and Mildly Positive; please define the criterion and cite the specific experiments behind each entry.","section":"Section 2 / Table 1"},{"comment":"The train/validation curves in Fig. 1(a) lack axis labels and units, which makes the Gap_train/Gap_val comparison unverifiable.","section":"Fig. 1(a)"},{"comment":"After the sentence 'The resulting linear regression model w2 is given by', the displayed equation is labelled w1; this is a typo that should be corrected.","section":"Appendix A.12"},{"comment":"The paper says it intends to open-source the code and checkpoint but provides no repository link; please add one in the final version.","section":"Section 4.1.2 / Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The internal pre-training ablation is the strongest and most original part of the paper, and the masking-ratio and scaling analyses support the qualitative story. The external benchmark section is the weakest: it compares against published numbers on a single test year and omits the strongest available baselines. I would not reject the paper on this basis, but the next version must either re-run the baselines under a common protocol or substantially temper the benchmark claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe new thing here is the Siamese-MAE pretraining recipe: two frames, one unmasked and one heavily masked, with a reconstruction-plus-history objective. The controlled ablation is the real value. Against standard MAE and against no pretraining, Siamese MAE gives a consistent 8–10% relative RMSE gain on the lower-resolution setup, and the effect grows with model scale. That is a useful, reproducible within-subfield result, assuming the promised code and checkpoints actually appear. The masking-ratio sweep (0.5 to 0.99) is an honest exploration of task difficulty, and the scaling analysis helps the story. The paper cites the relevant pretraining work (W-MAE, ClimaX, Prithvi) and positions itself fairly relative to them. The theory section is stylized but it is not the main event.\n\nThe soft spots are real and concentrated in the headline external comparison. The 4.7% improvement over Pangu-Weather and 23.2% over IFS are computed on a single test year (2018) against published numbers, not re-run baselines. Variable sets, pressure levels, normalization, and climatology all matter at the few-percent level, and the paper does not show that the published curves match Baguan's protocol. One unusual year could shift those averages. The paper also explicitly excludes FengWu, FuXi, and GraphCast, so the 'outperforms' claim is against a 2023-era baseline in a 2025 paper. That is addressable, not fatal: the internal ablation supports the pretraining story on its own.\n\nThe overfitting claim is thinner. The train-validation gap measured over the last 50 steps (3.6e-4 vs 4.0e-4) is a small absolute difference and could easily be noise. The attention-map evidence is qualitative and shown for a single center token. The theory has typos and loose steps in Appendix A.12, including a slip where the pre-trained solution is labeled w1. None of this kills the central ablation, but it means the paper overstates how well the mechanism is established.\n\nThe S2S and regional sections read as teasers. They lean on the companion paper [10] and on deployment claims that cannot be checked here. I would trim those or mark them explicitly as pointers.\n\nWho is this for: researchers working on weather foundation models and on pretraining as a regularizer. It deserves a serious referee. My recommendation is to send it to review with major-revision expectations: evaluate on multiple years, use standardized baselines (WeatherBench2) or re-run the prior models under the same protocol, tighten the overfitting evidence, and release code and checkpoints. If those land, the core method is a solid contribution.","headline":"A clean internal ablation of Siamese-MAE pretraining for weather forecasting, wrapped in headline claims against Pangu/IFS that need stronger baselines and multi-year evaluation.","tokens_in":24863,"tokens_out":4240,"would_cite":true,"duration_ms":41499,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Siamese masked-autoencoder pre-training task, tuned to the right difficulty, gives a plain vision-transformer weather model lower error than Pangu-Weather and IFS.","keywords":["weather forecasting","pre-training","Siamese masked autoencoder","overfitting","vision transformer","locality bias","medium-range forecasting","self-supervised learning"],"falsifier":"Re-run the published baselines on the same ERA5 test data with identical pressure levels, variables, and normalization, and extend the evaluation across multiple years (2018–2022) under a standardized protocol; if Baguan's average RMSE advantage over Pangu-Weather and IFS falls below 4.7% and 23.2% or reverses, the headline claim fails. The companion check is the overfitting story itself: if the Siamese-MAE-pretrained Baguan's average 10.1% advantage over its from-scratch twin shrinks or disappears on the multi-year test window, then pre-training is not conferring the claimed generalization benefit.","tokens_in":23779,"feed_emoji":"🌦️","tokens_out":21717,"duration_ms":171041,"temperature":0.7,"pith_summary":"Weather reanalysis data spans only a few decades, so data-driven forecast models tend to overfit by learning arbitrary global correlations; this paper claims that a properly chosen pre-training task can inject locality bias and suppress that overfitting without changing the model architecture. The chosen task is Siamese masked-autoencoder pre-training — reconstructing a future frame with 75% of its patches masked, aided by an unmasked current frame — whose difficulty sits between vanilla MAE and no pre-training. The resulting model, Baguan, is a plain vision transformer that reports an average 10.1% RMSE improvement over its from-scratch twin (vanilla MAE pre-training: 4.5%), lower RMSE than Pangu-Weather for 75.2% of key variables with a 4.7% average improvement, and a 23.2% average improvement over IFS. The paper's case matters because pre-training is model-agnostic: if it really controls overfitting by tuning task difficulty, any vision-transformer-based weather model could adopt it, and the same encoder transfers to subseasonal and kilometer-scale regional forecasting.","feed_headline":"Baguan beats Pangu and IFS with one pretraining trick","feed_subtitle":"Tuned self-supervised pretraining curbs overfitting on four decades of weather data and transfers to seasonal and regional forecasts.","key_machinery":"The load-bearing object is the Siamese masked-image-modeling pre-training task: a weight-shared encoder processes an unmasked frame $X_{t_0}$ alongside a frame $X_{t_0+\\Delta t}$ with 75% of patches masked, and the cross-self decoder reconstructs the masked frame by letting its tokens attend to the source frame. Its defining property is task difficulty, formalized as $D_{\\text{task}} \\propto 1/(\\lambda_t(1-r_t)+\\lambda_{t-1}(1-r_{t-1}))$: vanilla MAE sets $r_{t-1}=1$ (the past frame carries no information), while Siamese MAE sets $r_{t-1}=0$, placing the task between the too-easy MAE and the hardest case of no pre-training. The masking ratio $r$ is the difficulty dial (0.75 chosen for stability; 0.95 slightly better but loss-spiky). The theoretical support is an explicitly simplified linear-regression analogue: a denoising filter $M^*=\\Sigma(\\Sigma+\\gamma I)^{-1}$ shrinks small-eigenvalue directions, and the paper assumes transformer attention maps behave like that covariance matrix, with the top-k% spectral energy of attention scores as the measurable signature of overfitting control. The three-stage training pipeline (pre-train, fixed-lead fine-tune, iterative autoregressive fine-tune) and the pruned lead-time ensemble strategy carry the operational performance.","core_discovery":"Baguan is a vision-transformer weather model whose encoder is pre-trained by a Siamese masked autoencoder: the atmospheric state at $t_0$ is fed unmasked while the state at $t_0+\\Delta t$ has 75% of its patches masked with Gaussian noise, and a cross-self decoder must reconstruct the masked frame with the two frames sharing one encoder. The paper's central claim is that this task sits at the right difficulty level and is what introduces locality bias into the attention maps, which controls overfitting; the masking ratio is the difficulty dial, with $r=0.95$ performing best but being unstable, so $r=0.75$ is chosen. After fixed-lead-time fine-tuning (1, 6, 24 hours) and autoregressive iterative fine-tuning, Baguan reports lower RMSE than Pangu-Weather for 75.2% of key variables (average 4.7%) on the 2018 test set, consistently beats IFS (average 23.2%), and the pre-training stage alone delivers an average 10.1% RMSE improvement over the same model trained without pre-training, versus 4.5% for vanilla MAE. The causal story is supported by a linear-regression analysis in which denoising pre-training prunes the small-eigenvalue subspace of the covariance matrix, improving the generalization bound from $O(d^{1/4}/n^{1/2})$ to $O(1/\\sqrt{n})$, and by attention evidence that Siamese MAE activates 0.25% of tokens versus 1.61% for MAE. The paper is explicit that it did not compare against GNN- or local-attention-based systems such as GraphCast, FengWu, and FuXi because it lacks training codes for a fair comparison, and that the S2S results come from a companion paper.","pith_inferences":["If locality bias is the active ingredient, grafting Siamese MAE pre-training onto models that already build in locality — Swin-based or GNN-based systems — should compound the gain; the paper names exactly this combination as future work, making it the most direct test of the mechanism.","The masking-ratio pattern (0.95 best but unstable, 0.75 chosen) implies a difficulty sweet spot; an untested extension is an annealing schedule that raises the mask ratio during pre-training to reach the better optimum without the observed loss spikes.","The companion S2S result implies the pre-trained encoder retains skill in low signal-to-noise regimes where the paper argues overfitting is worst; a direct test is whether Siamese-MAE pre-training alone, without the companion's specialized optimization, already improves S2S skill over from-scratch training.","The attention-energy diagnostic could double as a general overfitting monitor for weather models — a model whose attention stays diffuse on validation data is likely memorizing — an application the paper does not itself develop."],"forward_implications":["Pre-training becomes an architecture-agnostic remedy for overfitting: any vision-transformer-based weather model that currently skips an independent pre-training stage could add Siamese MAE without redesigning for locality.","The benefit scales with capacity and resolution: pre-training's contribution grows from +4.20% to +10.1% as models get larger at 5.625°, and moving to 0.25° inputs cuts validation MSE by 38.8–74.0% relative to coarser grids.","The pre-trained encoder is a transferable foundation: fine-tuned Baguan reports S2S skill above ECMWF-S2S (via the companion optimization approach) and a 57.8% average RMSE improvement over EC-IFS on 0.05° regional forecasting, including variables absent from pre-training.","Mixing lead-time combinations such as [6,6,6,6] and [24] with a pruning rule adds roughly 2% accuracy at longer leads, directly attacking the error accumulation of autoregressive rollout.","The train-validation loss gap shrinks with pre-training (3.6e−4 versus 4.0e−4), and attention energy concentrates in leading eigenvectors — two measurable signs the paper offers for overfitting control."],"supporting_citations":[{"why":"Pangu-Weather is the primary headline baseline and the source of the data split (1979–2015 train, 2016–2017 validation, 2018 test) and the 1-hourly data convention.","marker":"[2]"},{"why":"Supplies the Siamese masked autoencoder method — two weight-shared frames with asymmetric masking — that Baguan's pre-training stage adapts to weather data.","marker":"[11]"},{"why":"The vanilla MAE pre-training baseline whose 4.5% average improvement is compared against Siamese MAE's 10.1%.","marker":"[12]"},{"why":"The ERA5 reanalysis is the sole data source for pre-training, fine-tuning, and testing.","marker":"[14]"},{"why":"GraphCast provides the iterative fine-tuning precedent (raising rollout steps from 2 to 7) and is the GNN-family system excluded from the headline comparison.","marker":"[16]"},{"why":"ClimaX contributes the weather-specific variable aggregation and serves as the comparison model at 1.40625° and 5.625° resolution.","marker":"[21]"},{"why":"The companion paper that produces Baguan's S2S results, including the claim of beating ECMWF-S2S; the S2S experiments are not re-derived here.","marker":"[10]"},{"why":"The reported result that plain MAE pre-training significantly hinders subsequent weather forecasting, which motivates the paper's task-difficulty argument.","marker":"[26]"}],"fun_headline_variants":["Baguan's Siamese pretraining cuts weather overfitting","Locality bias from masked autoencoder boosts weather forecasts","Weather model beats Pangu and IFS with smarter pretraining","One masking trick curbs overfitting in weather AI","Self-supervised pretraining slashes weather model overfitting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the published Pangu-Weather and IFS scores, evaluated on a single test year (2018) without re-running either baseline, are a fair and representative basis for the claimed 4.7% and 23.2% improvements; if those published numbers used different variable sets, pressure levels, or normalization, or if 2018 was atypically easy, the headline margins overstate Baguan's real skill advantage.","fun_headline_variants_meta":{"raw":{"variants":["Baguan's Siamese pretraining cuts weather overfitting","Locality bias from masked autoencoder boosts weather forecasts","Weather model beats Pangu and IFS with smarter pretraining","One masking trick curbs overfitting in weather AI","Self-supervised pretraining slashes weather model overfitting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000274,"raw_usage":{"total_tokens":1729,"prompt_tokens":1124,"completion_tokens":605,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":740,"completion_tokens_details":{"reasoning_tokens":525}},"tokens_in":740,"tokens_out":605,"duration_ms":5528,"temperature":1.0,"reasoning_tokens":525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:08:38.855132+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the published baselines on the same ERA5 test data with identical pressure levels, variables, and normalization, and extend the evaluation across multiple years (2018–2022) under a standardized protocol; if Baguan's average RMSE advantage over Pangu-Weather and IFS falls below 4.7% and 23.2% or reverses, the headline claim fails. The companion check is the overfitting story itself: if the Siamese-MAE-pretrained Baguan's average 10.1% advantage over its from-scratch twin shrinks or disappears on the multi-year test window, then pre-training is not conferring the claimed generalization benefit.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Siamese masked autoencoder method — two weight-shared frames with asymmetric masking — that Baguan's pre-training stage adapts to weather data."},{"cited_title":"Gupta, and Aditya Grover","cited_arxiv_id":null,"evidence_quote":"ClimaX contributes the weather-specific variable aggregation and serves as the comparison model at 1.40625° and 5.625° resolution."}],"review_version":1}