Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

A multi-scale loss formulation for learning a probabilistic model with proper score optimisation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A scale-split training loss lets a probabilistic machine-learned weather model keep its overall forecast skill while suppressing spurious small-scale variability, bringing forecast spectra closer to the verifying analysis.

desk verdict A clean, practical multi-scale loss for probabilistic ML weather models; the AIFS gains are real but the attribution to scale decomposition needs a control run with smoothing alone. read the letter →

arxiv 2506.10868 v1 pith:TEZYOUS2 submitted 2025-06-12 physics.ao-ph cs.AI

Simon Lang , Martin Leutbecher , Pedro Maciel This is my paper · ORCID
classification physics.ao-phcs.AI PACS 92.60.Wc
keywords multi-scalelossproperscoringrulecontinuousrankedprobabilityscoreafCRPSprobabilisticweatherforecastingmachinelearningspatialscaledecompositionforecastspectra
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a probabilistic machine-learned weather forecasting model can be improved not by changing its architecture but by changing what its training loss sees. Standard proper-score training applies the almost fair continuous ranked probability score (afCRPS) pointwise to the full forecast field, which ignores spatial scale. The authors replace this with a two-scale loss that scores a smoothed large-scale field and the small-scale residual separately. They claim the multi-scale loss leaves overall forecast skill unchanged while reducing spurious small-scale variability, so forecast spectra lie closer to the verifying analysis. If true, scale-aware training is a cheap way to get more realistic fields from proper-score optimisation.

What carries the argument

The load-bearing object is a family of ordered smoothing operators $D_i$ that partition a field $\phi$ into $n$ scales, $\phi_{\mathrm{scale}1}=D_1(\phi)$, $\phi_{\mathrm{scale}2}=D_2(\phi)-D_1(\phi)$, ..., $\phi_{\mathrm{scale}n}=\phi-D_{n-1}(\phi)$, with the loss $L_{n\text{-scale}}=\sum_i \zeta_i \int_M S(\{x_{j,\mathrm{scale}i}\},y_{\mathrm{scale}i})\,d\mu$. In the experiments $n=2$, the smoother is a Gaussian kernel with standard deviation eight grid spacings, and the two scale losses are weighted equally. This machinery makes the training objective sensitive to where forecast error lives in wavenumber space, which the pointwise afCRPS is not.

What would settle it

Retrain the reference model with the same Gaussian smoothing (standard deviation eight grid spacings) applied to the predictions and targets before computing the scale-unaware CRPS; if the small-scale spectra collapse to the reanalysis without per-scale weighting, the scale decomposition itself is not the active ingredient.

Watch

Extended reading notes

Core claim

The central discovery is that splitting the afCRPS training objective by spatial scale changes what a probabilistic weather model learns about variability without sacrificing skill. The loss is built from smoothing operators: the first scale is a Gaussian-smoothed field and the second is the difference between the full field and the smoothed field, with the afCRPS computed on each and summed with equal weights. When used to train AIFS-CRPS, the scale-unaware and multi-scale versions produce nearly identical fair CRPS scores across lead times, variables and regions, but individual fields and power spectra show that the multi-scale model generates less small-scale variance and matches the reanalysis more closely, especially for smooth fields like 500 hPa geopotential.

Load-bearing premise

The benefit of the multi-scale loss is attributed to the scale decomposition, but no control experiment applies the same smoothing to the scale-unaware loss, and the smoothing width and equal scale weights were chosen without a sensitivity study.

Editorial extensions

If this is right

  • Forecast skill, measured by fair CRPS, stays the same while forecast fields become physically more realistic, so the loss appears to trade only spurious variability, not predictive accuracy.
  • The multi-scale formulation is generic: it applies on any manifold with suitable smoothing operators or spectral transforms, so it can be carried to other gridded prediction tasks and to more than two scales.
  • Increasing the number of scales is a natural next step, since the reported spectra still show a small-scale offset between forecasts and reanalysis.
  • The scale weights give the same kind of control that diffusion models get from their noise schedule, letting a proper-score trainer emphasise or suppress specific spatial scales.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported improvement is not yet isolated from the smoothing itself: a control that adds the same Gaussian smoothing to the scale-unaware loss would show whether per-scale weighting is the active ingredient or merely the smoothing regularises the field.
  • Because the kernel width and equal weights are fixed without sensitivity analysis, the two-scale result is best read as a proof of concept; tuning these hyper-parameters per variable or region might change the size, or even the sign, of the effect.
  • The near-identical CRPS curves suggest the score is dominated by large scales, so scale-aware terms may act as a regulariser that the CRPS does not penalise; an interesting extension is to monitor score decomposition by scale during training.
  • A direct test of the mechanism would be to compare the multi-scale loss against a spectral CRPS term that penalises the power spectrum directly, to see whether wavenumber-space constraints are interchangeable with real-space scale splitting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces a multi-scale loss formulation for training probabilistic machine-learned weather forecasting models that optimize proper scores. The loss decomposes prediction and target fields into scales via smoothing operators D_i and computes a weighted sum of the almost fair continuous ranked probability score (afCRPS) on each scale component. A 1D simulation with monochromatic sine waves shows that a 3-scale loss is sensitive to the target wavenumber, whereas the scale-unaware loss is invariant. In AIFS-CRPS experiments at N320 resolution, a two-scale version (Gaussian smoother with sigma equal to eight grid spacings, equal weights zeta_i=1) yields nearly identical fair CRPS curves for 500 hPa geopotential, 850 hPa temperature, and windspeed, but improves the small-scale power spectra of 12h forecasts for geopotential and temperature, bringing them closer to ERA5. The paper concludes that the multi-scale loss better constrains small-scale variability without negatively impacting forecast skill.

Significance. If the claimed effect holds, the multi-scale loss is a simple and computationally cheap modification to proper-scoring-rule training that improves the physical realism of machine-learned weather forecasts without degrading CRPS skill. The paper benefits from a clean 1D illustrative experiment, evaluation on a state-of-the-art operational model (AIFS-CRPS) at a realistic resolution, and use of independent 2019 data. The authors also openly acknowledge remaining hyperparameter work in Section 4. However, the experimental evidence for the central causal claim is incomplete: the design does not isolate the effect of the scale decomposition from the smoothing operator, lacks uncertainty quantification, and is based on a single hand-chosen hyperparameter configuration. These gaps currently prevent a strong endorsement of the paper's headline conclusion.

major comments (3)
  1. [Section 2.2.1 / Section 3 (Eq. 2, Figs. 5–7)] The central claim that the multi-scale loss 'better constrains small scale variability' is not isolated by the experimental design. The two-scale loss in Eq. (2) with n=2 differs from the scale-unaware loss in Eq. (1) in three simultaneous respects: it applies a Gaussian smoother with standard deviation of eight grid spacings to both prediction and target, it adds a second afCRPS term evaluated on the residual field (phi - D_1(phi)), and it reweights the terms with equal weights zeta_i = 1. Any of these changes, in particular the smoothing alone, could reduce spurious small-scale variance (e.g., as label smoothing or regularization against analysis noise). To attribute the improvement to the scale decomposition, the authors should include a control experiment with the same Gaussian smoother applied in a scale-unaware loss (e.g., afCRPS on the filtered field only, or afCRPS plus a smoothing penalty). Without this control, Figures 5–7 support the conclusion that the modified training objective improves spectral representation, but not that the scale decomposition is the operative mechanism.
  2. [Section 3 (Figs. 5 and 7)] The claims of unchanged skill and improved spectra are not accompanied by any uncertainty quantification. Figure 5 shows fair CRPS curves that are visually overlapping, but the paper provides no confidence intervals, significance tests, or numerical differences (e.g., mean and standard error over the 365 initialisations). The statement 'the curves are on top of each other' is qualitative. Similarly, Figure 7 shows spectra averaged over 11 initial dates for a single ensemble member, with no spread across dates or ensemble members. Because the headline claim includes 'without negatively impacting forecast skill', it is important to demonstrate that the CRPS differences are within sampling uncertainty. The authors should add, at minimum, confidence intervals on the CRPS differences (e.g., by bootstrapping over initial dates) and measure the spread of the spectral ratios across initial dates.
  3. [Section 2.2.1 and Section 4 (kernel width and weights)] The multi-scale loss is evaluated only for a single choice of kernel width (Gaussian standard deviation equal to eight grid spacings) and equal scale weights (zeta_i = 1 for both scales). Section 4 acknowledges that 'more work will be required to assess what the best set of hyper-parameters is', but the abstract and conclusion state a general benefit of the multi-scale loss. Without a sensitivity analysis (e.g., varying the kernel width over at least a few values, and varying the relative weights), it is unclear whether the reported spectral improvement is robust or specific to this hand-chosen configuration. A minimal sensitivity study, even on a subset of the evaluation, would support the generality of the claim.
minor comments (6)
  1. [Section 2.1.1] The 1D simulation is a useful didactic illustration, but it shows only that the loss is wavenumber-dependent, not that optimizing the multi-scale loss leads to better forecasts in a realistic setting; consider stating this explicitly as motivation rather than an implied proof of the method's value.
  2. [Section 2.2.1] The text says the Gaussian kernel's standard deviation is 'eight times the grid spacing'; at N320 this corresponds to roughly 2 degrees. Stating the physical scale would help readers not familiar with the N320 grid.
  3. [Section 3] In the sentence 'On the other hand, The multi-scale experiment and the ERA5 analysis are in better agreement.', the 'The' after the comma should be lowercase.
  4. [Section 4] The text contains a typo: 'probabilist forecasts' should be 'probabilistic forecasts'.
  5. [Figure 7 caption] The spectra are averaged over 11 initial dates; please specify which dates and why 11 were chosen, as the number of days in 2019 is 365.
  6. [Equation (2)] For n=2, the scale definitions are scale 1 = D_1(phi) and scale 2 = phi - D_1(phi); this is used in the experiments but not explicitly written in the text, which may help readers map the notation to the implementation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the multi-scale loss is tested against held-out data with hand-set hyperparameters, so the claimed improvement is an empirical result rather than a reduction to the loss definition.

full rationale

The paper compares two training objectives, a scale-unaware afCRPS loss and a two-scale afCRPS loss, and evaluates the resulting forecast models on independent 2019 data using fair CRPS (Figure 5) and spectral power relative to ERA5 (Figure 7). The multi-scale loss is defined in Section 2.1 as a weighted sum of afCRPS scores on smoothed scale components, but the evaluation metrics are not the same functions as the loss terms; the evaluation uses standard scale-unaware CRPS and spatial spectra of full fields. The hyperparameters (Gaussian kernel standard deviation of eight grid spacings and equal scale weights) are chosen by hand (Section 2.2.1) and are not fitted to the evaluation metrics, so no fitted parameter is being renamed as a prediction. Self-citations to Lang et al. (2024b) for afCRPS and AIFS-CRPS provide the base model and score definition, but the central claim is about the incremental effect of adding a scale decomposition to the loss, which is tested empirically against an external reference (ERA5). The absence of a control with the same smoothing in a single-scale loss is a threat to attribution but not a circular step, because the claimed effect is not guaranteed by construction. No equation in the paper reduces the observed spectral improvement to the loss definition, and no load-bearing argument relies on an unverified self-citation. The derivation chain is therefore self-contained for the purpose of circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the scale-decomposition loss being an effective training objective; the key unstated premises are the validity of the smoothing-based partition, the propriety of the weighted per-scale score, and the adequacy of the chosen hyperparameters, none of which are tested in depth.

free parameters (4)
  • Smoothing kernel standard deviation = 8 grid spacings
    Chosen by hand in Section 2.2.1; no sensitivity analysis reported, and the scale decomposition depends on this width.
  • Scale weights zeta_i = 1 and 1 (equal weighting)
    Set to equal in Section 2.2.1; no tuning or justification beyond simplicity.
  • Number of scales in loss = 2
    Paper uses a two-scale version; authors note more scales may improve results but do not test them.
  • afCRPS mixing parameter alpha = 0.95
    Taken from prior work (Lang et al. 2024b), not varied here; affects ensemble degeneracy handling.
assumptions (4)
  • domain assumption The smoothing operators D_i define a valid partition of the field into scales such that per-scale scores are meaningful.
    Invoked in Section 2.1; the filters are Gaussian smoothers on the sphere, but the paper does not analyze spectral leakage or orthogonality of the decomposition.
  • standard math The afCRPS is a proper scoring rule, so a positive weighted sum of per-scale afCRPS terms is also proper and can be optimized.
    Section 2.2.1 refers to afCRPS properties from Lang et al. (2024b); no proof repeated here.
  • ad hoc to paper The fixed Gaussian kernel width (8 grid spacings) is adequate at N320 resolution for all variables, regions, and lead times.
    Section 2.2.1, not justified by experiments or theory.
  • domain assumption The initialized noise in AIFS-CRPS remains calibrated when the loss is split across scales.
    Section 2.2.2; the paper does not verify calibration diagnostics after the change, only CRPS and spectra.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A multi-scale loss formulation for learning a probabilistic model with proper score optimisation." pith.science (2026). https://pith.science/paper/TEZYOUS2

@misc{pith2026250610868,
  author       = {Pith},
  title        = {Pith review of: A multi-scale loss formulation for learning a probabilistic model with proper score optimisation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TEZYOUS2}},
  note         = {Machine review of arXiv:2506.10868}
}
read the original abstract

We assess the impact of a multi-scale loss formulation for training probabilistic machine-learned weather forecasting models. The multi-scale loss is tested in AIFS-CRPS, a machine-learned weather forecasting model developed at the European Centre for Medium-Range Weather Forecasts (ECMWF). AIFS-CRPS is trained by directly optimising the almost fair continuous ranked probability score (afCRPS). The multi-scale loss better constrains small scale variability without negatively impacting forecast skill. This opens up promising directions for future work in scale-aware model training.

Figures

Figures reproduced from arXiv: 2506.10868 by the authors.

Figure 1
Figure 1. Smoothing kernels for the 1-dim simulation study [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Predictions and targets are sine waves with random phases. The three [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Fair CRPS as function of the predicted wavenumber for the scale [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: ERA5 v-component of wind (in m s−1 ) at 850 hPa, full field (a), field after filtering (b) and difference between filtered and full field (c). 6 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Fair CRPS for (a) northern hemisphere geopotential at 500 hPa, (b) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Spectra of (a) geopotential at 500 hPa and (b) temperature at 850 hPa [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HourGlass: A probabilistic data-driven temporal downscaler for global and regional weather forecasting

    physics.ao-ph 2026-07 accept novelty 6.5 of 10

    HourGlass probabilistically reconstructs hourly weather evolution between 6-hourly forecast states using CRPS training on NWP trajectories, preserving skill and small-scale variability better than deterministic downscalers.

  2. CRPS-LAM: Probabilistic Regional Weather Forecasting with Continuous Ranked Probability Score

    cs.LG 2025-10 conditional novelty 5.0 of 10

    CRPS-LAM produces 57-hour probabilistic limited-area forecasts on MEPS at diffusion-comparable accuracy with single-forward-pass sampling, roughly 39x faster than Diffusion-LAM.

Reference graph

Works this paper leans on

17 extracted references · 5 canonical work pages · cited by 2 Pith papers

  1. [13]

    GenCast: Diffusion-based ensemble forecasting for medium-range weather.arXiv preprint arXiv:2312.15796,

    Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Timo Ewalds, Andrew El-Kadi, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, Remi Lam, and Matthew Willson. GenCast: Diffusion-based ensemble forecasting for medium-range weather.arXiv preprint arXiv:2312.15796,

    arXiv

  2. [14]

    doi: 10.1029/2023ms004177

    ISSN 1942-2466. doi: 10.1029/2023ms004177. URLhttp://dx.doi.org/10.1029/2023MS004177. 13 Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Gan- guli. Deep unsupervised learning using nonequilibrium thermodynamics,

    work page doi:10.1029/2023ms004177 1942

  3. [16]

    URLhttps://arxiv.org/abs/2011. 13456. N. P. Wedi. Increasing the horizontal resolution in numerical weather prediction and climate simulations: illusion or panacea?Philosophical Transactions of the Royal Society A, 372,

    work page 2011

  4. [2000]

    doi: 10.1175/1520-0434(2000)015⟨0559:DOTCRP⟩2.0. CO;2. URLhttps://journals.ametsoc.org/view/journals/wefo/15/5/ 1520-0434_2000_015_0559_dotcrp_2_0_co_2.xml. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.arXiv preprint arXiv:2006.11239,

    arXiv 2000

  5. [2007]

    Diffusion is spectral autoregression.https://sander

    11 Sander Dieleman. Diffusion is spectral autoregression.https://sander. ai/2024/09/02/spectral-autoregression.html, September

    work page 2024

  6. [2008]

    Elucidating the design space of diffusion-based generative models.arXiv preprint arXiv:2206.00364,

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models.arXiv preprint arXiv:2206.00364,

    arXiv

  7. [2013]

    doi: 10.1002/qj.2270

    ISSN 0035-9009. doi: 10.1002/qj.2270. URLhttp://dx.doi.org/10.1002/qj.2270. H. Hersbach, B. Bell, P. Berrisford, et al. The ERA5 global reanalysis.QJ R Meteorol Soc, 146:1999–2049,

    doi:10.1002/qj.2270 1999

  8. [2014]

    doi: 10.1098/rsta.2013.0289. 14

    arXiv 2013

Show all 17 references
  1. [2015]

    Yang Song, Jascha Sohl-Dickstein, Diederik P

    URLhttps://arxiv.org/abs/1503.03585. Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations,

    arXiv

  2. [2017]

    Franco Molteni, Roberto Buizza, Tim N Palmer, and Thomas Petroliagis

    URLhttps://doi.org/10.21957/H20RZ8. Franco Molteni, Roberto Buizza, Tim N Palmer, and Thomas Petroliagis. The ECMWF ensemble prediction system: Methodology and validation.Quarterly Journal of the Royal Meteorological Society, 122(529):73–119,

    doi

  3. [2019]

    URLhttps://rmets.onlinelibrary

    doi: https://doi.org/10.1002/qj.3387. URLhttps://rmets.onlinelibrary. wiley.com/doi/abs/10.1002/qj.3387. Martin Leutbecher and Tim N Palmer. Ensemble forecasting.Journal of Com- putational Physics, 227(7):3515–3539,

    doi

  4. [2020]

    Hans Hersbach

    doi: 10.1002/qj.3803. Hans Hersbach. Decomposition of the continuous ranked probability score for ensemble prediction systems.Weather and Forecasting, 15(5): 559 – 570,

    doi

  5. [2021]

    Erik Larsson, Joel Oskarsson, Tomas Landelius, and Fredrik Lindsten

    doi: 10.1002/qj.4181. Erik Larsson, Joel Oskarsson, Tomas Landelius, and Fredrik Lindsten. Diffusion- lam: Probabilistic limited area weather forecasting with diffusion,

    doi

  6. [2022]

    Brenner, and Stephan Hoyer

    Dmitrii Kochkov, Janni Yuval, Ian Langmore, Peter Norgaard, Jamie Smith, Griffin Mooers, Milan Kl¨ ower, James Lottes, Stephan Rasp, Peter D¨ uben, Sam Hatfield, Peter Battaglia, Alvaro Sanchez-Gonzalez, Matthew Willson, Michael P. Brenner, and Stephan Hoyer. Neural general ci...

    arXiv

  7. [2023]

    Simon Lang, Mihai Alexe, Matthew Chantry, Jesper Dramsch, Florian Pinault, Baudouin Raoult, Mariana C

    doi: 10.21957/slk503fs2i. Simon Lang, Mihai Alexe, Matthew Chantry, Jesper Dramsch, Florian Pinault, Baudouin Raoult, Mariana C. A. Clare, Christian Lessig, Michael Maier- Gerber, Linus Magnusson, Zied Ben Bouall` egue, Ana Prieto Nemesio, Pe- ter D. Dueben, Andrew Brown, Flor...

    arXiv

  8. [2024]

    Blog post

    URL https://sander.ai/2024/09/02/spectral-autoregression.html. Blog post. C. A. T. Ferro. Fair scores for ensemble forecasts.Quarterly Journal of the Royal Meteorological Society, 140(683):1917–1923, December

    2024

  9. [2025]

    Martin Leutbecher

    URL https://arxiv.org/abs/2502.07532. Martin Leutbecher. Ensemble size: How suboptimal is less than infinity?Quar- terly Journal of the Royal Meteorological Society, 145(S1):107–128,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.