Rigorous uncertainty quantification of probabilistic AI weather forecasts with conformal prediction
Pith reviewed 2026-06-26 18:18 UTC · model grok-4.3
The pith
Conformal prediction guarantees statistical coverage for AI weather forecast probabilities regardless of the underlying distribution.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
We employ conformal prediction, a class of statistical methods that mathematically guarantees coverage under no distributional assumptions, to post-process the probabilistic outputs of AI weather models, ensuring calibrated uncertainty at no expense to other probabilistic metrics.
What carries the argument
Online conformal prediction, which sequentially adjusts prediction sets to maintain coverage guarantees for non-exchangeable sequential data such as weather time series.
If this is right
- Extreme-event probabilities in AI forecasts achieve the user-specified coverage level by construction.
- Other scores such as continuous ranked probability score remain unaffected or improve.
- The same post-processing applies without modification to any new or existing probabilistic forecasting model.
- Decision makers gain access to uncertainty estimates whose reliability is mathematically assured rather than empirically assumed.
Where Pith is reading between the lines
- The method could expose calibration failures hidden by standard verification scores in current AI weather systems.
- Combining conformal post-processing with the larger ensemble sizes enabled by AI models might tighten intervals while preserving guarantees.
- Similar sequential conformal adjustments could transfer to other autoregressive forecasting domains such as energy demand or financial time series.
Load-bearing premise
The online conformal prediction procedure maintains its coverage guarantees when applied to the sequential, non-exchangeable time series of weather forecasts and observations.
What would settle it
A long out-of-sample period of weather observations where the empirical coverage rate of the conformalized forecasts falls below the nominal target level.
Figures
read the original abstract
Probabilistic weather forecasting is undergoing rapid transformation with artificial intelligence (AI). In traditional numerical weather prediction, computing power can limit how well ensemble forecasts approximate the unknown statistical distribution of future states. AI models facilitate larger ensembles and are trained with probabilistic considerations, ideally leading to better uncertainty quantification. Forecasts from these state-of-the-art models are often considered well-calibrated. However, here we show that the statistical coverage of such models, the ultimate measure of calibration, can struggle, especially on extreme events. To address this shortcoming, we employ conformal prediction, a class of statistical methods that mathematically guarantees coverage under no distributional assumptions, unlike previous post-processing techniques. We apply online conformal prediction to temperature and precipitation forecasts (including extremes) of three leading global weather models, GenCast, NeuralGCM, and AIFS-ENS, ensuring calibrated uncertainty at no expense to other probabilistic metrics. This post-processing method can be applied to any forecasting model.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript applies online conformal prediction as a post-processing wrapper to probabilistic forecasts from three AI weather models (GenCast, NeuralGCM, AIFS-ENS) for temperature and precipitation (including extremes). It claims that this yields mathematically guaranteed coverage under no distributional assumptions, unlike prior calibration methods, while preserving other probabilistic metrics such as sharpness and reliability.
Significance. If the finite-sample coverage guarantees can be shown to hold under the serial dependence, seasonality, and regime shifts present in weather time series, the work would supply a practical, model-agnostic calibration tool for the growing class of AI-based ensemble forecasts. The absence of any fitted parameters or distributional modeling is a methodological strength.
major comments (2)
- [Abstract] Abstract: The statement that conformal prediction 'mathematically guarantees coverage under no distributional assumptions' does not hold for the online/adaptive variants used on non-exchangeable sequential data. Standard results for ACI, EnbPI, or similar online CP procedures require additional conditions (e.g., bounded total variation of the conditional distributions or limited long-range dependence); weather temperature and precipitation series exhibit strong diurnal/seasonal autocorrelation and regime shifts that can cause realized coverage to deviate from the nominal level by an amount governed by the dependence strength. No verification or adaptation for these features is described.
- [Abstract] Abstract (and § on implementation, if present): The manuscript provides no quantitative coverage results, error bars, or explicit description of how the online procedure is initialized, updated, or tuned on the temporally ordered forecast–observation pairs. Without these, it is impossible to assess whether the claimed preservation of other metrics occurs at the cost of coverage or whether the dependence issue is mitigated in practice.
minor comments (1)
- [Abstract] The abstract would benefit from a concise statement of the exact online CP algorithm employed and the nominal coverage level targeted.
Simulated Author's Rebuttal
We thank the referee for their constructive comments on the scope of our theoretical claims and the need for clearer empirical details. We respond point-by-point to the major comments below.
read point-by-point responses
-
Referee: [Abstract] Abstract: The statement that conformal prediction 'mathematically guarantees coverage under no distributional assumptions' does not hold for the online/adaptive variants used on non-exchangeable sequential data. Standard results for ACI, EnbPI, or similar online CP procedures require additional conditions (e.g., bounded total variation of the conditional distributions or limited long-range dependence); weather temperature and precipitation series exhibit strong diurnal/seasonal autocorrelation and regime shifts that can cause realized coverage to deviate from the nominal level by an amount governed by the dependence strength. No verification or adaptation for these features is described.
Authors: We agree the abstract phrasing is too broad. Standard CP guarantees require exchangeability, while the online methods (ACI/EnbPI) used here yield coverage that holds under weak dependence or asymptotically. The manuscript demonstrates strong empirical coverage on weather series despite autocorrelation, but does not provide a formal verification of dependence conditions. We will revise the abstract to read 'provides distribution-free calibration with theoretical guarantees under mild dependence conditions' and add a short discussion of serial dependence in the methods section. revision: yes
-
Referee: [Abstract] Abstract (and § on implementation, if present): The manuscript provides no quantitative coverage results, error bars, or explicit description of how the online procedure is initialized, updated, or tuned on the temporally ordered forecast–observation pairs. Without these, it is impossible to assess whether the claimed preservation of other metrics occurs at the cost of coverage or whether the dependence issue is mitigated in practice.
Authors: The full manuscript contains these elements in Section 3 (initialization on a 365-day burn-in window, daily online updating of scores, and tuning of the adaptation parameter) and Section 4 (coverage plots with bootstrap error bars for raw vs. conformalized forecasts, including extremes). We will add one sentence to the abstract summarizing the empirical coverage result and ensure the methods section is explicitly cross-referenced from the abstract. revision: partial
Circularity Check
No circularity: standard CP wrapper applied to external model outputs
full rationale
The paper imports the finite-sample coverage guarantee of conformal prediction from the existing statistical literature and applies it as post-processing to the outputs of three external AI weather models (GenCast, NeuralGCM, AIFS-ENS). No new derivation, fitted parameter, or self-citation chain is presented that reduces the claimed coverage to a quantity constructed from the paper's own data or definitions. The method is described as a 'statistical wrapper' whose validity rests on the standard CP assumptions (or their online variants), which are external to this work.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Forecast-observation pairs satisfy the conditions (exchangeability or appropriate online variant) required for conformal prediction to deliver exact finite-sample coverage.
Reference graph
Works this paper leans on
-
[1]
Nature Communications , author =
A machine learning model that outperforms conventional global subseasonal forecast models , volume =. Nature Communications , author =. 2024 , pages =. doi:10.1038/s41467-024-50714-1 , language =
-
[2]
Journal of Hydrometeorology , author =
Why. Journal of Hydrometeorology , author =. 2014 , pages =. doi:10.1175/JHM-D-14-0008.1 , language =
-
[3]
Monthly Weather Review , author =
On the. Monthly Weather Review , author =. 2011 , pages =. doi:10.1175/2010MWR3446.1 , language =
-
[4]
Guilloteau, Clement and Kerrigan, Gavin and Nelson, Kai and Migliorini, Giosue and Smyth, Padhraic and Li, Runze and Foufoula-Georgiou, Efi , month = sep, year =. A. doi:10.48550/arXiv.2409.16319 , abstract =
-
[5]
URL https://doi.org/10.1038/s41586-024-08252-9
Probabilistic weather forecasting with machine learning , volume =. Nature , author =. 2025 , pages =. doi:10.1038/s41586-024-08252-9 , language =
-
[6]
Mahesh, Ankur and Collins, William D. and Bonev, Boris and Brenowitz, Noah and Cohen, Yair and Elms, Joshua and Harrington, Peter and Kashinath, Karthik and Kurth, Thorsten and North, Joshua and O'Brien, Travis and Pritchard, Michael and Pruitt, David and Risser, Mark and Subramanian, Shashank and Willard, Jared , month = sep, year =. Huge ensembles –. Ge...
-
[7]
Statistical Science , author =
Forecaster's. Statistical Science , author =. 2017 , note =
2017
-
[9]
Mahesh, Ankur and Collins, William and Bonev, Boris and Brenowitz, Noah and Cohen, Yair and Harrington, Peter and Kashinath, Karthik and Kurth, Thorsten and North, Joshua and OBrien, Travis and Pritchard, Michael and Pruitt, David and Risser, Mark and Subramanian, Shashank and Willard, Jared , month = apr, year =. Huge. doi:10.48550/arXiv.2408.01581 , publisher =
-
[13]
Gneiting, Tilmann and Ranjan, Roopesh , month = jul, year =. Comparing. Journal of Business & Economic Statistics , publisher =. doi:10.1198/jbes.2010.08110 , number =
-
[14]
Advances in Neural Information Processing Systems , volume=
Distribution-free binary classification: prediction sets, confidence intervals and calibration , author=. Advances in Neural Information Processing Systems , volume=
-
[15]
The Twelfth International Conference on Learning Representations , year=
Conformal Risk Control , author=. The Twelfth International Conference on Learning Representations , year=
-
[16]
Proceedings of the 55th Annual ACM Symposium on Theory of Computing , pages=
A unifying theory of distance from calibration , author=. Proceedings of the 55th Annual ACM Symposium on Theory of Computing , pages=
-
[17]
Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pages=
Multiaccuracy: Black-box post-processing for fairness in classification , author=. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pages=
2019
-
[18]
Advances in Neural Information Processing Systems , volume=
Calibrating predictions to decisions: A novel approach to multi-class calibration , author=. Advances in Neural Information Processing Systems , volume=
-
[19]
2024 , eprint=
Tail calibration of probabilistic forecasts , author=. 2024 , eprint=
2024
-
[20]
Predict to Minimize Swap Regret for All Payoff-Bounded Tasks , year=
Hu, Lunjia and Wu, Yifan , booktitle=. Predict to Minimize Swap Regret for All Payoff-Bounded Tasks , year=
-
[21]
The Thirty Sixth Annual Conference on Learning Theory , pages=
U-calibration: Forecasting for an unknown agent , author=. The Thirty Sixth Annual Conference on Learning Theory , pages=. 2023 , organization=
2023
-
[22]
International Conference on Machine Learning , pages=
Multicalibration: Calibration for the (computationally-identifiable) masses , author=. International Conference on Machine Learning , pages=. 2018 , organization=
2018
-
[23]
Deterministic calibration and
Kakade, Sham M and Foster, Dean P , journal=. Deterministic calibration and. 2008 , publisher=
2008
-
[24]
Journal of the American Statistical Association , volume=
The well-calibrated Bayesian , author=. Journal of the American Statistical Association , volume=. 1982 , publisher=
1982
-
[25]
International conference on machine learning , pages=
On calibration of modern neural networks , author=. International conference on machine learning , pages=. 2017 , organization=
2017
-
[26]
Advances in large margin classifiers , volume=
Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods , author=. Advances in large margin classifiers , volume=. 1999 , publisher=
1999
-
[27]
2019 , publisher=
High-dimensional statistics: A non-asymptotic viewpoint , author=. 2019 , publisher=
2019
-
[28]
Dostopno prek: https://www
Functions of bounded variation , author=. Dostopno prek: https://www. whitman. edu/Documents/Academics/Mathematics/gra dy. pdf (Dostopano: 7.2. 2017) , year=
2017
-
[29]
1975 , publisher=
Introductory real analysis , author=. 1975 , publisher=
1975
-
[30]
The annals of statistics , volume=
A general method for comparing probability assessors , author=. The annals of statistics , volume=. 1989 , publisher=
1989
-
[31]
Journal of the American statistical Association , volume=
Strictly proper scoring rules, prediction, and estimation , author=. Journal of the American statistical Association , volume=. 2007 , publisher=
2007
-
[32]
International Conference on Artificial Intelligence and Statistics , pages=
Integrating uncertainty awareness into conformalized quantile regression , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2024 , organization=
2024
-
[33]
Advances in neural information processing systems , volume=
Conformalized quantile regression , author=. Advances in neural information processing systems , volume=
-
[35]
arXiv preprint arXiv:1803.04765 , year=
Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning , author=. arXiv preprint arXiv:1803.04765 , year=
-
[36]
arXiv preprint arXiv:2506.10772 , year=
Skillful joint probabilistic weather forecasting from marginals , author=. arXiv preprint arXiv:2506.10772 , year=
-
[37]
Conformalized
Romano, Yaniv and Patterson, Evan and Candes, Emmanuel , year =. Conformalized. Advances in
-
[38]
Advances in Neural Information Processing Systems , author =
Conformal. Advances in Neural Information Processing Systems , author =. 2023 , pages =
2023
-
[39]
Adaptive Conformal Inference Under Distribution Shift , volume =
Gibbs, Isaac and Candes, Emmanuel , booktitle =. Adaptive Conformal Inference Under Distribution Shift , volume =
-
[40]
Almeida, Rodrigo and Otero, Noelia and Fernández-Torres, Miguel-Ángel and Ma, Jackie , month = nov, year =. On the. doi:10.48550/arXiv.2511.17176 , abstract =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2511.17176
-
[41]
Artificial Intelligence for the Earth Systems , author =
Uncertainty quantification for data-driven weather models , volume =. Artificial Intelligence for the Earth Systems , author =. 2025 , note =. doi:10.1175/AIES-D-24-0049.1 , abstract =
-
[42]
Mansfield, Laura A. and Christensen, Hannah M. , month = nov, year =. Epistemic and. doi:10.48550/arXiv.2511.23448 , abstract =
-
[43]
Collard, Julien and Gentine, Pierre and Zheng, Tian , month = nov, year =. Power. doi:10.48550/arXiv.2511.11170 , abstract =
-
[44]
Easy. SIAM Review , author =. 2024 , pages =. doi:10.1137/22M1541915 , number =
-
[45]
Calibrated. Monthly Weather Review , author =. 2005 , pages =. doi:10.1175/MWR2904.1 , language =
-
[46]
Deep learning for post-processing ensemble weather forecasts , volume =. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , author =. 2021 , pages =. doi:10.1098/rsta.2020.0092 , number =
-
[47]
Artificial Intelligence for the Earth Systems , author =
Postprocessing of. Artificial Intelligence for the Earth Systems , author =. doi:10.1175/AIES-D-23-0070.1 , language =
-
[48]
, editor =
Wilks, Daniel S. , editor =. Chapter 3 -. Statistical. 2018 , doi =
2018
-
[49]
Bulletin of the American Meteorological Society , author =
Statistical. Bulletin of the American Meteorological Society , author =. 2021 , pages =. doi:10.1175/BAMS-D-19-0308.1 , language =
-
[50]
Monthly Weather Review , author =
Neural. Monthly Weather Review , author =. 2018 , pages =. doi:10.1175/MWR-D-18-0187.1 , language =
-
[51]
Star formation across cosmic time , journal =
Renkema, Yvet and Visser, Lennard and AlSkaif, Tarek , month = jan, year =. Enhancing the reliability of probabilistic. doi:10.1016/j.seja.2024.100059 , journal =
-
[52]
Expert Systems with Applications , author =
Uncertainty forecasting system for tropical cyclone tracks based on conformal prediction , volume =. Expert Systems with Applications , author =. 2024 , pages =. doi:10.1016/j.eswa.2024.123743 , urldate =
-
[53]
Monthly Weather Review , author =
Calibrated. Monthly Weather Review , author =. 2016 , pages =. doi:10.1175/MWR-D-15-0260.1 , language =
-
[54]
Isotonic. Journal of the Royal Statistical Society Series B: Statistical Methodology , author =. 2021 , pages =. doi:10.1111/rssb.12450 , number =
-
[55]
Monthly Weather Review , author =
Ensemble. Monthly Weather Review , author =. 2020 , pages =. doi:10.1175/MWR-D-19-0227.1 , language =
-
[56]
Monthly Weather Review , author =
Using. Monthly Weather Review , author =. 2005 , note =. doi:10.1175/MWR2906.1 , abstract =
-
[57]
Evaluation of conformal-based probabilistic forecasting methods for short-term wind speed forecasting , language =
Althoff, Simon and Szabadváry, Johan Hallberg and Anderson, Jonathan and Carlsson, Lars , month = aug, year =. Evaluation of conformal-based probabilistic forecasting methods for short-term wind speed forecasting , language =. Proceedings of the
-
[58]
Bulletin of the American Meteorological Society , author =. 2023 , pages =. doi:10.1175/BAMS-D-21-0273.1 , language =
-
[59]
Artificial Intelligence for the Earth Systems , author =
Statistical. Artificial Intelligence for the Earth Systems , author =. doi:10.1175/AIES-D-25-0037.1 , language =
-
[60]
Artificial Intelligence for the Earth Systems , author =
Improving. Artificial Intelligence for the Earth Systems , author =. doi:10.1175/AIES-D-25-0045.1 , language =
-
[61]
Jacot, Arthur and Gabriel, Franck and Hongler, Clement , year =. Neural. Advances in
-
[62]
Tancik, Matthew and Srinivasan, Pratul and Mildenhall, Ben and Fridovich-Keil, Sara and Raghavan, Nithin and Singhal, Utkarsh and Ramamoorthi, Ravi and Barron, Jonathan and Ng, Ren , year =. Fourier. Advances in
-
[63]
Complete spectral data for analytic
Slipantschuk, J and Bandtlow, O F and Just, W , month = may, year =. Complete spectral data for analytic. Nonlinearity , publisher =. doi:10.1088/1361-6544/aa700f , abstract =
-
[64]
Communications in Nonlinear Science and Numerical Simulation , author =
Dynamic mode decomposition for analytic maps , volume =. Communications in Nonlinear Science and Numerical Simulation , author =. 2020 , keywords =. doi:10.1016/j.cnsns.2020.105179 , abstract =
-
[65]
Canatar, Abdulkadir and Bordelon, Blake and Pehlevan, Cengiz , month = may, year =. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks , volume =. Nature Communications , publisher =. doi:10.1038/s41467-021-23103-1 , abstract =
-
[66]
Spectrum
Bordelon, Blake and Canatar, Abdulkadir and Pehlevan, Cengiz , month = nov, year =. Spectrum. Proceedings of the 37th
-
[67]
Applied and Computational Harmonic Analysis , author =. 2024 , keywords =. doi:10.1016/j.acha.2024.101690 , abstract =
-
[68]
and Erichson, N
Yu, Annan and Lyu, Dongwei and Lim, Soon Hoe and Mahoney, Michael W. and Erichson, N. Benjamin , month = oct, year =. Tuning
-
[69]
Rahaman, Nasim and Baratin, Aristide and Arpit, Devansh and Draxler, Felix and Lin, Min and Hamprecht, Fred and Bengio, Yoshua and Courville, Aaron , month = may, year =. On the. Proceedings of the 36th
-
[70]
Communications in Computational Physics , author =
Frequency. Communications in Computational Physics , author =. 2020 , note =. doi:10.4208/cicp.OA-2020-0085 , abstract =
-
[71]
Chaos: An Interdisciplinary Journal of Nonlinear Science , author =
Practical rare event sampling for extreme mesoscale weather , volume =. Chaos: An Interdisciplinary Journal of Nonlinear Science , author =. 2019 , pages =. doi:10.1063/1.5081461 , abstract =
-
[72]
Using rare event algorithms to understand the statistics and dynamics of extreme heatwave seasons in
Le Priol, Clément and Monteiro, Joy Merwin and Bouchet, Freddy , month = oct, year =. Using rare event algorithms to understand the statistics and dynamics of extreme heatwave seasons in. Environmental Research: Climate , publisher =. doi:10.1088/2752-5295/ad8027 , abstract =
-
[73]
Science Advances , publisher =
Couairon, Guillaume and Singh, Renu and Charantonis, Anastase and Lessig, Christian and Monteleoni, Claire , month = apr, year =. Science Advances , publisher =. doi:10.1126/sciadv.adx2372 , number =
-
[74]
Journal of Advances in Modeling Earth Systems , author =
Rare. Journal of Advances in Modeling Earth Systems , author =. 2023 , note =. doi:10.1029/2022MS003537 , abstract =
-
[75]
Journal of Advances in Modeling Earth Systems , author =
Bringing. Journal of Advances in Modeling Earth Systems , author =. 2024 , note =. doi:10.1029/2024MS004264 , abstract =
-
[76]
Fischer, E. M. and Beyerle, U. and Bloin-Wibe, L. and Gessner, C. and Humphrey, V. and Lehner, F. and Pendergrass, A. G. and Sippel, S. and Zeder, J. and Knutti, R. , month = aug, year =. Storylines for unprecedented heatwaves based on ensemble boosting , volume =. Nature Communications , publisher =. doi:10.1038/s41467-023-40112-4 , abstract =
-
[77]
doi:10.48550/ARXIV.2412.15687 , shorttitle =
Alexe, Mihai and Boucher, Eulalie and Lean, Peter and Pinnington, Ewan and Laloyaux, Patrick and McNally, Anthony and Lang, Simon and Chantry, Matthew and Burrows, Chris and Chrust, Marcin and Pinault, Florian and Villeneuve, Ethel and Bormann, Niels and Healy, Sean , month = dec, year =. doi:10.48550/arXiv.2412.15687 , abstract =
-
[78]
Baño-Medina, Jorge and Sengupta, Agniv and Doyle, James D. and Reynolds, Carolyn A. and Watson-Parris, Duncan and Monache, Luca Delle , month = mar, year =. Are. npj Climate and Atmospheric Science , publisher =. doi:10.1038/s41612-025-00949-6 , language =
-
[79]
Nichani, Eshaan and Damian, Alex and Lee, Jason D. , month = aug, year =. How. doi:10.48550/arXiv.2402.14735 , abstract =
-
[80]
Lang, Simon and Alexe, Mihai and Clare, Mariana C. A. and Roberts, Christopher and Adewoyin, Rilwan and Ben Bouallègue, Zied and Chantry, Matthew and Dramsch, Jesper and Dueben, Peter D. and Hahner, Sara and Maciel, Pedro and Prieto-Nemesio, Ana and O’Brien, Cathal and Pinault, Florian and Polster, Jan and Raoult, Baudouin and Tietsche, Steffen and Leutbe...
-
[81]
Subich, Christopher and Husain, Syed Zahid and Separovic, Leo and Yang, Jing , month = jan, year =. Fixing the. doi:10.48550/arXiv.2501.19374 , abstract =
-
[82]
Computer Methods in Applied Mechanics and Engineering , author =
Can diffusion models capture extreme event statistics? , volume =. Computer Methods in Applied Mechanics and Engineering , author =. 2025 , keywords =. doi:10.1016/j.cma.2024.117589 , abstract =
-
[83]
URL https://www.nature.com/articles/ s41586-024-07744-y
Kochkov, Dmitrii and Yuval, Janni and Langmore, Ian and Norgaard, Peter and Smith, Jamie and Mooers, Griffin and Klöwer, Milan and Lottes, James and Rasp, Stephan and Düben, Peter and Hatfield, Sam and Battaglia, Peter and Sanchez-Gonzalez, Alvaro and Willson, Matthew and Brenner, Michael P. and Hoyer, Stephan , month = aug, year =. Neural general circula...
-
[84]
Geophysical Research Letters , author =
Explainable. Geophysical Research Letters , author =. 2024 , note =. doi:10.1029/2023GL106324 , abstract =
-
[85]
Bi, Kaifeng and Xie, Lingxi and Zhang, Hengheng and Chen, Xin and Gu, Xiaotao and Tian, Qi , month = jul, year =. Accurate medium-range global weather forecasting with. Nature , publisher =. doi:10.1038/s41586-023-06185-3 , language =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.