REVIEW 3 major objections 6 minor 3 cited by
Exploring Large Language Models for Climate Forecasting
T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read GPT-4o rainfall forecasts cling to 30-year averages in climate test.
desk verdict Useful cautionary negative result about GPT-4o rainfall forecasts, but the 'consistent conservatism' claim needs more than one test window. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the 30-year historical average as an implicit anchoring baseline, made visible by comparing GPT-4o's outputs under four prompt conditions against that baseline. The expert model is a two-layer LSTM, a recurrent neural network that learns temporal dependencies from 60-day or 60-month input windows, used to generate the expert rainfall and temperature predictions that GPT-4o was asked to incorporate in some experiments. Correlations with the 30-year average diagnose the smoothing tendency: the higher GPT's correlation with history, the more it damped the expert model's peaks. The standard-deviation experiment adds historical monthly rainfall variability as an uncertainty signal, and when GPT-4o receives it, its forecasts move closer to the expert model rather than to the average.
What would settle it
Run the same four prompts over a period containing a well-documented rainfall extreme, such as the 2015-2016 El Nino winter in the southern United States, with the model's timestamp set before that winter, and check whether GPT-4o's forecast deviations from the 30-year average grow large enough to track observed rainfall. The paper's claim predicts it will still damp the peaks; if instead the model responds to the extreme signal, the claim of consistent conservatism is false.
Extended reading notes
Core claim
The central claim is that GPT-4o, when asked to produce numerical rainfall forecasts, consistently chooses stable predictions close to historical averages regardless of what additional information it is given. In short-term tests, average RMSE stayed around 0.20-0.23, far above the LSTM expert model's 0.06; in long-term tests, feeding expert data raised error and lowered correlation instead of improving results. Correlations between GPT-4o's outputs and the 30-year average were 0.86 for GPT alone, 0.82 with expert rainfall, 0.76 with regional temperatures, and 0.62 with teleconnection indices, while the expert model's correlation with the average was only 0.59. The paper interprets this as evidence that GPT substitutes text-pattern common sense for physical climate reasoning, reverting to safe historical norms whenever no strong trend signal is visible.
Load-bearing premise
The load-bearing premise is that the single test window (October 1-15, 2023, and October 2023-September 2024) represents GPT-4o's general forecasting behavior; if that period is unusual or too short, the conclusion that GPT always reverts to historical averages does not follow.
Editorial extensions
If this is right
- If GPT-4o always reverts to historical averages, LLM-based public climate tools will understate extreme rainfall risk unless they are explicitly constrained or fine-tuned.
- Adding expert model outputs through prompts is not enough to change GPT-4o's behavior, so direct integration strategies need more than a prompt.
- Using historical variability as an uncertainty cue can move GPT-4o's forecasts closer to expert-model results, suggesting a cheap improvement path.
- Any practical LLM climate service should treat LLM outputs as baseline summaries, not event forecasts, until the averaging bias is overcome.
- The smoothing tendency is stronger at longer horizons and more visible in high-rainfall cities.
Reading between the lines
- A testable extension the paper leaves implicit: if the behavior is a general pretraining artifact rather than a prompt effect, other LLM families should show the same averaging bias under the same protocol.
- The paper's results imply a calibration strategy: measure how far an LLM's forecast sits from climatology and treat that distance as a trust score.
- Since the evidence covers one test window, a strong El Nino or hurricane-season repeat would clarify whether the conservative bias is absolute or limited to quiet periods.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper investigates whether GPT-4o can generate useful rainfall forecasts at short-term (15-day) and long-term (12-month) horizons for 15 U.S. cities. It compares four prompting conditions: GPT-only (Exp1), GPT with direct LSTM expert model rainfall predictions (Exp2), GPT with indirect temperature predictions (Exp3), and GPT with teleconnection indices (Exp4). The authors report that GPT-4o's outputs are closer to a 30-year historical average than to the expert model's outputs, and they conclude that GPT-4o consistently prioritizes stable, conservative predictions aligned with historical averages, even when provided with expert information. An additional experiment in the Discussion adds standard deviation as uncertainty information and shows improved agreement with the EM, but this experiment is not part of the core evaluation.
Significance. If the central finding were robust, it would provide a useful empirical characterization of a widely used LLM's behavior in a specialized forecasting domain: GPT-4o tends to smooth toward climatology rather than following expert-model signals. The paper's strengths include its multi-city design, the use of publicly available data, a transparent LSTM baseline, and an honest acknowledgment of the look-ahead nature of the teleconnection experiment. The four-way experimental design is a sensible framework for probing how LLMs respond to different types of auxiliary information. However, the evidence base is a single forecast window with no repeated sampling or uncertainty quantification, so the generalizability of the main claim is not yet established; the paper reads as an exploratory case study rather than a definitive evaluation.
major comments (3)
- [Sections III and IV] The central claim of consistent conservative behavior is based on exactly one test period: October 1–15, 2023 for short-term and October 2023–September 2024 for long-term forecasts, with a single set of GPT-4o outputs per condition per city. GPT-4o is stochastic; no repeated sampling, confidence intervals, or significance tests accompany the reported differences (e.g., short-term RMSE 0.23 vs. 0.20; long-term correlations with the 30-year average of 0.86, 0.82, 0.76, and 0.62 vs. the EM's 0.59). The observed closeness to the 30-year average could be an artifact of this particular El Niño-adjacent period, the specific prompts, or one draw from the model. Because the Conclusion states that GPT "consistently" prioritizes stable predictions, the paper must evaluate multiple independent forecast periods and multiple stochastic draws before that generalization can be supported.
- [Section III, Experiment 4] The teleconnection experiment provides GPT-4o with actual values of Nino3.4, PDO, and NAO for the forecast period itself (October 2023–September 2024). These values would not be known to a forecaster operating at the stated timestamp of September 30, 2023, so Exp4 is an oracle-input condition, not a realistic forecast condition. The paper acknowledges this in a note, but the subsequent interpretation treats Exp4 as evidence about how GPT uses teleconnection information in forecasting (e.g., "adding global teleconnection factors, GPT's results declined"). This framing conflates a data-leakage condition with a legitimate input scenario and undermines the comparative claims. The experiment should be either reframed as an oracle study or run with predicted/forecasted teleconnection indices.
- [Section IV, correlations with 30-year average] The claim that GPT-generated predictions "closely align with the 30-year average" is supported primarily by Pearson correlations between the predictions and the 30-year average (values of 0.86, 0.82, 0.76, and 0.62 for Exp1–Exp4, and 0.59 for the EM). A high correlation with a smooth climatological seasonal cycle may simply reflect the strong annual periodicity in rainfall, rather than a deliberate conservative strategy. To substantiate the interpretation that GPT is reverting to historical averages, the paper should also report direct distances (e.g., RMSE, MAE) between each method's outputs and the 30-year baseline, and ideally a skill score such as Nash-Sutcliffe efficiency computed against that baseline. Without such metrics, the "alignment" claim is not quantitatively established.
minor comments (6)
- [Abstract and Introduction] The paper inconsistently refers to the model as "ChatGPT-4" in some places and "GPT-4o" in others, including within the abstract text of the manuscript; choose one name and use it throughout.
- [Section III, Experiment 4 prompt] The prompt sample for Experiment 4 says the prediction period is "October 1, 2023, to October 15, 2023," but the experiment is described as a 12-month forecast; correct the period in the prompt to match the intended monthly-scale evaluation.
- [Figure 4 caption] The caption for Figure 4 appears to be a leftover from another manuscript ("Comparison of predicted results and observation for different token setup and different lead time...") and does not describe the short-term time series comparisons shown; replace it with a proper caption.
- [Section II, evaluation metrics] The paper lists Nash-Sutcliffe efficiency as an evaluation metric but no NSE values are reported in the text or figures; either report NSE results in tables or remove the mention to avoid an unmet expectation.
- [Section II, baseline definition] The "30-year historical average" baseline is not precisely defined; specify the exact period (e.g., 1993–2022) used to compute the daily and monthly climatological means.
- [References] Reference [10] has an incomplete DOI string with a trailing period embedded in the text; provide the full DOI for the dataset citation.
Circularity Check
No significant circularity: GPT-4o outputs are generated independently and compared against a 30-year baseline, not derived from it.
full rationale
The paper's central claim is an empirical characterization of GPT-4o's forecast outputs under several prompting conditions. The 30-year historical average is used only as a comparison baseline (Section III: 'we used the average daily/monthly rainfall over the past 30 years as a baseline, against which we compared GPT's predictions'), and no equation or prompt defines GPT's output as that average. The LSTM expert model is trained on 1900-2022 data and evaluated on October 2023-September 2024, so its predictions are out-of-sample rather than fitted to the test targets. The correlation coefficients with the 30-year average (0.86, 0.82, 0.76, 0.62) are post-hoc statistics on GPT outputs, not construction constraints. There is likewise no load-bearing self-citation: the cited references do not include the present authors' prior work and are used for data sources and context, not to justify the conclusion. The use of actual teleconnection indices as inputs in Experiment 4 is a data-leakage/validity concern, not a circularity of derivation. The single test window and single stochastic draw weaken generalizability, but that is an evidential limitation rather than a circular reduction.
Assumptions & free parameters
free parameters (5)
- LSTM hidden size =
128
- LSTM input window =
60 time steps
- LSTM output window =
15 or 12 time steps
- Training/validation split =
80/20
- 30-year historical average baseline =
30 years
assumptions (4)
- domain assumption Historical daily temperature and precipitation data from 1900 to 2022 are sufficient to train an LSTM that produces reliable future rainfall predictions.
- domain assumption The selected 15 cities and the single evaluation period (October 2023 to September 2024) are representative of GPT-4o's forecasting behavior in general.
- domain assumption GPT-4o's output is deterministic and stable across runs; no sampling temperature, random seed, or run-to-run variability is considered.
- domain assumption The prompt templates used in the experiments fairly and effectively elicit GPT-4o's forecasting ability.
Cite this review
Pith. "Pith review of Exploring Large Language Models for Climate Forecasting." pith.science (2026). https://pith.science/paper/HANDISQI
@misc{pith2026241113724,
author = {Pith},
title = {Pith review of: Exploring Large Language Models for Climate Forecasting},
year = {2026},
howpublished = {\url{https://pith.science/paper/HANDISQI}},
note = {Machine review of arXiv:2411.13724}
}
read the original abstract
With the increasing impacts of climate change, there is a growing demand for accessible tools that can provide reliable future climate information to support planning, finance, and other decision-making applications. Large language models (LLMs), such as GPT-4, present a promising approach to bridging the gap between complex climate data and the general public, offering a way for non-specialist users to obtain essential climate insights through natural language interaction. However, an essential challenge remains under-explored: evaluating the ability of LLMs to provide accurate and reliable future climate predictions, which is crucial for applications that rely on anticipating climate trends. In this study, we investigate the capability of GPT-4 in predicting rainfall at short-term (15-day) and long-term (12-month) scales. We designed a series of experiments to assess GPT's performance under different conditions, including scenarios with and without expert data inputs. Our results indicate that GPT, when operating independently, tends to generate conservative forecasts, often reverting to historical averages in the absence of clear trend signals. This study highlights both the potential and challenges of applying LLMs for future climate predictions, providing insights into their integration with climate-related applications and suggesting directions for enhancing their predictive capabilities in the field.
Figures
Forward citations
Cited by 3 Pith papers
-
Dynamic and Generalizable Process Reward Modeling
DG-PRM stores multi-dimensional process reward criteria in a hierarchical tree, selects them dynamically per step, and uses Pareto dominance to build preference pairs, claiming state-of-the-art process reward modeling.
-
Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts
A position paper advocating large-scale training of event forecasting LLMs, with proposals for label selection, counterfactual training data, auxiliary rewards, and multi-source datasets.
-
ClimaEmpact: Domain-Aligned Small Language Models and Datasets for Extreme Weather Analytics
The paper introduces EWRA, a two-stage reasoning-aware fine-tuning method for small language models, together with the ExtremeWeatherNews dataset of over 127,000 extreme-weather news sentences, and evaluates both on t...
Reference graph
Works this paper leans on
-
[1]
Weather forecasting for weather derivatives,
S. D. Campbell and F. X. Diebold, “Weather forecasting for weather derivatives,” J Am Stat Assoc, vol. 100, no. 469, pp. 6–16, Mar. 2005, doi: 10.1198/016214504000001051
-
[4]
Comparison of predicted results and observation for different token setup and different lead time. A: whole test period; B: JFM prediction with lead time 6; C: AMJ prediction with lead time 14 V. DISCUSSION AND FUTURE RESEARCH As a large language model, GPT is fundamentally trained to learn language patterns from vast amounts of textual data, rather than ...
work page 2023
-
[5]
II. DATA AND METHOD In this study, we focused on rainfall prediction at two different time scales: short-term (15-day scale) and long-term (12-month scale). We considered 15 cities across the contiguous United States: ‘Washington DC’ ‘Tucson, AZ’ ‘Salt Lake City, UT’ ‘Reno, NV’ ‘Phoenix, AZ’ ‘Pensacola, FL’ ‘New York, NY’ ‘Mobile, AL’ ‘Forks, WA’ ‘El Paso...
-
[7]
Local climate services for all, courtesy of large language models,
N. Koldunov and T. Jung, “Local climate services for all, courtesy of large language models,” Dec. 01, 2024, Nature Publishing Group. doi: 10.1038/s43247-023-01199-1
-
[8]
Thus spoke GPT-3: Interviewing a large-language model on climate finance,
M. Leippold, “Thus spoke GPT-3: Interviewing a large-language model on climate finance,” Financ Res Lett, vol. 53, May 2023, doi: 10.1016/j.frl.2022.103617
-
[10]
• https://www.weather.gov/wrh/climate?wfo=sew. Figure 1 The locations of selected cities in the United States and their corresponding annual rainfall amounts. For rainfall prediction, historical daily maximum temperature, minimum temperature, and precipitation were used as inputs. For temperature prediction, only minimum and maximum temperature were used ...
work page 1900
-
[11]
Use of Historical Data to Assess Regional Climate Change
Y. Lai and D. A. Dzombak, “Use of Historical Data to Assess Regional Climate Change”, doi: 10.1175/JCLI-D-18. 1
-
[12]
M. Zhang, J. D. Rojo-Hernández, L. Yan, Ó. J. Mesa, and U. Lall, “Hidden Tropical Pacific Sea Surface Temperature States Reveal Global Predictability for Monthly Precipitation for Sub-Season to Annual Scales,” Geophys Res Lett, vol. 49, no. 20, pp. 1–9, 2022, doi: 10.1029/2022GL099572
Show all 11 references
-
[2012]
doi: 10.1038/nclimate1745
-
[2023]
For long-term predictions, we used the 60 months prior to October 2023 to forecast monthly rainfall from October 2023 to September
2023
-
[2024]
This setup allowed us to evaluate the EM model’s performance on both short-term and long-term rainfall forecasting tasks. III. EXPERIMENTAL SETUP In this study, to systematically evaluate the performance of LLMs in climate forecasting tasks and to analyze their ability to gene...
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.