REVIEW 4 major objections 5 minor 27 references
Comparative Analysis of Zero-Shot Capability of Time-Series Foundation Models in Short-Term Load Prediction
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Untrained time-series foundation models can outperform trained classical predictors in short-term load forecasting.
desk verdict Useful benchmark on load forecasting, but the zero-shot claim needs a leakage check and the novelty claim is overblown. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the pretrained Transformer architecture of time-series foundation models operating in zero-shot mode. Chronos tokenizes each real-valued load observation into a discrete token via a quantization function, treats the token sequence like text, and samples future tokens from the model's predicted token distribution; detokenization maps those tokens back to load values. Because sampled tokens come from the set of historical tokens, predictions stay within the observed range, which avoids the negative-load problem seen in the Gaussian process and in Lag-llama, whose distributional assumptions allow negative draws. The comparison apparatus is a fixed 60/40 train-test split with a three-day context window and next-day forecasts, evaluated by mean absolute error, root mean squared error, and quantile losses at the 10%, 50%, and 90% levels.
What would settle it
Re-run the comparison over several random or rolling train/test splits and context windows while checking whether the foundation models' pretraining data included these load series; if trained GP or SVR wins on most splits once leakage is ruled out, the claimed zero-shot advantage does not generalize beyond the paper's single split.
Extended reading notes
Core claim
The central discovery the authors claim is that zero-shot time-series foundation models can match or surpass classically trained models in short-term load prediction. In their experiments, Chronos-small and Chronos-tiny achieve lower MAE, RMSE, and quantile losses than the trained Gaussian process and support vector regression baselines across almost all tested aggregate and individual load datasets, with GP only outperforming Chronos in one case. They also observe that the Gaussian process frequently predicts negative loads because of its multivariate Gaussian assumption, whereas Chronos, whose predictions are sampled from its token vocabulary built from historical values, produces non-negative forecasts and captures peaks better. The paper concludes that pretrained TSFMs require little or no additional data for a new forecasting task, making them a promising route for short-term load prediction in data-scarce settings.
Load-bearing premise
The comparison is a fair zero-shot test only if the foundation models never saw these UK, German, and Dutch load datasets during their earlier training, and the single fixed 60/40 split with a three-day look-back is representative of the prediction task.
Editorial extensions
If this is right
- Short-term load prediction at a new site would no longer require collecting and training on site-specific load data, removing a major barrier in data-scarce distribution networks.
- Chronos-style tokenization enforces non-negative forecasts by sampling from historical tokens, avoiding the physically impossible negative loads produced by GP and Lag-llama.
- The differing results across models show that the zero-shot advantage is not automatic; model architecture and pretraining matter, so users should select among TSFMs.
- Because current TSFMs cannot ingest weather or other external covariates, integrating those inputs is a stated next step for improving forecast accuracy.
- When probabilistic forecasts are not needed, TimeGPT and TimesFM offer competitive point-forecast alternatives to Chronos.
Reading between the lines
- The paper's fixed setup leaves open how sensitive the ranking is to the choice of split and context window; an extension would rerun the comparison over multiple random or rolling splits, context windows of one to seven days, and several forecast horizons, with error bars.
- A stronger test of the zero-shot claim would audit the pretraining corpora or use temporally held-out data that postdates model training, especially for the proprietary TimeGPT, to rule out data leakage.
- The tokenization insight plausibly extends beyond load: any bounded or non-negative time series such as prices, inventory, or water demand should favor token-sampling forecasters over distribution-parameter forecasters when physical range violations matter.
- If the zero-shot advantage proves stable across sites, these models could serve as cross-site priors for transfer learning or as fallback forecasters when live telemetry is missing before a site-specific model is fit.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates the zero-shot forecasting performance of five time-series foundation models (Chronos, Moment, Lag-llama, TimesFM, TimeGPT) against two classical trained models (Gaussian Process and Support Vector Regression) on short-term load prediction using UK, German, and Dutch datasets at multiple temporal resolutions. The authors report MAE, RMSE, and quantile losses, and conclude that several TSFMs, particularly Chronos, can outperform the trained classical baselines without task-specific training. The paper includes a public code/data repository and discusses qualitative prediction examples.
Significance. If the central claim holds, the paper would provide useful evidence that zero-shot TSFMs are a viable alternative to trained classical models for short-term load prediction, which is relevant for data-scarce or privacy-constrained settings. The study covers a range of TSFMs and classical baselines, uses multiple datasets and resolutions, and reports probabilistic metrics in addition to point errors. These are strengths. However, the significance is tempered by unresolved questions about pretraining contamination, lack of uncertainty quantification, and incomplete specification of the experimental protocol, all of which affect the interpretation of the comparison as a genuine zero-shot generalization test.
major comments (4)
- [Section III-A and Section II] The zero-shot claim relies on the assumption that the test datasets (UK, German, Dutch load data) were not part of the TSFM pretraining corpora. This is not verified. Chronos [20] and TimesFM [23] are pretrained on large public time-series collections that include electricity data, and TimeGPT [24] has undisclosed training data. The paper should either perform a leakage/contamination analysis (e.g., checking exact or near-duplicate series overlap between the test sets and the pretraining corpora) or report the specific model versions and knowledge cutoffs for TimeGPT to make the check possible. Without this, the phrase 'without training' may not reflect genuine generalization.
- [Section III-B and Tables III-IV] The evaluation uses a single fixed 60/40 train-test split and a single three-day context window, with no repeated trials, resampling, or error bars. All tables report single point values for MAE, RMSE, and quantile losses. This makes it impossible to assess whether observed differences are statistically significant or robust to the choice of split and context length. The authors should report standard deviations or confidence intervals across multiple random splits, context lengths, or bootstrap resamples.
- [Section III-B] The hyperparameter choices for GP and SVR are not described. For GP, the kernel type, length-scale initialization/bounds, and noise treatment are omitted; for SVR, the kernel, regularization parameter C, and epsilon are not given. Also, the protocol for using the 60% training set for rolling test forecasts is unspecified: it is unclear whether the models are retrained periodically, how the three-day context is incorporated for GP/SVR, and how time alignment between training and test periods is handled. Without this information, the classical baselines may be unintentionally underconfigured, making the comparison less fair.
- [Section IV and Abstract] There are internal inconsistencies in the reported claims. Section IV states that 'GP does not outperform Chronos in any experiment except NL-I-60,' but Tables III and IV show that Chronos-small achieves lower MAE than GP on NL-I-60 (0.1420 vs 0.2044) and on every other dataset; there is no experiment where GP has a lower MAE. Similarly, the abstract and conclusion state that TimeGPT can surpass GP and SVR, but in Table III (GE-A-15 and GE-A-30) TimeGPT has higher MAE and RMSE than both GP and SVR. The claims should be qualified to specific models and datasets, or the abstract/conclusion should be revised to reflect the observed variability.
minor comments (5)
- [Table II] The table caption and entries contain typos: 'TimeFM' should be 'TimesFM', and the caption formatting is inconsistent. Also, 'Probabilistic' should be aligned with a clear yes/no level.
- [Section IV] The text refers to 'TimesGPT' instead of 'TimeGPT' in the point-prediction discussion, and 'IN-I-60' in the Figure 4 discussion should likely be 'NL-I-60'.
- [Table III and Table IV] The captions read 'EXPERIENTIAL RESULTS' and should be 'EXPERIMENTAL RESULTS'.
- [Section IV] The explanation that 'Chronos's prediction is essentially randomly sampled tokens from the historical tokens' is imprecise: Chronos samples from a learned token distribution, not directly from the historical observed tokens. This should be reworded to avoid misunderstanding.
- [Section I] The claim 'To our knowledge, this study represents the first exploration of TSFMs in the context of load prediction' appears too strong given the existing literature on time-series foundation models for energy forecasting; the authors should expand the related-work discussion to verify novelty.
Circularity Check
No significant circularity: this is an empirical benchmark comparing independently pre-trained TSFMs with trained GP/SVR on a fixed split; no fitted quantity is renamed as a prediction.
full rationale
This paper is an empirical benchmark rather than a derivation, so the circularity patterns do not apply. The central claim—that zero-shot TSFMs can outperform trained GP and SVR—is supported by the direct experiment in Tables III and IV. TSFMs are not trained on the task-specific split (60/40), while GP and SVR are trained on the 60% training portion; the 'prediction' is the TSFM's actual output, not a parameter refit to the test data. No equation in the paper defines a fitted parameter in terms of the target quantity and then presents it as a prediction. The only mild self-citation is reference [17] (Xia, Palensky, Vergara) in the literature review on flow-based probabilistic STLP; it is not load-bearing for the zero-shot comparison and is not used to justify any modeling choice in this paper. The more serious validity concern—whether the TSFMs' pretraining corpora, especially TimeGPT's undisclosed data, already contained the UK/German/Dutch test load series—is a data-leakage or generalization risk, not circular reasoning: even if leakage occurred, the benchmark would measure memorization rather than zero-shot generalization, but the paper's logic would not be self-referential. Similarly, the single 60/40 split with no repeated trials weakens robustness but does not make any result true by construction. Under the requested definition, no circular step can be exhibited with a quote and a specific reduction, so the appropriate score is 0.
Assumptions & free parameters
free parameters (2)
- context_length =
72, 144, or 288 samples (three days at 60, 30, or 15 minute resolution)
- train_test_split_ratio =
60/40
assumptions (3)
- domain assumption The TSFM pretraining corpora did not contain the evaluation datasets.
- domain assumption The GP and SVR implementations are representative, reasonably configured classical baselines.
- domain assumption The fixed context window and train-test split are representative of real STLP settings.
Cite this review
Pith. "Pith review of Comparative Analysis of Zero-Shot Capability of Time-Series Foundation Models in Short-Term Load Prediction." pith.science (2026). https://pith.science/paper/SI6DYZRH
@misc{pith2026241212834,
author = {Pith},
title = {Pith review of: Comparative Analysis of Zero-Shot Capability of Time-Series Foundation Models in Short-Term Load Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/SI6DYZRH}},
note = {Machine review of arXiv:2412.12834}
}
read the original abstract
Short-term load prediction (STLP) is critical for modern power distribution system operations, particularly as demand and generation uncertainties grow with the integration of low-carbon technologies, such as electric vehicles and photovoltaics. In this study, we evaluate the zero-shot prediction capabilities of five Time-Series Foundation Models (TSFMs)-a new approach for STLP where models perform predictions without task-specific training-against two classical models, Gaussian Process (GP) and Support Vector Regression (SVR), which are trained on task-specific datasets. Our findings indicate that even without training, TSFMs like Chronos, TimesFM, and TimeGPT can surpass the performance of GP and SVR. This finding highlights the potential of TSFMs in STLP.
Figures
Reference graph
Works this paper leans on
-
[20]
Chronos: Learning the language of time series,
A. F. Ansari, L. Stella, C. Turkmen, X. Zhang, P. Mercado, H. Shen, O. Shchur, S. S. Rangapuram, S. P. Arango, S. Kapoor et al., “Chronos: Learning the language of time series,” arXiv preprint arXiv:2403.07815, 2024
arXiv 2024
-
[23]
A decoder-only foundation model for time-series forecasting,
A. Das, W. Kong, R. Sen, and Y . Zhou, “A decoder-only foundation model for time-series forecasting,” arXiv preprint arXiv:2310.10688 , 2023
arXiv 2023
-
[24]
A. Garza, C. Challu, and M. Mergenthaler-Canseco, “Timegpt-1,” arXiv preprint arXiv:2310.03589, 2023
arXiv 2023
-
[1]
A review of machine learning in building load prediction,
L. Zhang, J. Wen, Y . Li, J. Chen, Y . Ye, Y . Fu, and W. Livingood, “A review of machine learning in building load prediction,” Applied Energy, vol. 285, p. 116452, 2021
work page 2021
-
[2]
Research on short-term load prediction based on seq2seq model,
G. Gong, X. An, N. K. Mahato, S. Sun, S. Chen, and Y . Wen, “Research on short-term load prediction based on seq2seq model,” Energies, vol. 12, no. 16, p. 3199, 2019
work page 2019
-
[3]
Load demand forecasting of residen- tial buildings using a deep learning model,
L. Wen, K. Zhou, and S. Yang, “Load demand forecasting of residen- tial buildings using a deep learning model,” Electric Power Systems Research, vol. 179, p. 106073, 2020
work page 2020
-
[4]
Real-time very short-term load prediction for power-system automatic generation control,
D. J. Trudnowski, W. L. McReynolds, and J. M. Johnson, “Real-time very short-term load prediction for power-system automatic generation control,” IEEE Trans. Control Systems Technology , vol. 9, no. 2, pp. 254–260, 2001
work page 2001
-
[5]
S.-J. Huang and K.-R. Shih, “Short-term load forecasting via arma model identification including non-gaussian process considerations,” IEEE Trans. power systems , vol. 18, no. 2, pp. 673–679, 2003
work page 2003
Show all 27 references
-
[6]
Short-term load forecasting of industrial customers based on svmd and xgboost,
Y . Wang, S. Sun, X. Chen, X. Zeng, Y . Kong, J. Chen, Y . Guo, and T. Wang, “Short-term load forecasting of industrial customers based on svmd and xgboost,” Int. J of Electrical Power & Energy Systems , vol. 129, p. 106830, 2021
2021
-
[7]
Short-term electric load forecasting based on singular spectrum analysis and support vector machine opti- mized by cuckoo search algorithm,
X. Zhang, J. Wang, and K. Zhang, “Short-term electric load forecasting based on singular spectrum analysis and support vector machine opti- mized by cuckoo search algorithm,” Electric Power Systems Research , vol. 146, pp. 270–285, 2017
2017
-
[8]
A regional hybrid goa- svm model based on similar day approach for short-term load forecasting in assam, india,
M. Barman, N. D. Choudhury, and S. Sutradhar, “A regional hybrid goa- svm model based on similar day approach for short-term load forecasting in assam, india,” Energy, vol. 145, pp. 710–720, 2018
2018
-
[9]
Stacking ensemble learning for short-term electricity consumption forecasting,
F. Divina, A. Gilson, F. Gom ´ez-Vela, M. Garc´ıa Torres, and J. F. Torres, “Stacking ensemble learning for short-term electricity consumption forecasting,” Energies, vol. 11, no. 4, p. 949, 2018
2018
-
[10]
A short- term and high-resolution distribution system load forecasting approach using support vector regression with hybrid parameters optimization,
H. Jiang, Y . Zhang, E. Muljadi, J. J. Zhang, and D. W. Gao, “A short- term and high-resolution distribution system load forecasting approach using support vector regression with hybrid parameters optimization,” IEEE Trans. Smart Grid , vol. 9, no. 4, pp. 3341–3350, 2016
2016
-
[11]
Robust deep gaussian process-based probabilistic electrical load fore- casting against anomalous events,
D. Cao, J. Zhao, W. Hu, Y . Zhang, Q. Liao, Z. Chen, and F. Blaabjerg, “Robust deep gaussian process-based probabilistic electrical load fore- casting against anomalous events,” IEEE Trans. Industrial Informatics , vol. 18, no. 2, pp. 1142–1153, 2021
2021
-
[12]
Gaussian process-aided transfer learning for probabilistic load forecasting against anomalous events,
P. Zhao, D. Cao, Y . Wang, Z. Chen, and W. Hu, “Gaussian process-aided transfer learning for probabilistic load forecasting against anomalous events,” IEEE Trans. power systems , vol. 38, no. 3, pp. 2962–2965, 2023
2023
-
[13]
Spatial-temporal residential short-term load forecasting via graph neural networks,
W. Lin, D. Wu, and B. Boulet, “Spatial-temporal residential short-term load forecasting via graph neural networks,” IEEE Trans. Smart Grid , vol. 12, no. 6, pp. 5373–5384, 2021
2021
-
[14]
Short- term load forecasting by using a combined method of convolutional neural networks and fuzzy time series,
H. J. Sadaei, P. C. d. L. e Silva, F. G. Guimaraes, and M. H. Lee, “Short- term load forecasting by using a combined method of convolutional neural networks and fuzzy time series,” Energy, vol. 175, pp. 365–377, 2019
2019
-
[15]
A deep model for short- term load forecasting applying a stacked autoencoder based on lstm supported by a multi-stage attention mechanism,
Z. Fazlipour, E. Mashhour, and M. Joorabian, “A deep model for short- term load forecasting applying a stacked autoencoder based on lstm supported by a multi-stage attention mechanism,” Applied Energy , vol. 327, p. 120063, 2022
2022
-
[16]
Modeling load forecast un- certainty using generative adversarial networks,
Y . Wang, G. Hug, Z. Liu, and N. Zhang, “Modeling load forecast un- certainty using generative adversarial networks,” Electric Power Systems Research, vol. 189, p. 106732, 2020
2020
-
[17]
A flow-based model for conditional and probabilistic electricity consumption profile generation and prediction,
W. Xia, C. Wang, P. Palensky, and P. P. Vergara, “A flow-based model for conditional and probabilistic electricity consumption profile generation and prediction,” arXiv preprint arXiv:2405.02180 , 2024
2024
-
[18]
Resi- dential electricity load scenario prediction based on transferable flow generation model,
L. Lin, C. Chen, B. Wei, H. Li, J. Shi, J. Zhang, and N. Huang, “Resi- dential electricity load scenario prediction based on transferable flow generation model,” Journal of Electrical Engineering & Technology , vol. 18, no. 1, pp. 99–109, 2023
2023
-
[19]
A compre- hensive survey of few-shot learning: Evolution, applications, challenges, and opportunities,
Y . Song, T. Wang, P. Cai, S. K. Mondal, and J. P. Sahoo, “A compre- hensive survey of few-shot learning: Evolution, applications, challenges, and opportunities,” ACM Computing Surveys, vol. 55, no. 13s, pp. 1–40, 2023
2023
-
[21]
Moment: A family of open time-series foundation models,
M. Goswami, K. Szafer, A. Choudhry, Y . Cai, S. Li, and A. Dubrawski, “Moment: A family of open time-series foundation models,” arXiv preprint arXiv:2402.03885, 2024
2024 arXiv
-
[22]
Lag-llama: Towards foundation models for probabilistic time series forecasting,
K. Rasul, A. Ashok, A. R. Williams, H. Ghonia, R. Bhagwatkar, A. Khorasani, M. J. D. Bayazi, G. Adamopoulos, R. Riachi, N. Hassen et al. , “Lag-llama: Towards foundation models for probabilistic time series forecasting,” Preprint, 2024
2024
-
[25]
Smart meter data - london households,
“Smart meter data - london households,” https://data.london.gov.uk/ dataset/smartmeter-energy-use-data-in-london-households
-
[26]
Household data - open power system data,
“Household data - open power system data,” https://data. open-power-system-data.org/household data/2020-04-15
2020
-
[27]
Liander open data,
“Liander open data,” https://www.liander.nl/partners/datadiensten/ open-data/data
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.