Time Series Forecastability Measures

Alexis Roos; Rui Wang; Steven Klee

REVIEW 4 major objections 4 minor 2 cited by

Time Series Forecastability Measures

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two cheap statistics computed before training predict forecast error at r≈0.9 on M5.

desk verdict Sensible idea and honest synthetic checks, but the M5 headline correlation is an aggregate-level artifact until shown otherwise. read the letter →

arxiv 2507.13556 v1 pith:Z2EWPZE6 submitted 2025-07-17 cs.LG cs.AI

Rui Wang , Steven Klee , Alexis Roos This is my paper · ORCID

classification cs.LGcs.AI

keywords timeseriesforecastabilityspectralpredictabilityLyapunovexponentM5datasetsupplychainforecastingmodel-agnosticmetricsforecasterrorcorrelation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a time series' forecastability is measurable before any forecasting model is trained, using two cheap statistics: spectral predictability, which measures how concentrated the series' frequency spectrum is, and the largest Lyapunov exponent, which measures how quickly nearby states of the underlying dynamics diverge. If this is right, practitioners can rank thousands of demand series by expected forecast difficulty, focus modeling effort on the tractable ones, and set lower expectations for chaotic or noisy ones, all without training a single model. The evidence comes from synthetic series of increasing complexity and from the M5 hierarchical sales data, where the two metrics correlate strongly ($r \approx 0.9$) with the actual forecast error of ETS, RecursiveTabular, and Chronos across aggregation levels and daily/weekly frequencies.

What carries the argument

The load-bearing objects are two scalar statistics computed directly from the raw series. Spectral predictability is $1 - H_a(\boldsymbol{y})/\log_a(2\pi)$, one minus the entropy of the Hann-windowed power spectral density, normalized so that a flat spectrum gives 0 and a constant series gives 1. The largest Lyapunov exponent is estimated by reconstructing the state space via time-delay embedding $\boldsymbol{x}_t = (y_t, y_{t+\tau}, \dots, y_{t+(m-1)\tau})$, finding the nearest neighbor of each embedded state, and averaging the logarithmic divergence rate of those pairs over $\Delta t$ steps, with positive $\lambda$ indicating chaos. The argument works because the two statistics capture complementary failure modes: spectral entropy measures how much structure lives in the frequency domain, while $\lambda$ measures instability in the time domain, and the M5 experiments show both line up with downstream forecast error.

What would settle it

Recompute the M5 correlations while varying the embedding dimension $m$ and delay $\tau$ in Eq. (3) over standard ranges and while excluding series with sparsity above 0.8; if the $r \approx 0.9$ correlation with WAPE disappears for any reasonable parameter choice, or if the product-level correlation vanishes once the sparsest series are removed, the claim that these metrics robustly reflect inherent forecastability would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that the inherent forecastability of a time series can be captured by the spectral predictability score $\Omega(\boldsymbol{y}) = 1 - H_a(\boldsymbol{y})/\log_a(2\pi)$, where $H_a$ is the entropy of the normalized power spectrum, together with the largest Lyapunov exponent $\lambda$ estimated from time-delay embedding and nearest-neighbor divergence. Higher $\Omega$ means energy is concentrated in few frequency components; non-positive $\lambda$ means the reconstructed dynamics do not exponentially amplify small differences. The paper reports that on M5, these two numbers, computed before training, track the WAPE of three quite different forecasting models with $r \approx 0.9$: higher spectral predictability and lower Lyapunov exponents go with smaller errors. It concludes that the metrics reflect the intrinsic difficulty of the forecasting task, not the quirks of any particular model.

Load-bearing premise

The load-bearing premise is that the largest Lyapunov exponent can be reliably estimated for short, sparse retail demand series by the nearest-neighbor divergence method with the paper's default embedding choices, even though the paper's own sensitivity results show that sparsity above 0.8 falsely signals stability and product-level estimates vary enormously.

Editorial extensions

If this is right

Planners can screen large catalogs of demand series with an $O(T\log T)$ spectral score before training and rank them by expected forecastability.
Series with spectral predictability below 0.2 or Lyapunov exponent above 1.0 can be flagged as low-forecastability, prompting different strategies or hedges.
Aggregating product-level M5 series from daily to weekly raises spectral predictability and lowers the Lyapunov exponent, so temporal aggregation is a concrete lever for improving forecastability.
Computing both metrics over sliding windows can flag regime shifts or distributional changes and signal when to retrain forecast models.
The same two metrics offer a model-agnostic explanation for why one model type succeeds on some series and fails on others, since they correlate with error across ETS, RecursiveTabular, and Chronos.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the $r \approx 0.9$ relationship generalizes beyond M5, the metrics could serve as a fast, model-free proxy for held-out error in automated model selection and retraining pipelines, a use the paper lists as future work.
The product-level Lyapunov estimates in Table 1 have enormous variance (0.833 ± 1.495 for daily series), which suggests the estimator's reliability at disaggregated scales is the main open question for practical deployment; a robust estimator or longer series would be needed.
Because the paper notes neither metric can distinguish deterministic chaos from pure noise, a natural extension would be a third statistic or statistical test to separate those cases, which would sharpen the forecastability interpretation.
A direct testable extension would be to run the same two metrics on a different hierarchical dataset (for instance, intermittent or cross-sectional sales data) and check whether the correlation with forecast error survives when the hierarchy levels and sparsity patterns differ.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the core idea is sensible — use cheap, model-free metrics to rank time series by forecastability before training models — and the synthetic experiments are a reasonable sanity check. But the headline M5 result, r≈0.9 between the metrics and WAPE, is computed on aggregate hierarchy levels (at most 8 points), so it mostly says that lower levels are noisier and harder to forecast. That's true but trivially so; it does not show the metrics rank individual products, which is the stated practical use case.

What's new: applying Goerg's spectral predictability and Dingwell's Lyapunov exponent to the M5 hierarchy, together with a recommendation to combine them. That is a legitimate application, not a new method. To the authors' credit, they explicitly note the analysis doesn't determine which model performs best under varying forecastability, and they flag that sparsity >0.8 can falsely make Lyapunov exponents look stable.

Soft spots, in order of severity. First, Eq. (2) normalizes spectral entropy by log_a(2π), but the maximum entropy of a discrete spectrum depends on the number of frequency bins, not on 2π; as written the claimed [0,1] bound is not correct. Second, the Lyapunov estimation is under-specified — no embedding dimension m, no delay tau, no details on nearest-neighbor selection — and Table 1 shows L3 daily Lyapunov values of 0.833 ± 1.495, which is essentially noise at the product level where the metric is most needed. Third, the M5 validation is aggregate-only: no within-level correlations, no confidence intervals, no controlling for series length or sparsity. The stress-test note is right that the r≈0.9 likely reflects hierarchy-level differences rather than per-series forecastability. Fourth, no code or data are provided, so the numbers cannot be checked.

The practical claim — that these metrics let practitioners focus planning on individual forecastable products — is unsupported as presented. But the paper is not careless; it is a short workshop paper with a plausible idea and honest caveats. I would send it to peer review with the expectation that the M5 analysis be redone at the per-series level, with proper controls and full details on the Lyapunov estimation. Not a desk reject, but a revise-and-resubmit.

Referee Report

4 major / 4 minor

Summary. The paper proposes two pre-training metrics for assessing time-series forecastability: a spectral predictability score based on spectral entropy and the largest Lyapunov exponent estimated from time-delay embedding. The authors argue that these model-agnostic measures can identify inherently unpredictable series before any forecasting model is trained, allowing practitioners to allocate effort more effectively. Validation is attempted on synthetic series with varying complexity and on the hierarchical M5 dataset, where the metrics are compared with WAPE from ETS, RecursiveTabular, and Chronos. The central claim is that both metrics correlate strongly (r ≈ 0.9) with actual forecast errors across hierarchy levels and temporal frequencies.

Significance. If the central claim were established, the paper would offer a lightweight, model-agnostic screening tool for supply-chain forecasting, which is practically valuable and complements standard post hoc error evaluation. The spectral metric is computationally efficient, and the synthetic experiment provides an intuitive illustration of how entropy and divergence respond to changing signal complexity. The sensitivity study in Section 3.2 is also a useful check on length and sparsity effects. However, the manuscript's headline quantitative evidence rests on aggregate-level correlations with a handful of points, and the normalization of the spectral predictability score is mathematically flawed as written. The Lyapunov estimator is under-specified and shown by the authors' own sensitivity analysis to be unreliable for sparse series, which is exactly the regime of product-level M5 data. These issues block acceptance in the current form, but they are addressable with additional per-series analysis and corrected definitions.

major comments (4)

[§2.1, Eqs. (1)–(2)] The normalization of the spectral predictability score is mathematically incorrect. With normalized power p_i summing to 1 over the discrete Fourier bins, each p_i is at most 1, so log_a p_i is non-positive and the quantity H_a defined in Eq. (1) is non-positive. The expression Ω = 1 − H_a / log_a(2π) then cannot be bounded in [0,1] as claimed. Moreover, the maximum entropy for N discrete frequency bins is log_a N (or log_a(N/2+1) for one-sided spectra), not log_a(2π). The threshold of 0.2 used in Section 4 therefore does not follow from the definition. Please replace Eq. (1) with the conventional spectral entropy H = −∑ p_i log_a p_i and normalize by the appropriate bin count, or explicitly state which frequency domain and binning convention justifies log_a(2π).
[§3.3, Figure 5 and Table 1] The reported strong correlation (r ≈ 0.9) is computed over aggregate-level points—one average metric value and one WAPE per hierarchy level and temporal frequency—rather than over individual M5 series. With at most four hierarchy levels and two frequencies, the correlation is dominated by the well-known fact that lower-level series are shorter, sparser, and noisier; it does not establish that the metrics rank individual products or series by forecastability. This is load-bearing for the stated practical use of focusing planning effort on products. Please report per-series within-level correlations (for example, across L3 product series), partial correlations controlling for series length and sparsity, exact correlation coefficients with confidence intervals, and scatterplots at the per-series level.
[§2.2, Eq. (6), and Table 1] The Lyapunov exponent estimator is under-specified and unreliable in the regime where it is needed. The embedding dimension m, delay τ, and evolution time Δt are not stated, and Eq. (6) as written computes a finite-time nearest-neighbor divergence ratio rather than a standard largest-Lyapunov estimator. The paper's own sensitivity analysis in Section 3.2 says that sparsity above 0.8 falsely indicates stability, and Table 1 reports L3 daily Lyapunov values of 0.833 ± 1.495, meaning many product-level series have negative or near-zero estimates and the average is not meaningful. Since M5 product-level series are sparse, the Lyapunov-based aggregate correlation is not trustworthy. Please adopt an established estimation method with explicit parameters (e.g., Rosenstein or Kantz) and either restrict the validation to series satisfying the stated length and sparsity requirements or omit the Lyapunov part from the headline claim.
[§4, Discussion] The practical guidance that spectral predictability below 0.2 or Lyapunov exponents above 1.0 indicate low forecastability is proposed after inspecting the same M5 data used to demonstrate the correlation, and no out-of-sample validation or statistical derivation is provided. These thresholds are a central part of the practical contribution, so they should be justified on held-out data or presented as exploratory with appropriate caveats.

minor comments (4)

[§3.1, Figure 2] The moving-window analysis uses window sizes of 200 and 300 without any sensitivity analysis; a brief statement on how these choices affect the reported transitions would improve reproducibility.
[§3.2, Figures 3 and 4] The conclusion that sparsity above 0.8 falsely indicates stability is based on visual inspection; reporting the exact values or providing a table would make the claim testable.
[References] Reference [17] is cited in the Introduction as support for the a priori forecastability approach, but the cited paper is about a Koopman neural operator forecaster, not about forecastability measures; please check the citation or replace it with a more appropriate reference.
[General] The manuscript contains inconsistent naming for the tabular model ('RecursiveTabular' in the abstract and Section 3.3 versus 'RecursiveTab' in the text) and the running header still reads 'Trovato et al.'; these presentation issues should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the forecastability metrics are defined independently of forecast errors and are validated against external WAPE benchmarks.

full rationale

The core derivation chain is self-contained. Spectral predictability (Eqs. 1-2) and the largest Lyapunov exponent (Eqs. 3-6) are computed directly from each time series without using downstream forecast errors or model outputs; no parameter is fitted to WAPE. The validation in Section 3.3 compares these pre-computed metrics against WAPE from ETS, RecursiveTabular, and Chronos, which are external benchmarks, and the synthetic experiments in Section 3.1 provide an additional independent check. No claim reduces by construction to its inputs: the observed negative correlation for spectral predictability and positive correlation for Lyapunov exponents with WAPE are empirical, not definitional. The paper's own caution that the analysis does not determine which forecasting model performs best, and the aggregate-level (eight-point) nature of the M5 correlation, are statistical-validity concerns rather than circularity. The only M5-derived elements, the heuristic thresholds (spectral predictability below 0.2, Lyapunov exponent above 1.0), are presented as practical guidelines, not as predictions validated by the same experiment, and they do not feed back into the metric definitions. Self-citations (e.g., [17]) are not load-bearing. Overall score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper relies on established mathematical tools (FFT, time-delay embedding) and domain assumptions about the relationship between spectral flatness/chaos and forecastability. It introduces no new entities. The free parameters are mostly unspecified hyperparameters and post hoc thresholds, which a critic should treat as assumptions.

free parameters (3)

moving window sizes = 200 (spectral), 300 (Lyapunov) in synthetic experiment
Chosen for illustration; not justified or tested across datasets.
embedding dimension m and delay tau = not specified
Required for time-delay embedding (Eq. 3); the paper does not state how these are set for each series, yet the Lyapunov estimate depends on them.
thresholds for low forecastability = spectral predictability < 0.2, Lyapunov > 1.0
Proposed in Section 4 as practical guidelines; they appear to be derived from the observed data without out-of-sample validation.

assumptions (4)

domain assumption The spectral entropy is inversely related to forecastability
Section 2.1; used to define the spectral predictability score.
standard math Time-delay embedding reconstructs the system's state space (Takens' theorem)
Equation (3) relies on this theorem.
domain assumption The largest Lyapunov exponent can be reliably estimated with the naive nearest-neighbor method and the chosen hyperparameters
Section 2.2; no validation on benchmark chaotic systems; the paper itself notes failure under high sparsity.
domain assumption M5 dataset aggregation levels are representative of supply-chain forecasting tasks
Used for validation; generalizability beyond M5 is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Time Series Forecastability Measures." pith.science (2026). https://pith.science/paper/Z2EWPZE6

@misc{pith2026250713556,
  author       = {Pith},
  title        = {Pith review of: Time Series Forecastability Measures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2EWPZE6}},
  note         = {Machine review of arXiv:2507.13556}
}

read the original abstract

This paper proposes using two metrics to quantify the forecastability of time series prior to model development: the spectral predictability score and the largest Lyapunov exponent. Unlike traditional model evaluation metrics, these measures assess the inherent forecastability characteristics of the data before any forecast attempts. The spectral predictability score evaluates the strength and regularity of frequency components in the time series, whereas the Lyapunov exponents quantify the chaos and stability of the system generating the data. We evaluated the effectiveness of these metrics on both synthetic and real-world time series from the M5 forecast competition dataset. Our results demonstrate that these two metrics can correctly reflect the inherent forecastability of a time series and have a strong correlation with the actual forecast performance of various models. By understanding the inherent forecastability of time series before model training, practitioners can focus their planning efforts on products and supply chain levels that are more forecastable, while setting appropriate expectations or seeking alternative strategies for products with limited forecastability.

Figures

Figures reproduced from arXiv: 2507.13556 by the authors.

**Figure 3.** The sensitivity of the spectral predictability to vary [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** The sensitivity of the Lyapunov Exponent to vary [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Spectral predictability vs prediction WAPE across [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

The Spectrum Is Not Enough: When Context Helps Time-Series Forecasting
cs.LG 2026-07 conditional novelty 7.0 of 10

Spectral indices cannot predict when context helps time-series forecasting; a coverage-deficit diagnostic measuring analog-over-linear gain can.
End-to-end probabilistic hierarchical forecasting of large hierarchies via probabilistic top-down
stat.ME 2026-06 unverdicted novelty 6.0 of 10

e2eTD forecasts a small subset of aggregate series (~0.3% of hierarchy) and propagates joint samples via probabilistic top-down disaggregation to produce coherent probabilistic forecasts, achieving lowest weighted sca...

Reference graph

Works this paper leans on

17 extracted references · 14 canonical work pages · cited by 2 Pith papers

[1]

Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebas- tian Pineda Arango, Shubham Kapoor, et al. 2024. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815 (2024)

arXiv 2024
[2]

Yossi Aviv. 2003. A time-series framework for supply-chain inventory manage- ment. Operations Research 51, 2 (2003), 210–227

work page 2003
[3]

Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang Wang, Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael Bohlke-Schneider, David Salinas, Lorenzo Stella, et al. 2022. Deep learning for time series forecasting: Tutorial and literature survey. Comput. Surveys 55, 6 (2022), 1–36

work page 2022
[4]

Jonathan B Dingwell. 2006. Lyapunov exponents.Wiley encyclopedia of biomedical engineering (2006)

work page 2006
[5]

Pierre Duhamel and Martin Vetterli. 1990. Fast Fourier transforms: a tutorial review and a state of the art. Signal processing 19, 4 (1990), 259–299

work page 1990
[6]

Everette S Gardner Jr. 1985. Exponential smoothing: The state of the art. Journal of forecasting 4, 1 (1985), 1–28

work page 1985
[7]

Georg Goerg. 2013. Forecastable component analysis. In International conference on machine learning. PMLR, 64–72

work page 2013
[8]

James D Hamilton. 2020. Time series analysis. Princeton university press

work page 2020

Show all 17 references

[9]

Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. 2024. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining . 6555–6565

2024
[10]

Bryan Lim and Stefan Zohren. 2021. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A 379, 2194 (2021), 20200209

2021
[11]

Douglas A Lyon. 2009. The discrete fourier transform, part 4: spectral leakage. Journal of object technology 8, 7 (2009)

2009
[12]

John T Mentzer, William DeWitt, James S Keebler, Soonhong Min, Nancy W Nix, Carlo D Smith, and Zach G Zacharia. 2001. Defining supply chain management. Journal of Business logistics 22, 2 (2001), 1–25

2001
[13]

Nicolas Pielawski and Carolina Wählby. 2020. Introducing Hann windows for reducing edge-effects in patch-based image segmentation. PloS one 15, 3 (2020), e0229839

2020
[14]

Damien Power. 2005. Supply chain management integration and implementation: a literature review. Supply chain management: an International journal 10, 4 (2005), 252–263

2005
[15]

Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. 2018. Deep state space models for time series forecasting. Advances in neural information processing systems 31 (2018)

2018
[16]

Oleksandr Shchur, Ali Caner Turkmen, Nick Erickson, Huibin Shen, Alexander Shirkov, Tony Hu, and Bernie Wang. 2023. AutoGluon–TimeSeries: AutoML for probabilistic time series forecasting. In International Conference on Automated Machine Learning. PMLR, 9–1

2023
[17]

Rui Wang, Yihe Dong, Sercan O Arik, and Rose Yu. 2023. Koopman Neural Operator Forecaster for Time-series with Temporal Distributional Shifts. In The Eleventh International Conference on Learning Representations. https://openreview. net/forum?id=kUmdmHxK5N

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebas- tian Pineda Arango, Shubham Kapoor, et al. 2024. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815 (2024)

arXiv 2024

[2] [2]

Yossi Aviv. 2003. A time-series framework for supply-chain inventory manage- ment. Operations Research 51, 2 (2003), 210–227

work page 2003

[3] [3]

Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang Wang, Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael Bohlke-Schneider, David Salinas, Lorenzo Stella, et al. 2022. Deep learning for time series forecasting: Tutorial and literature survey. Comput. Surveys 55, 6 (2022), 1–36

work page 2022

[4] [4]

Jonathan B Dingwell. 2006. Lyapunov exponents.Wiley encyclopedia of biomedical engineering (2006)

work page 2006

[5] [5]

Pierre Duhamel and Martin Vetterli. 1990. Fast Fourier transforms: a tutorial review and a state of the art. Signal processing 19, 4 (1990), 259–299

work page 1990

[6] [6]

Everette S Gardner Jr. 1985. Exponential smoothing: The state of the art. Journal of forecasting 4, 1 (1985), 1–28

work page 1985

[7] [7]

Georg Goerg. 2013. Forecastable component analysis. In International conference on machine learning. PMLR, 64–72

work page 2013

[8] [8]

James D Hamilton. 2020. Time series analysis. Princeton university press

work page 2020