REVIEW 5 major objections 4 minor 1 cited by
Beyond Model Ranking: Predictability-Aligned Evaluation for Time Series Forecasting
T0 review · 5 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read A spectral-coherence score separates a time series' intrinsic difficulty from a model's forecasting error, and a companion ratio reveals how much of the linear predictability each model actually exploits.
desk verdict The SCP/LUR diagnostic idea is worth a look, but the paper's central 'lower bound' is not a lower bound for the predictors it evaluates, and the empirical validation is thinner than claimed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Squared spectral coherence γ²_xy(f) between history and future, estimated by Welch's method, is the load-bearing object. Its complement (1 − γ²) scales the target spectrum to give the residual spectrum Ŝ_e(f) = Ŝ_yy(f)(1 − γ²_xy(f)); summing this over frequencies (plus a boundary mean-shift term Δ²) gives MSE_lb, the linear lower bound. SCP is the normalized inverse of that bound; LUR compares the prediction-target coherence against this same linear ceiling.
What would settle it
Run SCP on a synthesized chirp or regime-switching process with a known linear forecaster and check whether the realized MSE ever falls below the claimed MSE_lb; a single violation on a stationary Gaussian process with known covariance would also falsify the bound's universality, as would a Monte Carlo where the Welch coherence estimate's confidence interval excludes the true coherence for a known spectrum.
Extended reading notes
Core claim
The paper's central claim is that for a history x and future y, the quantity MSE_lb = Δ² + Σ_f Ŝ_yy(f)(1 − γ²_xy(f)) is a lower bound on the MSE of any linear time-invariant predictor using x, so the score P_xy = 1 − MSE_lb / dVar(y) measures the linearly predictable fraction of the target's variance. On jointly Gaussian wide-sense stationary segments this is a consistent estimator of the Bayes-risk predictability P*. The companion LUR = P_model / P_linear, with P_model and P_linear being power-weighted coherences between prediction and target and history and target, respectively, tells whether a model under-utilizes, saturates, or exceeds the linear information available. The paper asserts
Load-bearing premise
The framework's load-bearing premise (acknowledged in Appendix B) is that a single short history–future pair, windowed with length ⌊0.25N⌋, gives a reliable squared-coherence estimate under local wide-sense stationarity; if real series are not stationary at that scale, SCP and LUR become noisy and MSE_lb may not be a valid lower bound.
Editorial extensions
If this is right
- A model's MSE can be decomposed into an intrinsic data-dependent floor (MSE_lb) plus the model's extra error; evaluations can be stratified by SCP to compare models only within comparable difficulty.
- Predictability drift implies aggregate test-set statistics are misleading; reporting should be per-sample or per-band and conditioned on predictability.
- Band-wise LUR reveals that linear models behave as broadband filters while transformers focus on high-energy low-frequency bands, explaining their comparative strengths.
- SCP can guide curriculum learning and data sampling by difficulty, and LUR can inform regularizers; the paper suggests these as future uses.
Reading between the lines
- If SCP is valid, benchmark leaderboards could be re-reported as a function of P, making claims like 'model A beats B' conditional on difficulty; this could change how model selection is done for deployed systems where the data regime is known.
- Because SCP isolates linear predictability, a model with LUR > 1 signals real nonlinear or exogenous information capture; the framework could be used to audit whether a predictive model is genuinely learning dynamics or exploiting trivial correlations.
- A testable extension: use SCP as a dynamic difficulty measure for active learning or online learning, re-estimating predictability on a rolling window to guide when to retrain or switch models.
- The per-instance lower bound is only for linear time-invariant predictors; a natural extension would be a comparable bound for nonlinear classes (e.g., via conditional coherence or transfer entropy) to tighten the baseline.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a predictability-aligned evaluation framework for time-series forecasting. It defines a Spectral Coherence Predictability (SCP) score and a Linear Utilization Ratio (LUR), both based on Welch spectral estimates and squared coherence between history x and future y. The central theoretical claim is that MSElb = Δ² + Σ_f Ŝ_e(f) in Eq. (7) lower-bounds the MSE of any linear time-invariant predictor using x. The paper validates SCP on synthetic and real benchmarks, reports high correlation between MSElb and realized model MSE, and uses the framework to argue for 'predictability drift' and complementary strengths of linear vs. nonlinear architectures.
Significance. If the theoretical foundation were correct, the framework would be a valuable and computationally efficient diagnostic tool. The paper ships reproducible code, gives clear algorithms, and addresses an important problem: separating model error from irreducible data difficulty. However, the central lower-bound claim is false as stated. The error propagates into the definition of SCP, the interpretation of Fig. 1a, the correlation analyses in Table 1, and the 'fundamental limit' narrative. The other contributions (LUR, band-wise diagnostics) are interesting but rest on the same questionable spectral-coherence foundation, and the empirical validation is weakened by variance-induced correlation and a partly mechanical inverse relation between P and NMSE. I cannot recommend publication in the current form.
major comments (5)
- [Section 4.1, Eq. (7)] The claimed lower bound is not a lower bound for the stated class of predictors. After mean removal, the spectra capture AC power, while Δ²=(mean(y)-mean(x))² assumes the predictor has DC gain exactly 1. An affine/linear predictor with arbitrary DC gain can eliminate the mean error. Concrete counterexample: x_t = 2 + sin(2πt/64), y_t = 0.5 x_t = 1 + 0.5 sin(2πt/64). Then coherence is 1, residual spectral sum ≈0, Δ²=1, so MSElb≈1, but the linear predictor f(x)=0.5x has MSE=0. Since the models evaluated in the paper (DLinear, iTransformer, etc.) all include affine/bias components, the bound does not apply to them. This invalidates the theoretical interpretation in Section 4.1 and the validation claims in Sections 5.1–5.2. The bound would only hold for predictors with DC gain exactly 1, a restriction not stated or used. The correct unconstrained linear lower bound would be Σ_f Ŝ_e(f) withou
- [Section 5.2, Table 1 and Fig. 2] The reported Pearson correlations R between MSE and MSElb are inflated by shared dependence on target variance. Both quantities scale with Var(y): MSElb contains Σ_f Ŝ_yy(f)(1−γ²), and model MSE is also proportional to Var(y) whenever the model is not much better than predicting the mean. Thus high R can arise even if the bound has no predictive content beyond variance. The paper should report partial correlations controlling for Var(y), or correlations of normalized errors (NMSE vs. MSElb/Var(y)). Without such a control, 'strongly correlates' in Section 5.2 does not substantiate the claimed alignment.
- [Section 5.1, toy study] The toy study in Section C.1 validates the bound only for a single least-squares linear forecaster on one synthetic multiband Gaussian process, with 3 trials. This is insufficient to establish that MSElb is a valid lower bound 'for any linear time-invariant predictor,' especially given the DC-gain issue in Eq. (7). The experiment should include nonlinear predictors, processes with mismatched segment means, and varying Welch parameters. As written, Fig. 1a merely shows one Wiener filter's MSE cannot go below a quantity that is not actually a lower bound.
- [Section 5.3 and Appendix B/C.4] The 'predictability drift' evidence is based on a single channel of ETTh1 at horizon N=96 (Fig. 3), with per-instance Welch estimates using n_win=⌊0.25N⌋=24 and 50% overlap. For short windows, the coherence estimate has high variance, and Appendix B explicitly acknowledges reliance on second-order stationarity within windows. A single-channel, single-dataset, high-variance estimate does not support the paper's claim of 'the first systematic evidence' of predictability drift. The authors should provide multi-dataset, multi-channel evidence with confidence intervals or statistical tests for drift.
- [Section 5.2, Fig. 1b] The inverse relationship between P and NMSE is partly mechanical. Since MSElb ≤ MSE (if the bound were correct), NMSE = MSE/Var(y) ≥ MSElb/Var(y) = 1−P. Thus datasets with higher P necessarily have a higher lower bound on achievable NMSE. The observed inverse trend is therefore not independent evidence that P 'aligns with realized errors.' The text should treat Fig. 1b as a consistency check, not as validation of the metric.
minor comments (4)
- [Section 3, Eq. (4)] P⋆ is defined as 1 − MSE⋆/Var(y). This is the theoretical R², but the paper should clarify that for finite samples and non-Gaussian processes it is an approximation; the current wording in Eq. (8) treats the estimator as automatically in [0,1].
- [Algorithm 2, Eq. (12)] LUR is undefined when P_linear = 0. Some frequency bands may have zero estimated linear explainable power, especially with short windows. The algorithm should specify a default value or a guard condition.
- [Section 2 and Appendix C.2] The time-mixer citation in Section C.2 points to 'TimeXer (Wang et al., 2024)' rather than a TimeMixer reference; the reference list entry is for TimeXer, and the model name in Table 1 is TimeMixer. Please reconcile.
- [Appendix C.4] The Welch parameters are fixed as n_win=⌊0.25N⌋, 50% overlap. For small horizons (e.g., N=96), this yields only a few frequency bins; the paper should report sensitivity to Welch parameter choices, since the SCP/LUR values and the drift evidence depend on them.
Circularity Check
No significant circularity: SCP/MSElb are data-only spectral quantities validated against independent model errors; the DC-gain counterexample is a correctness issue, not circularity.
full rationale
Walked the derivation chain. Algorithm 1 defines MSElb = Δ² + Σ_f Ŝ_e(f) with Ŝ_e = Ŝ_yy(1-γ²_xy) directly from Welch spectra of (x,y); SCP = 1 − MSElb/dVar(y). No model output is used to fit any parameter, and LUR uses ŷ only as an independent quantity. The claimed lower-bound property (Eq. 7) is a mathematical Wiener/coherence assertion, not an equivalence introduced by definition; it is checked against realized MSE (Sec. 5.1–5.2). The inverse P–NMSE trend is not forced: the bound only entails NMSE ≥ 1−P, which is very loose for many rows (e.g., Weather NMSE≈2.6 vs 1−P≈0.65). LUR interpretations are restatements of its definition but are not used to 'predict' model error by construction. Self-citations (Wang et al. 2021) are background, not load-bearing; no uniqueness theorem is imported. Appendix B explicitly acknowledges the stationarity/linearity limitations. A separate validity concern exists: the Δ² term assumes unity DC gain, so affine predictors can beat MSElb (e.g., DLinear ECL N=96 MSE 0.195 < reported MSElb 0.239, Table 1); this is a correctness risk, not a circular reduction.
Assumptions & free parameters
free parameters (4)
- Welch window length n_win =
floor(0.25N)
- Welch overlap =
50%
- Tikhonov stability constant epsilon =
not specified (described as small)
- Frequency band partition for band-wise LUR =
low/mid/high, boundaries unspecified
assumptions (4)
- domain assumption Wide-sense stationarity of x and y within Welch windows
- domain assumption Joint Gaussianity and stationarity around the boundary for P to estimate the Bayes risk P*
- standard math The squared coherence and residual spectrum give the MMSE of the optimal linear filter (Wiener theory)
- standard math Welch periodogram with variance-preserving normalization approximates true spectra
Cite this review
Pith. "Pith review of Beyond Model Ranking: Predictability-Aligned Evaluation for Time Series Forecasting." pith.science (2026). https://pith.science/paper/VJSRFNLQ
@misc{pith2026250923074,
author = {Pith},
title = {Pith review of: Beyond Model Ranking: Predictability-Aligned Evaluation for Time Series Forecasting},
year = {2026},
howpublished = {\url{https://pith.science/paper/VJSRFNLQ}},
note = {Machine review of arXiv:2509.23074}
}
abstract
In the era of increasingly complex AI models for time series forecasting, progress is often measured by marginal improvements on benchmark leaderboards. However, this approach suffers from a fundamental flaw: standard evaluation metrics conflate a model's performance with the data's intrinsic unpredictability. To address this pressing challenge, we introduce a novel, predictability-aligned diagnostic framework grounded in spectral coherence. Our framework makes two primary contributions: the Spectral Coherence Predictability (SCP), a computationally efficient ($O(N\log N)$) and task-aligned score that quantifies the inherent difficulty of a given forecasting instance, and the Linear Utilization Ratio (LUR), a frequency-resolved diagnostic tool that precisely measures how effectively a model exploits the linearly predictable information within the data. We validate our framework's effectiveness and leverage it to reveal two core insights. First, we provide the first systematic evidence of "predictability drift", demonstrating that a task's forecasting difficulty varies sharply over time. Second, our evaluation reveals a key architectural trade-off: complex models are superior for low-predictability data, whereas linear models are highly effective on more predictable tasks. We advocate for a paradigm shift, moving beyond simplistic aggregate scores toward a more insightful, predictability-aware evaluation that fosters fairer model comparisons and a deeper understanding of model behavior.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
The Spectrum Is Not Enough: When Context Helps Time-Series Forecasting
Spectral indices cannot predict when context helps time-series forecasting; a coverage-deficit diagnostic measuring analog-over-linear gain can.
Reference graph
Works this paper leans on
-
[11]
ISSN 2374-3468. doi: 10.1609/aaai.v37i9.26317. Kai Zhao, Denis Khryashchev, and Huy V o. Predicting Taxi and Uber Demand in Cities: Approach- ing the Limit of Predictability.IEEE Transactions on Knowledge and Data Engineering, 33(6): 2723–2736, June
-
[12]
doi: 10.1109/TKDE.2019.2955686
ISSN 1558-2191. doi: 10.1109/TKDE.2019.2955686. 11 Under review Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11106–11115,
arXiv 2019
-
[128]
drop-last
The forecaster is a causal FIR least- squares filter (Wiener approximation) of lengthL FIR = 64with ridge10 −6. The base process has four spectral peaks at rFFT bins{32,96,192,384}with widths{6,10,14,18}and amplitudes {3.0,2.0,1.5,1.0}. We sweep noise levels{0,0.25,0.5,1.0,2.0,4.0}on a single band (index 1 in the plot) and average over3trials, reporting m...
2024
-
[1948]
doi: 10.1002/j.1538-7305.1948.tb01338.x
ISSN 0005-8580. doi: 10.1002/j.1538-7305.1948.tb01338.x. Chaoming Song, Zehui Qu, Nicholas Blumm, and Albert-L´aszl´o Barab´asi. Limits of predictability in human mobility.Science, 327(5968):1018–1021, February
arXiv 1948
-
[1977]
ISSN 1557-9654. doi: 10.1109/TIT.1977. 1055714. 12 Under review A USAGE OFLLMS We utilized a large language model (LLM) to proofread and improve the grammatical clarity of this manuscript. All scientific ideas, methodologies, and conclusions presented are the original work of the authors. B LIMITATIONS AND OPPORTUNITIES Our estimates rely on windowed Welc...
-
[2000]
ISSN 0363-6135, 1522-1539. doi: 10.1152/ajpheart.2000. 278.6.H2039. C. E. Shannon. A mathematical theory of communication.The Bell System Technical Journal, 27 (3):379–423, July
-
[2002]
doi: 10.1103/PhysRevLett.88.174102
ISSN 0031-9007, 1079-7114. doi: 10.1103/PhysRevLett.88.174102. Christoph Bergmeir. Fundamental limitations of foundational forecasting models: The need for multimodality and rigorous evaluation. InProc. NeurIPS Workshop,
-
[2010]
Huandong Wang, Sihan Zeng, Yong Li, and Depeng Jin
doi: 10.1126/science.1177170. Huandong Wang, Sihan Zeng, Yong Li, and Depeng Jin. Predictability and Prediction of Human Mobility Based on Application-Collected Location Data.IEEE Transactions on Mobile Comput- ing, 20(7):2457–2472, July
Show all 15 references
-
[2014]
doi: 10.1103/PhysRevE.90.052910
ISSN 1539-3755, 1550-2376. doi: 10.1103/PhysRevE.90.052910. Marta C. Gonz´alez, C´esar A. Hidalgo, and Albert-L´aszl´o Barab´asi. Understanding individual human mobility patterns.Nature, 453(7196):779–782, June
-
[2018]
h” denotes hourly data and “m
Variants differ in sam- pling granularity: “h” denotes hourly data and “m” denotes 15-minute data; suffixes “1/2” indicate the two regions (ETTh1/ETTh2, ETTm1/ETTm2) (Zhou et al., 2021). •Weather .Weather contains 21 meteorological variables measured every 10 minutes during 20...
2021
-
[2019]
doi: 10.1002/ecm.1359
ISSN 0012-9615, 1557-7015. doi: 10.1002/ecm.1359. S M Pincus. Approximate entropy as a measure of system complexity.Proceedings of the National Academy of Sciences, 88(6):2297–2301, March
-
[2021]
doi: 10.1109/TMC.2020.2981441
ISSN 1558-0660. doi: 10.1109/TMC.2020.2981441. Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Guo Qin, Haoran Zhang, Yong Liu, Yunzhong Qiu, Jian- min Wang, and Mingsheng Long. TimeXer: Empowering Transformers for Time Series Forecast- ing with Exogenous Variables. InThe Thirty-eighth ...
2020
-
[2023]
2211.14730
doi: 10.48550/arXiv. 2211.14730. Frank Pennekamp, Alison C. Iles, Joshua Garland, Georgina Brennan, Ulrich Brose, Ursula Gaedke, Ute Jacob, Pavel Kratina, Blake Matthews, Stephan Munch, Mark Novak, Gian Marco Palamara, Bj¨orn C. Rall, Benjamin Rosenbaum, Andrea Tabi, Colette W...
-
[2024]
ISBN 979-8-4007-0490-1
Association for Computing Machinery. ISBN 979-8-4007-0490-1. doi: 10.1145/3637528.3671995. Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. arXiv, March
-
[7527]
Ioannis Kontoyiannis, Paul H
doi: 10.1007/s10514-009-9119-x. Ioannis Kontoyiannis, Paul H. Algoet, Yu M. Suhov, and Abraham J. Wyner. Nonparametric entropy estimation for stationary processes and random fields, with applications to English text.IEEE transactions on information theory, 44(3):1319–1327,
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.