REVIEW 5 major objections 6 minor 32 references
Towards Reliable Zero-Shot Crowd Forecasting: Evaluating Time Series Foundation Models for Special Event Pedestrian Forecasting
T0 review · 5 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read Pretrained time series foundation models, applied with no local retraining, can produce reliable 80% prediction intervals for pedestrian flow that stay stable for roughly 30 minutes during a large special event, and up to 40–45 minutes for
desk verdict Real-data zero-shot benchmark with a useful problem setup, but the main reliability claim is unanchored — no baseline, no error bars, and a few internal inconsistencies. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the pretrained foundation model used in zero-shot mode: TimesFM, a decoder-only Transformer with patch-based input, and Chronos-2, a T5-based model that tokenizes time series and predicts tokens. The evaluation machinery is a rolling 40-step forecast with an 80% central prediction interval formed by the models' 10th and 90th percentile outputs, judged by coverage and stability metrics — most importantly Stable Lead Time, which counts consecutive steps from t=1 that the interval contains the true value. The claimed utility rests on these intervals remaining calibrated (covering about 80% of realizations) while staying narrow enough to be useful, and on the whole pipel
What would settle it
Run the same zero-shot models on a second large special-event dataset (or a held-out event year) and compute the empirical coverage of the 80% intervals and the Stable Lead Time; if coverage falls well below 80% or SLT drops below about 20 steps, the claimed reliability horizon does not transfer.
Extended reading notes
Core claim
The paper claims that zero-shot deployment of pretrained time series foundation models can provide operationally reliable multi-step probabilistic pedestrian-flow forecasts during special events. On the SAIL2025 data, across almost all 22 directional time series, the Stable Lead Time — the number of 90-second steps from the forecast origin over which the 80% interval continuously covers the realized count — reaches about 20 steps (30 minutes), and Chronos-2 with an increasing context window plus time covariates reaches 27–30 steps (40–45 minutes). The paper interprets these horizons as meeting the 15–30 minute planning window common in traffic and crowd management. Chronos-2 generally outper
Load-bearing premise
The reliability conclusion rests on one five-day event at one location, with missing values filled in as zeros, and the models' quantile intervals are taken as calibrated at face value, with no comparison to simple baseline forecasters.
Editorial extensions
If this is right
- Crowd managers at infrequent events can deploy foundation models without gathering years of local training data, generating new probabilistic forecasts every 90 seconds.
- During core operational hours (08:00–22:00), operators can treat the 80% interval as reliable for roughly 30 minutes ahead; with the best Chronos-2 variant, about 40–45 minutes.
- Chronos-2 with an increasing context window is the recommended default setting, as it consistently beats fixed-context and non-covariate versions on coverage and stable lead time.
- The approach is computationally feasible: all models produce a 40-step forecast in under 0.5 seconds on CPU, with rolling memory below 52 MB after warm-up, so no GPU infrastructure is required.
- The Scaled Winkler Score, combined with coverage and width metrics, gives practitioners a compact reliability-oriented way to select among models, consistent with point-accuracy behavior.
Reading between the lines
- The reported 30–45 minute reliability horizon is likely to shorten near extreme surges or pattern shifts that were absent from the four training days; operators should monitor coverage online and treat intervals as less trustworthy beyond that window.
- Zero-imputation of missing counts could distort interval width and coverage; a sensitivity analysis that masks missing segments rather than filling them with zeros would either confirm or weaken the observed fail-safe widening behavior.
- Because the study covers a single event at one location, a multi-event comparison (different festival types, weather, city layouts) is needed to test whether the stable-lead-time numbers generalize beyond SAIL2025.
- The absence of simple baselines means the paper does not yet demonstrate that foundation-model intervals beat a naive seasonal forecast with conformalized intervals; adding such a comparison is the natural next step to quantify the added value.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates two pretrained time series foundation models, TimesFM and Chronos-2, for zero-shot probabilistic pedestrian-flow forecasting during the five-day SAIL2025 special event. It proposes decision-oriented metrics (FHC, NHC, SLT, NMPIW, SWS) and evaluates several model variants (fixed versus increasing context, with or without time features) on 22 directional sensor time series. The central claim is that these zero-shot models provide reliable multi-step probabilistic forecasts with a lead time of about 30 minutes (SLT ≈ 20–30 steps at a 90-second sampling interval), and that Chronos-2 with increasing context and time features is the best choice. The paper also reports CPU runtime and memory footprints to support operational feasibility.
Significance. If the central claim is established, the paper would offer a practical, no-retraining alternative for crowd management at infrequent special events, which is a genuinely useful contribution. The strengths include the use of real event data, concrete evaluation of computational feasibility (under 0.5 seconds per 40-step forecast on CPU, memory below 1 GB), and a set of decision-oriented metrics that can help practitioners select among models. The models are external pretrained black boxes, so there is no parameter fitting to the evaluation data, and the metrics are defined on observed outcomes rather than on model-internal quantities. However, the empirical basis is a single event, and the paper lacks comparison to simple baselines, statistical uncertainty quantification, and internally consistent interpretation of its own results. These gaps currently leave the central reliability claim insufficiently supported.
major comments (5)
- [Section V; Tables I and III] The central claim that zero-shot forecasts are 'reliable' with a lead time of about 30 minutes is not anchored by any baseline comparison. The metrics FHC, NHC, and SLT (Eqs. 2, 3, 5) are absolute; without comparing to a persistence or seasonal-naive probabilistic forecaster (e.g., using same-time previous-day values and an empirical residual interval), it is impossible to know whether the reported SLT values around 24–29 steps are attributable to the foundation models or merely to the strong daily periodicity of pedestrian flows and the persistence of coverage errors. Please add at least one simple baseline and report the same metrics, plus the realized per-lead-time coverage, to make the 'reliable' claim testable.
- [Section IV, text near Table I] The text states that 'across almost all sensors, SLT reaches about 20 steps ahead (30 minutes), and Chronos-2 (Increasing + TimeFeature) reaches about 27 to 30 steps ahead (40 to 45 minutes).' Table I shows the best Chronos-2 variant at 27–30 steps, but most other variants and TimesFM are mostly in the 24–28 range, not 'about 20.' The 20-step statement is inconsistent with the table. Moreover, the average SLT (Eq. 5) is not sufficient to establish a 30-minute operational lead time; the distribution of L_i and the probability that coverage is maintained for at least 20 consecutive steps should be reported.
- [Section IV, paragraph after Table II] The interpretation of NMPIW results is internally contradictory. The text notes that TimeFeature variants show larger NMPIW on several sensors, then says for high-variability sensors 'the model becomes less confident and outputs tighter intervals to maintain coverage.' Larger NMPIW means wider intervals, not tighter. Check against Table II (e.g., GASA-01-A2 135: Chronos-2 (Increasing) NMPIW 0.282 vs Chronos-2 (Increasing + TimeFeature) 0.298, i.e., wider with features). The explanation should be corrected to say that time features increase interval width in these cases, and the causal reasoning should be revised.
- [Section III-C; Table I] No uncertainty estimates are reported for any of the metrics. The paper compares models on a rolling forecast design, but does not report N (the number of forecast origins), standard errors, confidence intervals, or significance tests. For example, row GASA-01-A1 135 shows FHC 0.575 for Chronos-2 (Increasing + TimeFeature) versus 0.558 for TimesFM (Fixed); without sampling uncertainty it is unclear whether such differences are meaningful. Please report the number of forecast origins and provide bootstrap or analytical confidence intervals, or at least a paired test across the 22 time series.
- [Section III-A; Case III (Section IV-A3)] Missing values are imputed with zero, and Case III then analyzes a segment with about 30% 'missing' data by treating these zeros as if they were legitimate observations. Since the interval bounds are non-negative and missing values are replaced by zero, the coverage of these imputed points is systematically affected (a zero is covered only if the lower bound is exactly zero, which is unlikely, so it may artificially reduce coverage, or if the interval is wide, it may artificially increase it). The robustness experiment is confounded with the imputation rule. Please evaluate the sensitivity by withholding data and comparing with a model that simply skips the missing segment, or by reporting results on the complete-case subset.
minor comments (6)
- [Section III-C, Eq. (4)] Typos: 'groud-truth remain' should be 'ground truth remains.' Please proofread the equations and text.
- [Section IV, paragraph after Table I] The sentence 'The night-time experiments (22:00 - 08:00) across all sensors also show similar results: TimesFM is more reliable in low-flow periods; due to page limitations, these results are not shown here' makes a claim without providing the supporting evidence. Either include the results in an appendix or remove the claim.
- [Section IV-D, Fig. 7] The claim that 'SWS contains information beyond RMSE' because it accounts for interval width is not demonstrated by the high R^2 alone. Consider reporting cases where SWS and RMSE orderings differ, or a partial correlation controlling for RMSE.
- [Section II, reference [9]] Reference [9] is an arXiv preprint; please include the version number and access year, and ensure the citation is consistent with the actual Chronos-2 paper.
- [Section III-D, NMPIW definition] NMPIW normalizes by the historical min-max range, which is sensitive to outliers and to the zero-imputed missing values. Please discuss this limitation or use a robust normalization (e.g., interquartile range).
- [General] No data or code availability statement is provided. Given the emphasis on reproducibility and the use of publicly available pretrained models, please add a statement on where the sensor data and evaluation code can be accessed.
Circularity Check
No significant circularity: the evaluation is self-contained and uses external pretrained models with no fitted parameters.
full rationale
The paper's derivation chain is an empirical evaluation, not a mathematical derivation. The two models (TimesFM, Chronos-2) are external pretrained black boxes; no parameter is fitted to the SAIL2025 data. All metrics (FHC, NHC, SLT, NMPIW, SWS) are computed directly from model outputs and observed ground truth, and they are standard or clearly defined in the paper. The 80% interval (alpha=0.2) is a fixed, standard choice, not tuned to improve the reported results. The operational-reliability conclusion ('SLT reaches about 20 steps ahead (30 minutes)') is a factual report of the authors' own Stable Lead Time metric, not a derivation of that metric from something that already assumes the conclusion. No prediction is statistically forced by a fitted input. The paper's self-citations (e.g., refs. [2], [7] include a co-author) are background related-work citations and are not load-bearing evidence for the central claim. There is no uniqueness theorem imported from the authors' prior work, no ansatz smuggled in via citation, and no renaming of a known result as organization. The absence of a persistence or seasonal-naive baseline is an experimental-design limitation—it weakens the external validity and informativeness of the 'operationally reliable ~30 minutes' claim—but it is not a circularity reduction: nothing in the paper's equations or citations makes the conclusion equivalent to its inputs. Under the hard rules, a missing comparison is not a circularity step. Therefore no circular steps are identified and the score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption The 10th and 90th percentile predictions from the pretrained models are used at face value as an 80% prediction interval, with no post-hoc calibration or adjustment.
- domain assumption Missing count values are imputed with zeros, and this imputation does not systematically bias the evaluation.
- domain assumption The 5-day SAIL2025 event is representative of short special events for which the reliability conclusions are drawn.
- domain assumption The 90-second sampling interval and the 40-step (1-hour) prediction horizon are appropriate for operational crowd-management decisions.
Cite this review
Pith. "Pith review of Towards Reliable Zero-Shot Crowd Forecasting: Evaluating Time Series Foundation Models for Special Event Pedestrian Forecasting." pith.science (2026). https://pith.science/paper/AHR3O2BI
@misc{pith2026260717758,
author = {Pith},
title = {Pith review of: Towards Reliable Zero-Shot Crowd Forecasting: Evaluating Time Series Foundation Models for Special Event Pedestrian Forecasting},
year = {2026},
howpublished = {\url{https://pith.science/paper/AHR3O2BI}},
note = {Machine review of arXiv:2607.17758}
}
read the original abstract
Managing massive crowds during infrequent special events requires reliable real-time pedestrian-flow forecasting to ensure public safety and operational efficiency. However, supervised forecasting methods face limitations in these contexts due to scarce historical data, heterogeneous data distributions, and short in-event observation windows. To effectively support operational decision-making, forecasts should provide not only accurate point estimates but also informative predictive uncertainty. Probabilistic uncertainty quantification plays a critical role in this aspect, particularly capturing sudden volatility and tail risks. This paper investigates pretrained time series foundation models as a lightweight approach for zero-shot probabilistic forecasting without extensive local retraining. Using decision-oriented metrics tailored to short events, we conduct a comprehensive assessment of two time series foundation models on crowd forecasting, with the SAIL2025 event as a use case. We then distill practical insights for crowd managers, specifying when zero-shot forecasts remain operationally reliable.
Figures
Reference graph
Works this paper leans on
-
[1]
Ai for crisis decisions,
T. Comes, “Ai for crisis decisions,”Ethics and Information Technology, vol. 26, no. 1, p. 12, 2024
2024
-
[2]
Comparison of three algorithms for real-time pedestrian state estimation-supporting a monitoring dashboard for large-scale events,
Y . Yuan, W. Daamen, D. Duives, and S. Hoogendoorn, “Comparison of three algorithms for real-time pedestrian state estimation-supporting a monitoring dashboard for large-scale events,” in2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2016, pp. 2601–2606
2016
-
[3]
Strictly proper scoring rules, prediction, and estimation,
T. Gneiting and A. E. Raftery, “Strictly proper scoring rules, prediction, and estimation,”Journal of the American statistical Association, vol. 102, no. 477, pp. 359–378, 2007
2007
-
[4]
Probabilistic forecasting,
T. Gneiting and M. Katzfuss, “Probabilistic forecasting,”Annual Review of Statistics and Its Application, vol. 1, no. 1, pp. 125–151, 2014
2014
-
[5]
Robust probabilistic time series forecasting,
T. Yoon, Y . Park, E. K. Ryu, and Y . Wang, “Robust probabilistic time series forecasting,” inInternational Conference on Artificial Intelligence and Statistics. PMLR, 2022, pp. 1336–1358
2022
-
[6]
Traffic flow prediction with big data: A deep learning approach,
Y . Lv, Y . Duan, W. Kang, Z. Li, and F.-Y . Wang, “Traffic flow prediction with big data: A deep learning approach,”Ieee transactions on intelligent transportation systems, vol. 16, no. 2, pp. 865–873, 2014
2014
-
[7]
Dynamic spatial- temporal graph convolutional neural networks approach for active mode traffic prediction,
X. Wen, P. Krishnakumari, and S. Hoogendoorn, “Dynamic spatial- temporal graph convolutional neural networks approach for active mode traffic prediction,”IEEE Transactions on Intelligent Transportation Systems, 2025
2025
-
[8]
A decoder-only foundation model for time-series forecasting,
A. Das, W. Kong, R. Sen, and Y . Zhou, “A decoder-only foundation model for time-series forecasting,” inForty-first international conference on machine learning, 2024
2024
Show all 32 references
-
[9]
Chronos-2: From univariate to universal forecasting,
A. F. Ansari, O. Shchur, J. K ¨uken, A. Auer, B. Han, P. Mercado, S. S. Rangapuram, H. Shen, L. Stella, X. Zhanget al., “Chronos-2: From univariate to universal forecasting,”arXiv preprint arXiv:2510.15821, 2025
2025 arXiv
-
[10]
Frequency enhanced pre-training for cross-city few-shot traffic forecasting,
Z. Liu, J. Ding, and G. Zheng, “Frequency enhanced pre-training for cross-city few-shot traffic forecasting,” inJoint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 2024, pp. 35–52
2024
-
[11]
Communicating disaster risk,
F. Boersma, D. Allen, T. Comes, I. Stanciugelu, and T. Terpstra, “Communicating disaster risk,” inScience for disaster risk management 2017: Knowing better and losing less. Publications Office of the European Union/EC Joint Research Centre, 2017, pp. 388–440
2017
-
[12]
Deepar: Probabilistic forecasting with autoregressive recurrent networks,
D. Salinas, V . Flunkert, J. Gasthaus, and T. Januschowski, “Deepar: Probabilistic forecasting with autoregressive recurrent networks,”Inter- national journal of forecasting, vol. 36, no. 3, pp. 1181–1191, 2020
2020
-
[13]
N-beats: Neural basis expansion analysis for interpretable time series forecasting,
B. N. Oreshkin, D. Carpov, N. Chapados, and Y . Bengio, “N-beats: Neural basis expansion analysis for interpretable time series forecasting,” arXiv preprint arXiv:1905.10437, 2019
1905 arXiv
-
[14]
Are transformers effective for time series forecasting?
A. Zeng, M. Chen, L. Zhang, and Q. Xu, “Are transformers effective for time series forecasting?” inProceedings of the AAAI conference on artificial intelligence, vol. 37, no. 9, 2023, pp. 11 121–11 128
2023
-
[15]
Deep learning models for time series forecasting: A review,
W. Li and K. E. Law, “Deep learning models for time series forecasting: A review,”IEEE Access, vol. 12, pp. 92 306–92 327, 2024
2024
-
[16]
Quantileformer: Probabilistic time series forecasting with a pattern-mixture decomposed vae transformer,
Y . Shao, W. Li, K. Xia, K. Lin, M. Lin, and S. Lu, “Quantileformer: Probabilistic time series forecasting with a pattern-mixture decomposed vae transformer,” inProceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, 2025, pp. 6147–6155
2025
-
[17]
Conformal prediction for time-series forecasting with change points,
S. H. Sun and R. Yu, “Conformal prediction for time-series forecasting with change points,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems
-
[18]
Neural conformal control for time series forecasting,
R. Li and A. Rodr ´ıguez, “Neural conformal control for time series forecasting,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 17, 2025, pp. 18 439–18 447
2025
-
[19]
Exploring the limits of transfer learning with a unified text-to-text transformer,
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020
2020
-
[20]
Large language models are zero-shot time series forecasters,
N. Gruver, M. Finzi, S. Qiu, and A. G. Wilson, “Large language models are zero-shot time series forecasters,”Advances in neural information processing systems, vol. 36, pp. 19 622–19 635, 2023
2023
-
[21]
Experienced travel time prediction for congested freeways,
M. Yildirimoglu and N. Geroliminis, “Experienced travel time prediction for congested freeways,”Transportation Research Part B: Methodolog- ical, vol. 53, pp. 45–63, 2013
2013
-
[22]
Public transit for special events: Ridership prediction and train scheduling,
T. Santanam, A. J. Trasatti, P. Van Hentenryck, and H. Zhang, “Public transit for special events: Ridership prediction and train scheduling,” IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 8, pp. 8387–8403, 2024
2024
-
[23]
Compre- hensive review of neural network-based prediction intervals and new advances,
A. Khosravi, S. Nahavandi, D. Creighton, and A. F. Atiya, “Compre- hensive review of neural network-based prediction intervals and new advances,”IEEE Transactions on neural networks, vol. 22, no. 9, pp. 1341–1356, 2011
2011
-
[24]
A decision-theoretic approach to interval estimation,
R. L. Winkler, “A decision-theoretic approach to interval estimation,” Journal of the American Statistical Association, vol. 67, no. 337, pp. 187–191, 1972
1972
-
[25]
A gat-bilstma model for weather-aware prediction of traffic speed,
B. A. Muhammed, A. R. Hurson, S. S. Sarvestani, and L. Gamage, “A gat-bilstma model for weather-aware prediction of traffic speed,” in 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2024, pp. 2117–2124
2024
-
[26]
Metro ridership forecasting using inter-station-aware transformer networks,
K. Saleh, A.-S. Mihaita, and Y . Ou, “Metro ridership forecasting using inter-station-aware transformer networks,” in2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2023, pp. 1215–1220
2023
-
[27]
Short-term passenger flow prediction under passenger flow control using a dynamic radial basis function network,
H. Li, Y . Wang, X. Xu, L. Qin, and H. Zhang, “Short-term passenger flow prediction under passenger flow control using a dynamic radial basis function network,”Applied Soft Computing, vol. 83, p. 105620, 2019
2019
-
[28]
Bi-level model predictive control for metro networks: Integration of timetables, passenger flows, and train speed profiles,
X. Liu, A. Dabiri, J. Xun, and B. De Schutter, “Bi-level model predictive control for metro networks: Integration of timetables, passenger flows, and train speed profiles,”Transportation Research Part E: Logistics and Transportation Review, vol. 180, p. 103339, 2023
2023
-
[29]
A time series is worth 64 words: Long-term forecasting with transformers,
Y . Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, “A time series is worth 64 words: Long-term forecasting with transformers,”arXiv preprint arXiv:2211.14730, 2022
2022 arXiv
-
[30]
Deformabletst: Transformer for time series forecasting without over-reliance on patching,
D. Luo and X. Wang, “Deformabletst: Transformer for time series forecasting without over-reliance on patching,”Advances in Neural Information Processing Systems, vol. 37, pp. 88 003–88 044, 2024
2024
-
[31]
Learning pattern-specific experts for time series forecasting under patch-level distribution shift,
Y . Sun, Z. Xie, E. Eldele, D. Chen, Q. Hu, and M. Wu, “Learning pattern-specific experts for time series forecasting under patch-level distribution shift,”arXiv preprint arXiv:2410.09836, 2024
2024
-
[32]
Mlperf inference benchmark,
V . J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu, B. Anderson, M. Breughe, M. Charlebois, W. Chouet al., “Mlperf inference benchmark,” in2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 2020, pp. 446– 459
2020
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.