REVIEW 4 major objections 6 minor 39 references
Air Quality Arena: A Large-Scale Multi-Region Ground Monitoring Dataset and Benchmark for Air Quality Forecasting with Time-Series Foundation Models
T0 review · 4 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read This paper claims that pretrained time-series foundation models, used with zero fine-tuning, beat statistical and per-station supervised baselines at 24-hour air-quality forecasting, with the top model VisionTS++ ranking first across every
desk verdict A genuinely useful multi-country air-quality benchmark with a credible zero-shot TSFM comparison, but the abstract overclaims consistency and the contamination audit is too thin to fully verify the top-model ranking. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the AQA-Data benchmark itself: 14,139 station–pollutant hourly series drawn from six national monitoring networks (EPA, CPCB, AURN, CNEMC, EEA for France/Germany, SINAICA), restricted to stations with at least 70% valid readings and no gaps over two weeks, with remaining gaps imputed in deseasonalized space via MSTL. Evaluation uses a 168-hour context window predicting the next 24 hours on a rolling 24-hour step, with scores aggregated as the geometric mean over networks of pollutant-balanced MASE and CRPS normalized by Seasonal Naive. The mechanism that carries the argument is the zero-shot setting itself: foundation models are applied with frozen weights, so any
What would settle it
A concrete check: hash the timestamp, site, and pollutant triples of AQA-Data's test-window readings and search the released pretraining corpora of the evaluated TSFMs (Chronos, Moirai, TimesFM, TiRex, VisionTS++) for exact matches; any match in the evaluation window would directly refute the zero-shot interpretation, and even near-duplicate sequences could be flagged. As a behavioral check, an independent team could train a supervised baseline on the full three years of AQA-Data and show it beats the best zero-shot TSFM, contradicting the paper's asymmetry.
Extended reading notes
Core claim
On its own data, the paper establishes that the best pretrained time-series foundation model, VisionTS++, achieves the lowest normalized MASE (0.7785) and second-lowest CRPS (0.4537) on the pollutant-balanced leaderboard, ahead of all eleven TSFMs and six classical baselines; the top TSFMs as a group beat the best supervised baseline (PatchTST, 0.8300 MASE) and all statistical baselines. The comparison is deliberately asymmetric: TSFMs see no air-quality training data, while supervised baselines are trained per pollutant per network on the first year of the same stations. The finding holds across all seven networks and all six pollutants, with the gaps to statistical baselines largest on hig
Load-bearing premise
The paper's zero-shot claim rests on the assumption that the pretrained models never saw the July 2022–June 2025 test-period readings; the authors verified this only for the public portions of a few pretraining corpora and for a subset of overlapping stations, so the check is not airtight.
Editorial extensions
If this is right
- If the result holds, a single pretrained model can be dropped into any new monitoring station and produce 24-hour forecasts competitive with station-specific trained models, which is useful where historical data is sparse or unreliable.
- The benchmark provides a harder, more diverse test for TSFMs than existing single-region or single-pollutant datasets; future model improvements are likely to be measured on high-pollution networks where current MASE still hovers near 0.8–0.9.
- Since VisionTS++ leads on every network and on CO, NO2, PM10, and PM2.5, cross-modal architectures (image-based time-series encoders) become a plausible direction for environmental forecasting research.
- The finding that supervised baselines (DLinear, LightGBM) exceed MASE 1.0 on SO2 and on CPCB suggests that on these series, even target-trained models fail to beat naive seasonality — indicating intrinsic predictability limits rather than model immaturity.
Reading between the lines
- If zero-shot transfer is genuine, an immediate testable extension is few-shot adaptation: fine-tuning the top TSFMs on a few weeks of CPCB data could show whether the remaining gap to perfect forecasting closes faster than training supervised models from scratch — this would separate memorization from transfer.
- The cross-modal advantage invites a control experiment the paper does not run: train an equivalent pure-temporal transformer on the same image-rendered data (or ablate VisionTS++'s vision backbone) to confirm that the vision backbone, not the rendering or patch size, is responsible for the gain.
- The benchmark's fixed 168-hour context and 24-hour horizon may understate or overstate TSFM strengths; testing longer horizons (48–72h) and shorter contexts could reveal whether the zero-shot edge is specific to the 24-hour horizon.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Air Quality Arena (AQA), a large-scale multi-country, multi-pollutant dataset of hourly ground-monitoring air quality measurements, together with a benchmarking harness AQA-Bench. AQA-Data covers 14,139 station-pollutant series across seven countries and six pollutants over July 2022–June 2025. The benchmark evaluates 11 time-series foundation models (TSFMs) in a zero-shot setting against statistical, tree-based, and supervised deep-learning baselines for 24-hour forecasting from a 168-hour context. The headline claim is that TSFMs are effective zero-shot forecasters and consistently outperform classical baselines, with VisionTS++, a cross-modal vision-backed model, achieving the best normalized MASE. The paper also provides per-network and per-pollutant results, a preprocessing pipeline based on MSTL imputation, and a public release.
Significance. If the central claim holds, this is a useful contribution: it provides the largest geographically diverse air-quality forecasting benchmark to date, evaluates modern TSFMs in a realistic zero-shot regime, and identifies a promising cross-modal architecture. The public dataset and open evaluation harness are concrete assets that the community can build on. The paper is appropriately cautious in some places, such as noting high-pollution networks remain hardest. However, the headline claim is currently stronger than the evidence in the paper, and the zero-shot interpretation rests on a contamination audit that is not reproducible from the manuscript. The dataset itself is valuable regardless of the exact ranking, but the paper's scientific contribution is tied to the empirical comparison, so the audit and the wording of the central claim need to be fixed.
major comments (4)
- [Abstract; §4.1, Table 1] The abstract and Section 4.1 state that TSFMs 'consistently outperform classical baselines.' Table 1 directly contradicts this: Kairos, a TSFM, has normalized MASE 1.0121, worse than Seasonal Naive's 1.0000, and the text itself later says Kairos is 'the only model to fall below Seasonal Naive.' The data support a weaker statement such as 'most TSFMs' or 'top-performing TSFMs.' Since this is the paper's headline claim, the wording should be corrected, and ideally supplemented with a count of station-pollutant pairs where TSFMs beat the statistical baselines.
- [§3.1] The contamination assessment is load-bearing for the zero-shot claim but not verifiable. The paper states 'we found no temporal intersection between the pretraining data and AQA-Data' but gives no method, no per-model list of pretraining corpora, and no query procedure. Most evaluated TSFMs, including the top-performing VisionTS++, do not have fully public pretraining corpora. Because the headline per-country ranking in Table 48 includes CNEMC and AURN, where station-level overlap is admitted, a reader cannot rule out that part of the reported advantage comes from memorization rather than zero-shot generalization. Please provide a reproducible audit: per-model corpus provenance, date ranges, the exact overlap-check method, and a sensitivity analysis excluding all stations/networks with any overlap.
- [§3.2; Appendix B] The preprocessing pipeline imputes missing values with MSTL before the chronological train/test split described in Appendix B. MSTL decomposes the full three-year series, so an imputed value in a context window can depend on observations from the evaluation period. This is temporal leakage and can inflate reported accuracy on windows containing imputed points, for all model classes. Since the benchmark claims to measure zero-shot forecasting, the imputation should be restricted to training-period data only, or the authors should show that the results are unchanged when evaluation windows containing imputed values are excluded.
- [Table 1; §4.1; Appendix F] All results are point estimates without uncertainty quantification. The margin between VisionTS++ (0.7785) and TiRex (0.7825) is 0.004, and TiRex to TimesFM-2.5 (0.7831) is 0.0006, which is likely within cross-window and cross-site variation. The claims that VisionTS++ 'holds the top spot on each individual country' and that cross-modal pretraining is the reason for its success need bootstrap confidence intervals, per-network rank distributions, or a significance test. Without these, the top-model conclusion could be noise.
minor comments (6)
- [Appendix C] The degenerate-site exclusion rule uses a mean MASE/CRPS threshold of 50 averaged over all models. This is a data-dependent filter applied per pollutant and is not shown to be robust. Please report how the overall leaderboard changes if these sites are retained or if the threshold is varied.
- [§4.1; Table 48] 'VisionTS++ holds the top spot on each individual country' is imprecise: the evaluation is at the network level, and EEA covers two countries. The supporting table is per network, not per country.
- [Table 3; Table 5; Table 29] AURN CO has only 1 site and EEA-FR CO has only 5 sites. These entries are effectively single-station or tiny-sample results and should be flagged in the main text or excluded from aggregate leaderboards, not only in Appendix A.
- [Abstract; §1] The abstract says 'six monitoring networks, seven countries' but Table 2 lists seven networks (with EEA split as EEA-FR and EEA-DE) and seven countries. Clarify whether EEA is one network or two.
- [§3.2; Table 1] The metric normalization is only defined in the text ('normalized by Seasonal Naive'); state explicitly in the Table 1 caption that MASE and CRPS are divided by the Seasonal Naive values, and that Seasonal Naive therefore has value 1.0000 by construction.
- [Abstract] Typo: 'atAirQualityArena.github.io' is missing a space before 'at'.
Circularity Check
No significant circularity: this is an empirical benchmark that compares zero-shot TSFM forecasts against fitted baselines; no fitted parameter or self-citation is repackaged as a prediction.
full rationale
The paper is a dataset-and-benchmark contribution rather than a derivation. Its central claim—that zero-shot TSFMs outperform classical and supervised baselines—is supported by direct evaluation on a fixed chronological test split: TSFM checkpoints are applied without weight updates, while ML baselines are trained on the training split and statistical baselines are fitted per test window. MASE and CRPS are computed against held-out observations and normalized by Seasonal Naive, which serves as a reference baseline rather than as a fitted constant later relabeled as a finding. The aggregation procedure (mean over windows, then series, then pollutants, then geometric mean over networks) is transparent and standard and does not construct the result. No load-bearing self-citations appear: references to TIME, Gift-Eval, model papers, and toolkits are external prior work, and the authors' own prior publications are not used to justify the benchmark's conclusions. The Section 3.1 contamination audit is indeed under-specified—no method is given for checking temporal intersection and most pretraining corpora are not fully public—but that is a reproducibility and validity caveat about potential leakage, not a circular step in which the output is equivalent to an input by construction. The remaining choices, such as excluding sites with degenerate MASE/CRPS, are standard benchmark hygiene and do not force the reported ranking. Therefore the appropriate finding is no significant circularity (score 0).
Assumptions & free parameters
free parameters (5)
- Data completeness threshold (70%)
- Maximum gap length (2 weeks)
- Degenerate-site exclusion threshold (mean MASE/CRPS > 50) =
50
- CNEMC subsample size (200 sites per pollutant) =
200
- Context window, horizon, step (168h, 24h, 24h)
assumptions (4)
- domain assumption Official national monitoring networks provide reliable ground-truth pollutant concentrations.
- domain assumption Filtering to ≥70% completeness and ≤2-week gaps, then MSTL imputation, preserves the forecasting signal.
- domain assumption Pretraining contamination is negligible because overlap with AQA-Data is limited to CNEMC/AURN stations with no temporal intersection.
- domain assumption Excluding sites with degenerate MASE/CRPS does not bias the model comparison.
Cite this review
Pith. "Pith review of Air Quality Arena: A Large-Scale Multi-Region Ground Monitoring Dataset and Benchmark for Air Quality Forecasting with Time-Series Foundation Models." pith.science (2026). https://pith.science/paper/KI6CQV6L
@misc{pith2026260719381,
author = {Pith},
title = {Pith review of: Air Quality Arena: A Large-Scale Multi-Region Ground Monitoring Dataset and Benchmark for Air Quality Forecasting with Time-Series Foundation Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/KI6CQV6L}},
note = {Machine review of arXiv:2607.19381}
}
read the original abstract
Air pollution causes an estimated 7.9 million premature deaths annually, making accurate forecasting a critical public health priority. Machine learning is increasingly being applied to forecast air pollution levels, yet existing benchmarks remain narrow in both geographic scope and pollutant coverage, and fail to evaluate the latest generation of time series foundation models (TSFMs) on real world, large scale data. We present Air Quality Arena (AQA), a large scale multi-country and multi-pollutant dataset (AQA-Data) and benchmark (AQA-Bench) to address this gap. AQA covers 6 major pollutants over a three year period across 7 diverse countries and 4 continents, with more than 14,000 station-pollutant series, aiming to provide a comprehensive benchmark for air quality tasks. We benchmark this dataset across 11 leading time series foundation models and classical baselines to assess performance on short-term air quality forecasting. Our results demonstrate that TSFMs are effective zero-shot forecasters and consistently outperform classical baselines, with our top-performing model employing a cross-modal architecture that leverages a vision foundation model for time series forecasting. AQA is publicly released at AirQualityArena.github.io
Figures
Reference graph
Works this paper leans on
-
[1]
Analysis of Air Pollution Data in India between 2015 and 2019 , volume=
Sharma, Disha and Mauzerall, Denise , year=. Analysis of Air Pollution Data in India between 2015 and 2019 , volume=. Aerosol and Air Quality Research , publisher=. doi:10.4209/aaqr.210204 , number=
-
[2]
and Chen, Yue and Arnold, Steve R
Silver, Ben and Reddington, Carly L. and Chen, Yue and Arnold, Steve R. , journal =. A decade of. 2025 , doi =
2025
-
[3]
NeurIPS 2025 Workshop on Tackling Climate Change with Machine Learning , url=
Using Time Series Foundation Models for Atmospheric CO2 Concentration Forecasting , author=. NeurIPS 2025 Workshop on Tackling Climate Change with Machine Learning , url=
2025
-
[4]
Continuous Ambient Air Quality Monitoring (
-
[5]
Carslaw and Karl Ropkins , keywords =
David C. Carslaw and Karl Ropkins , keywords =. openair — An R package for air quality data analysis , journal =. 2012 , issn =. doi:https://doi.org/10.1016/j.envsoft.2011.09.008 , url =
-
[6]
China Air Quality Historical Data , author =
-
[7]
Air Quality Download Service , author =
-
[8]
Wilson, Robin , url =
Show all 39 references
-
[9]
Automatic Urban and Rural Network (
-
[10]
International Journal on Recent and Innovation Trends in Computing and Communication , volume =
Comparison of Imputation Methods for Univariate Time Series , author =. International Journal on Recent and Innovation Trends in Computing and Communication , volume =. 2023 , doi =
2023
-
[11]
Atmosphere , volume =
Mind the Large Gap: Novel Algorithm Using Seasonal Decomposition and Elastic Net Regression to Impute Large Intervals of Missing Data in Air Quality Data , author =. Atmosphere , volume =. 2023 , doi =
2023
-
[12]
and Cleveland, William S
Cleveland, Robert B. and Cleveland, William S. and McRae, Jean E. and Terpenning, Irma , title =. Journal of Official Statistics , volume =. 1990 , publisher =
1990
-
[13]
Olivares , title =
Azul Garza and Max Mergenthaler Canseco and Cristian Challú and Kin G. Olivares , title =. 2022 , howpublished=
2022
-
[14]
2024 , eprint=
GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation , author=. 2024 , eprint=
2024
-
[15]
arXiv preprint arXiv:2602.12147 , year=
It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks , author=. arXiv preprint arXiv:2602.12147 , year=
-
[16]
OpenAQ , howpublished =
-
[17]
World Air Quality Index , howpublished =
-
[18]
and Stomberg, T
Betancourt, C. and Stomberg, T. and Roscher, R. and Schultz, M. G. and Stadtler, S. , TITLE =. Earth System Science Data , VOLUME =. 2021 , NUMBER =
2021
-
[19]
arXiv preprint arXiv:2511.11698 , year=
Moirai 2.0: When less is more for time series forecasting , author=. arXiv preprint arXiv:2511.11698 , year=
-
[20]
Forty-first International Conference on Machine Learning , year=
Unified Training of Universal Time Series Forecasting Transformers , author=. Forty-first International Conference on Machine Learning , year=
-
[21]
Transactions on Machine Learning Research , issn=
Chronos: Learning the Language of Time Series , author=. Transactions on Machine Learning Research , issn=. 2024 , url=
2024
-
[22]
arXiv preprint arXiv:2510.15821 , year=
Chronos-2: From univariate to universal forecasting , author=. arXiv preprint arXiv:2510.15821 , year=
-
[23]
arXiv preprint arXiv:2508.04379 , year=
VisionTS++: Cross-Modal Time Series Foundation Model with Continual Pre-trained Vision Backbones , author=. arXiv preprint arXiv:2508.04379 , year=
-
[24]
The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=
TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=
-
[25]
Forty-second International Conference on Machine Learning , year=
Sundial: A Family of Highly Capable Time Series Foundation Models , author=. Forty-second International Conference on Machine Learning , year=
-
[26]
2026 , eprint=
Kairos: Toward Adaptive and Parameter-Efficient Time Series Foundation Models , author=. 2026 , eprint=
2026
-
[27]
2024 , eprint=
A decoder-only foundation model for time-series forecasting , author=. 2024 , eprint=
2024
-
[28]
State of Global Air 2025: A Report on Air Pollution and Its Role in the World's Leading Causes of Death , year =
2025
-
[29]
Air Pollution , howpublished =
-
[30]
Effective PM2.5 concentration forecasting based on multiple spatial-temporal GNN for areas without monitoring stations , journal =
I-Fang Su and Yu-Chi Chung and Chiang Lee and Pin-Man Huang , keywords =. Effective PM2.5 concentration forecasting based on multiple spatial-temporal GNN for areas without monitoring stations , journal =. 2023 , issn =. doi:https://doi.org/10.1016/j.eswa.2023.121074 , url =
2023
-
[31]
AI-based air quality PM2.5 forecasting models for developing countries: A case study of Ho Chi Minh City, Vietnam , journal =
Rajnish Rakholia and Quan Le and Khue Vu and Bang Quoc Ho and Ricardo Simon Carbajo , keywords =. AI-based air quality PM2.5 forecasting models for developing countries: A case study of Ho Chi Minh City, Vietnam , journal =. 2022 , issn =. doi:https://doi.org/10.1016/j.uclim.2...
2022
-
[32]
Zeng, Ailing and Chen, Muxi and Zhang, Lei and Xu, Qiang , title =. Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances i...
2023 doi
-
[33]
The Eleventh International Conference on Learning Representations , year=
A Time Series is Worth 64 Words: Long-term Forecasting with Transformers , author=. The Eleventh International Conference on Learning Representations , year=
-
[34]
Forecasting: Principles and Practice
Hyndman, \ Robin John\ and George Athanasopoulos. Forecasting: Principles and Practice. 2018
2018
-
[35]
2008 , publisher=
Forecasting with Exponential Smoothing: The State Space Approach , author=. 2008 , publisher=
2008
-
[36]
arXiv preprint arXiv:2003.06505 , year=
AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data , author=. arXiv preprint arXiv:2003.06505 , year=
2003 arXiv
-
[37]
Shchur, Oleksandr and Turkmen, Caner and Erickson, Nick and Shen, Huibin and Shirkov, Alexander and Hu, Tony and Wang, Yuyang , booktitle=
-
[38]
Proceedings of the 31st International Conference on Neural Information Processing Systems , pages =
Ke, Guolin and Meng, Qi and Finley, Thomas and Wang, Taifeng and Chen, Wei and Ma, Weidong and Ye, Qiwei and Liu, Tie-Yan , title =. Proceedings of the 31st International Conference on Neural Information Processing Systems , pages =. 2017 , isbn =
2017
-
[39]
DeepAR: Probabilistic forecasting with autoregressive recurrent networks , journal =
David Salinas and Valentin Flunkert and Jan Gasthaus and Tim Januschowski , keywords =. DeepAR: Probabilistic forecasting with autoregressive recurrent networks , journal =. 2020 , issn =. doi:https://doi.org/10.1016/j.ijforecast.2019.07.001 , url =
2020 doi
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.