{"id":"f46fb8c4-b1c9-44bc-ba42-dec6557c8824","arxiv_id":"2505.01415","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Chronos, a pretrained foundation model used zero-shot, beat 16 other models at forecasting Everglades water levels across five stations and horizons from 7 to 28 days.","lead":"This study compared 17 machine learning models for forecasting water levels in the Florida Everglades. The foundation model Chronos, used without fine-tuning, had the lowest errors at five stations and all forecast horizons.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Chronos's headline win is not attributable until the paper specifies what inputs the foundation models actually received: if task-specific models saw 37 variables and Chronos saw only the target series, the comparison conflates model class with input information.","rationale":"The reader's weakest assumption points to the same core ambiguity in Section 3.1, so I agree with the overall concern. I mark agreement as partial because the reader states that all five foundation models are univariate; that is not exactly right, since Moirai is any-variate and TimeGPT can accept exogenous features, which makes the missing input specification even harder to fill in rather than easier. I did not build the critique on the SEDI formula in Eq. (4) or the absence of significance tests: those are real but secondary, because the MAE/RMSE ranking that carries RQ1 stands independently of them. The input-feature ambiguity is load-bearing because it determines whether the reported Chronos advantage is a property of the pretrained model or of the modeling pipeline, for instance avoiding overfit to 37 noisy, collinear features on 989 training days. The paper deserves credit for releasing code and data, which makes the proposed matched-input ablation feasible; until such a check is run, a conditional verdict is the right call.","tokens_in":19322,"tokens_out":14978,"duration_ms":157985,"concrete_test":"Using the released repository, run a matched-input ablation on the same train/validation/test split: retrain NBEATS, PatchTST, TSMixer, and RMoK on the univariate target-station water-level series only, with the same 100-day window, the same 7/14/21/28-day horizons, and the same early-stopping rule, then compare their MAE/RMSE to the Chronos column of Table 1. Also run Moirai, which is any-variate, with the full 37-variable matrix if the library supports it. If the univariate task-specific models match or beat Chronos, the Table 1 ranking is an artifact of the input mismatch; if Chronos still wins on both matched variants, the input ambiguity is secondary and the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 says task-specific models take \"all variables from the previous 100 days\" and that foundation models run \"with the same setting\" in zero-shot inference. The paper never states which variables the foundation models received. Chronos and TimesFM are univariate predictors, and the description of Timer's S3 format also implies single-series input; Moirai is any-variate and TimeGPT can accept exogenous features, so the actual input format is architecture-dependent and never specified. Table 4 lists only input lengths (100 days, except TimesFM at 128), not input features. If Chronos saw only the target station's water-level history while NBEATS, PatchTST, and the other task-specific models saw all 37 variables, the benchmark confounds model class with input information. The direction is not simple: the extra covariates could help task-specific models or, on 989 training days, be overfit and hurt them. Either way, the reported margin, e.g., lead-28 overall MAE 0.088 for Chronos versus 0.176 for NBEATS, cannot be attributed to pretraining versus input specification from the text alone, and the RQ1 claim is not reproducible as written. If Chronos wins with strictly less information, the conclusion would be stronger, but that is a different claim from the one the paper currently documents.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript evaluates 12 task-specific deep learning time-series models and 5 pretrained time-series foundation models for daily water-level forecasting at five Everglades stations, with lead times of 7, 14, 21, and 28 days and a nominal 100-day input window. Using MAE, RMSE, and a metric the paper calls SEDI, it reports that the zero-shot foundation model Chronos achieves the lowest MAE and RMSE at all stations and lead times and the highest extreme-event scores, while most other foundation models underperform and task-specific models vary by architecture. The paper also examines input-length sensitivity, model size versus accuracy/efficiency, and possible reasons for Chronos's success, and it provides a GitHub repository for code and data.","tokens_in":19572,"tokens_out":7551,"duration_ms":74407,"significance":"If the central claim holds, the paper would be a practically useful benchmark showing that a general pretrained time-series model can outperform task-specific models trained on hydrologic data in zero-shot water-level forecasting. The study covers a real-world dataset, a large model zoo, multiple lead times, and an explicit attempt to evaluate extreme events, and the authors have made code and data available. However, the comparison is not fully specified as an experiment (univariate versus multivariate inputs are not documented), the extreme-event metric is misidentified, and the word 'significantly' is used without statistical support. These issues must be resolved before the central ranking claim can be accepted as stated.","major_comments":[{"comment":"The experimental protocol for foundation models is not specified. Section 3.1 says task-specific models use 'all variables from the previous 100 days' and foundation models run 'with the same setting' in zero-shot inference, but Chronos, TimesFM, Timer, and TimeGPT are univariate predictors, so they cannot receive the 37-channel matrix X_{t-w+1:t} in Eq. (1). The paper never states which single series is fed to each foundation model, nor whether Moirai receives a flattened multivariate input. Consequently, Table 1's comparison (e.g., lead-28 Overall MAE 0.088 for Chronos versus 0.176 for NBEATS) may conflate model-class differences with input-information differences, and the RQ1 claim is not reproducible as written. Please state the exact input features for every foundation model and, ideally, add a controlled variant: task-specific models trained on the same univariate target history, or a multivariate-capable foundation model given the same 37 variables.","section":"§3.1 and Table 4"},{"comment":"Equation (4) is not the Symmetric Extremal Dependence Index (SEDI); it is the pooled hit rate for correctly predicted upper and lower extremes divided by the observed number of extreme events. SEDI is a function of both hit rate and false-alarm rate, not this ratio. Therefore the values in Table 2 labeled 'SEDI' do not measure what the paper claims, and the RQ3 conclusion that 'Chronos significantly outperforms' on extremes is not supported by the reported statistic. Please replace Eq. (4) with the correct SEDI definition and report the associated hit and false-alarm rates, or rename the metric and qualify the extreme-value conclusions accordingly.","section":"§3.1, Eq. (4), Table 2"},{"comment":"The word 'significantly' is used repeatedly to describe Chronos's advantage, but the paper reports only point estimates and the reproducibility checklist's claim that 'the significance of any improvement ... is judged using appropriate statistical tests' is not backed by any reported test, confidence interval, or run-to-run variation. With 211 test days and five stations, a paired Diebold-Mariano test or a bootstrap over station/lead-time blocks would be feasible and would justify the wording. At minimum, 'significantly outperforms' should be replaced with 'achieves lower point estimates' unless a statistical test is provided.","section":"§4 (RQ1, RQ3), Abstract"},{"comment":"The Overall-column values for TSMixer at lead time 14 appear inconsistent with the station-level values. The five station MAEs are 0.213, 0.100, 0.115, 0.089, and 0.082, whose mean is approximately 0.120, but the table reports Overall MAE 0.204; the reported overall RMSE of 0.181 is likewise hard to reconcile with the station RMSEs. Please define how the Overall column is computed (station-length weighting or per-sample pooling) and correct the entry if it is a typographical error. This matters because Overall values are used in the summary comparisons and the table is the central evidence for RQ1.","section":"Table 1, TSMixer row"}],"minor_comments":[{"comment":"TSMixerx is presented as a separate model and cited to Chen et al. 2023, but that reference introduces TSMixer; please provide the correct source for the 'x' variant.","section":"§2.2"},{"comment":"The notation yp_up and yp_low is undefined, and the sentence 'The tests, ŷ < yp_up and y < yp_up judge whether the predicted and observed values are extremes' is garbled; please rewrite the threshold definitions clearly.","section":"§3.1, Eq. (4)"},{"comment":"The checklist states that the paper reports the number of algorithm runs per result and lists all final hyperparameters, but the body gives no seeds or number of runs; Table 4 lists only learning rate, optimizer, and library/API. Please reconcile the checklist with the actual text.","section":"Reproducibility Checklist and Appendix F"},{"comment":"The text says 'We have also experimented with SVM and XGBoost, but the results were relatively poor,' yet no results for these models appear in the main text or tables; either include them in the evaluation or remove the claim.","section":"§5 (Related Work)"},{"comment":"The datum adjustment values (e.g., +1.51 ft for G620, +1.54 ft for S333T) are stated without a source or uncertainty; please document where these values come from and how they were applied to all stations.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a potentially useful empirical benchmark, but the input-feature ambiguity and the mislabeled SEDI are substantive. The reproducibility checklist also appears to overstate what is reported (statistical tests, run counts, seeds). I recommend asking for a corrected or clarified evaluation before further consideration; the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, useful benchmark for anyone working on water-level forecasting in South Florida, and the Chronos result is genuinely surprising. But the paper's central claim—that Chronos zero-shot 'significantly outperforms' all task-specific models—is not yet supported as written. The biggest problem is that the paper never says what inputs the foundation models actually received. Task-specific models get 37 variables (Section 3.1 says 'all variables from the previous 100 days'), while Chronos and the other foundation models are univariate. If Chronos saw only the target station's water-level history, the comparison mixes model class with information available. That could make the result stronger (Chronos winning with less data) or weaker (task-specific models handicapped by noisy covariates on 989 training days), but either way the reported margin cannot be attributed to pretraining versus input specification. The paper needs to state exactly which series were fed to each foundation model, and ideally run a controlled comparison where task-specific models also see only the target series, or foundation models see the full multivariate context when they can.\n\nSecond, Eq. (4) is not SEDI. It is a hit rate (or probability of detection) for extremes. SEDI is a much more specific score based on the extremal dependence index. Table 2 measures something different from what the text claims. That is easily fixed—either re-derive the metric or rename it—but as it stands the extreme-value conclusions are built on a mislabeled formula.\n\nThird, the word 'significantly' appears in the abstract and RQ1, but there are no error bars, no repeated runs, and no statistical tests. The differences at lead-28 overall (Chronos MAE 0.088 vs NBEATS 0.176) are large, so the ranking likely survives, but the claim needs at least some uncertainty quantification or a paired test.\n\nWhat is genuinely good: the study covers 17 models, includes five stations and four horizons, provides code and data, and the RQ5 discussion about Chronos pretraining data is a reasonable hypothesis rather than a circular explanation. The linear-based vs MLP vs KAN vs transformer breakdown is informative for practitioners. The paper is honestly framed as a case study, not a general theory.\n\nFor whom: hydrologists and applied ML people who want a quick map of what off-the-shelf models do on Everglades water levels. It deserves a serious referee—the benchmark is useful and the issues are addressable in revision. I would not desk-reject it, but I would ask for the input-feature specification, the corrected metric, and some notion of uncertainty before the headline claim is publishable.","headline":"Useful benchmark with a big zero-shot claim from Chronos, but the comparison may conflate model class with input information, and the statistics are thinner than the prose.","tokens_in":20107,"tokens_out":1840,"would_cite":false,"duration_ms":18049,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that Chronos, a pretrained time-series foundation model used in zero-shot mode, achieves the lowest MAE and RMSE at all five Everglades stations and all four lead times, plus the highest SEDI for extreme water levels.","keywords":["time series foundation models","hydrology","water level forecasting","Everglades","zero-shot inference","Chronos","extreme event prediction","benchmark study"],"falsifier":"Rerun the five-station, four-horizon benchmark under a controlled input ablation: give every model either the full 37-variable context or only the target station's water-level history, then compare Chronos with the best task-specific models on MAE, RMSE, and SEDI. If Chronos no longer leads at all stations and lead times in both settings, the paper's central claim that zero-shot Chronos is uniformly superior is not supported.","tokens_in":19139,"feed_emoji":"🌊","tokens_out":8909,"duration_ms":82932,"temperature":0.7,"pith_summary":"This paper asks whether pretrained foundation models for time series can forecast Everglades water levels without task-specific retraining, and reports that one of them, Chronos, does more than hold its own: it produces the lowest mean absolute error and root mean squared error at all five stations and all four lead times (7, 14, 21, and 28 days) studied. The same model also earns the highest Symmetric Extremal Dependence Index (SEDI) scores, meaning it is best at flagging extreme high and low water levels. The other four foundation models largely underperform, so the result is not that foundation models in general win, but that this particular pretrained model transfers well to a specialized hydrological setting. A sympathetic reading of the paper's proposed explanation is that Chronos's pretraining corpus included daily U.S. weather-station measurements whose patterns resemble Everglades hydrology. If correct, the result would let operational forecasters deploy a single off-the-shelf model without collecting and fitting training data for each new station.","feed_headline":"Chronos zero-shot model beats trained baselines on Everglades","feed_subtitle":"Used with no retraining, it posted the lowest error at every station and lead time.","key_machinery":"The load-bearing artifact is Chronos's tokenizer: it scales and quantizes raw time-series values into a fixed vocabulary and trains an encoder-decoder Transformer with a cross-entropy loss, effectively learning a 'language of time series' that can be applied to a new series without fine-tuning. This mechanism is what lets a model pretrained on general time-series data transfer to Everglades water levels in zero-shot inference. The paper's benchmark harness supplies the contrast: 37 variables of daily hydrologic data, a 100-day input window, five stations, four forecast horizons, and MAE/RMSE for overall accuracy plus SEDI for extreme-event skill.","core_discovery":"The central claim is that Chronos, a pretrained time-series foundation model used in a zero-shot mode, significantly outperforms all 12 task-specific deep-learning models on daily water-level forecasting in the Everglades. In the reported tables, Chronos has the lowest MAE and RMSE for every station (NP205, P33, G620, NESRS1, NESRS2) and every horizon (7, 14, 21, 28 days), with its advantage persisting as the lead time lengthens; at the 28-day horizon it also achieves the highest SEDI at every station, with overall SEDI 0.710 versus the next-best 0.659. The paper additionally reports that the other foundation models (TimeGPT, TimesFM, Timer, Moirai) are not consistently competitive, and that among task-specific models NBEATS, TSMixer, PatchTST, and RMoK lead their architectural families. The authors attribute Chronos's edge to its tokenization of time series into a quantized vocabulary plus pretraining on data correlated with Everglades conditions, while other foundation models were pretrained on less similar data.","pith_inferences":["The fairest test of the paper's claim would feed every model the identical input set—either all 37 variables for all models or only the target station's water-level history for all models—because five of the foundation models are univariate by design; until that ablation is run, the size of Chronos's true advantage is uncertain.","If the controlled comparison still favors Chronos, the same zero-shot design could be carried to other wetland or riverine systems where station-specific training data are scarce, making this benchmark a template for evaluating foundation models in environmental forecasting.","Because Chronos's pretraining included U.S. weather-station records, a natural boundary condition is geography: its advantage may shrink outside regions whose hydroclimatology resembles that corpus, which could be tested by rerunning the benchmark on stations in markedly different climates.","The SEDI ranking depends on the chosen 10%/90% extreme thresholds; testing other quantiles (e.g., 5%/95%) would reveal whether Chronos's extreme-event skill is robust or threshold-sensitive."],"forward_implications":["Operational water-level forecasting in the Everglades could adopt Chronos immediately for 7- to 28-day horizons, since zero-shot inference matched or beat every trained model at every station tested.","Forecast horizon should guide model choice: linear models such as NLinear and DLinear do well short-term but decay sharply by 28 days, while NBEATS and PatchTST hold up better on longer leads.","Extreme-value warnings could improve in resolution: Chronos posts the highest SEDI for extreme high/low water levels, although all models still miss abrupt shifts in the underlying dynamics.","Pretraining data distribution matters more than model scale: Chronos's win alongside the weak results of TimesFM, Timer, and Moirai suggests that exposure to similar weather/hydrology data, not parameter count, drives transfer."],"supporting_citations":[{"why":"Defines Chronos's tokenization, model family, and pretraining corpus, including the U.S. weather-station data the paper links to its success.","marker":"(Ansari et al. 2024)"},{"why":"Everglades Depth Estimation Network (EDEN) provides the daily water-level records used as targets.","marker":"(Haider et al. 2020)"},{"why":"DBHYDRO supplies rainfall, PET, pump-flow, and gate-flow covariates used in the multivariate setup.","marker":"(South Florida Water Management District 2024)"},{"why":"Introduces DLinear and NLinear, the linear baselines whose short-horizon strength and long-horizon decay are part of the comparison.","marker":"(Zeng et al. 2023)"},{"why":"Defines NBEATS, the strongest task-specific MLP baseline Chronos is compared against.","marker":"(Oreshkin et al. 2019)"},{"why":"Supplies the SEDI metric and the 10%/90% threshold convention used to evaluate extreme water-level predictions.","marker":"(Han et al. 2024b)"}],"fun_headline_variants":["Chronos zero-shot beats all 12 baselines in Everglades","Zero-shot Chronos wins Everglades water-level forecasting","No retraining: Chronos outperforms all in Everglades","Chronos zero-shot beats 12 models in Everglades water-level","Everglades forecast: zero-shot Chronos beats all trained models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes every model saw the same information, but the paper does not state which variables the foundation models actually received; if they saw only the target station's water-level history while task-specific models saw all 37 variables, the reported win may partly reflect an input-information mismatch rather than model superiority.","fun_headline_variants_meta":{"raw":{"variants":["Chronos zero-shot beats all 12 baselines in Everglades","Zero-shot Chronos wins Everglades water-level forecasting","No retraining: Chronos outperforms all in Everglades","Chronos zero-shot beats 12 models in Everglades water-level","Everglades forecast: zero-shot Chronos beats all trained models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001875,"raw_usage":{"total_tokens":7391,"prompt_tokens":1011,"completion_tokens":6380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":6287}},"tokens_in":627,"tokens_out":6380,"duration_ms":43095,"temperature":1.0,"reasoning_tokens":6287,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:17:57.171451+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the five-station, four-horizon benchmark under a controlled input ablation: give every model either the full 37-variable context or only the target station's water-level history, then compare Chronos with the best task-specific models on MAE, RMSE, and SEDI. If Chronos no longer leads at all stations and lead times in both settings, the paper's central claim that zero-shot Chronos is uniformly superior is not supported.","supporting_citations":[{"cited_title":"D.; McCloskey, B.; and Henkel, H","cited_arxiv_id":null,"evidence_quote":"Everglades Depth Estimation Network (EDEN) provides the daily water-level records used as targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DBHYDRO supplies rainfall, PET, pump-flow, and gate-flow covariates used in the multivariate setup."}],"review_version":1}