{"id":"6c2ad3c6-38c5-4c00-a346-f6bc29cd2356","arxiv_id":"2412.19286","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Across five public datasets, weighted XGBoost and autoencoder baselines match or beat time series foundation models on anomaly detection and prediction, at a fraction of the computational cost.","lead":"This paper benchmarks five time series foundation models against statistical and deep learning baselines for anomaly detection and prediction on five public datasets. It finds that the baselines often match or outperform the foundation models while using far less compute.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Anomaly-prediction comparison conflates raw forecast error with anomaly-prediction ability, so the central claim is not yet supported; a unified classification-based protocol is needed.","rationale":"The reader's weakest assumption correctly identified the anomaly-prediction operationalization as the fragile point, but my concern is more specific: the reported metrics are not merely potentially unfair, they are not anomaly-prediction metrics at all for most TSFMs. A forecast error on raw sensor values cannot establish whether anomalies are predicted, because anomaly prediction requires a decision rule that maps forecasts to flags. The paper's Appendix D describes such thresholds but the main tables never report classification results from them. Meanwhile, MOIRAI is evaluated by forecasting the label itself, which is a different target. This makes the numerical comparisons in Tables 2 and 4 incommensurable across models. The central claim may still be true, and the paper's anomaly-detection results provide some support, but the prediction evidence as presented does not carry the weight of the abstract's conclusion. The reader's CONDITIONAL verdict is therefore appropriate: the conclusion is plausible but not currently supported without a unified, classification-based evaluation protocol and correction of the data inconsistencies.","tokens_in":14334,"tokens_out":4615,"duration_ms":43759,"concrete_test":"Recompute Tables 2 and 4 anomaly-prediction rows under one shared protocol on identical train/test splits: (1) define each feature's safe thresholds using training-period quantiles only; (2) for every model, forecast the same set of features (or the binary label for label-forecasting variants), apply the fixed thresholds to flag predicted anomalies; (3) report precision/recall/F1 and point forecast error separately; (4) ensure MSL splits sum to the actual dataset size and reconcile Table 4 SMD MAE (0.93 vs RMSE 0.28). If the ranking between TSFMs and XGBoost/Autoencoder changes on the classification-based anomaly metric, the central claim would require revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the anomaly-prediction columns in Tables 2–4 measure what they claim. For TimeGPT, FPT, Time-MOE, and Chronos, \"anomaly prediction\" is operationalized as multistep forecasting of raw sensor values, and the reported MSE/RMSE/MAE are ordinary forecast errors (Appendix C, D). For MOIRAI, the target is instead the binary anomaly label (Appendix C.4). These are not the same quantity: forecast error on sensor values does not equal ability to predict anomalous events, and a model with low MSE can still miss all anomalies (or vice versa). Baselines (Weighted XGBoost, Autoencoder) are benchmarked on the same table, but their prediction targets and training supervision are not aligned with those of the TSFMs. Unless a single protocol converts forecasts to anomaly flags via fixed thresholds defined a priori on training data and evaluates precision/recall/F1, the abstract's conclusion that traditional models \"often match or outperform\" TSFMs in anomaly prediction is not supported by the evidence presented. Secondary red flags, such as the anomalous MAE value for SMD in Table 4 (0.93 vs RMSE 0.28) and the MSL train/test counts (56K/73K exceeding the stated 66,709 total records), reinforce that the quantitative basis needs correction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates five time series foundation models (TimeGPT, FPT, Time-MOE, MOIRAI, Chronos) for anomaly detection and anomaly prediction on five datasets (Future Factories, Pulp, SMD, MSL, ETTh1), comparing zero-shot and fine-tuned performance against a weighted XGBoost baseline and an autoencoder. The authors report that TSFMs can be adapted for anomaly tasks, but that traditional statistical and deep learning models often match or outperform them, and that TSFMs incur higher computational costs. The paper contributes benchmark tables, a computational-cost analysis, and a GitHub repository with preprocessed data and code.","tokens_in":14600,"tokens_out":4982,"duration_ms":41094,"significance":"If the central claim held, the paper would be a valuable cautionary benchmark: practitioners should not assume pretrained TSFMs are superior for anomaly detection and prediction, and should compare against simple baselines. The study is broad (five models, five datasets, zero-shot and fine-tuning, computational cost) and the authors provide reproducibility artifacts. However, the validity of the central claim depends on the anomaly-prediction protocol, which is inconsistent across models, and on the accuracy of the reported tables. Those issues must be resolved before the conclusion can be considered supported.","major_comments":[{"comment":"Anomaly prediction is operationalized differently across methods. For TimeGPT, FPT, Time-MOE, and Chronos, the reported prediction metrics are ordinary forecast errors on raw sensor values, converted to anomaly flags only via thresholds described in Appendix D; for MOIRAI, the binary anomaly label column is used directly as the forecasting target (Appendix C.4). Raw forecast error is not equivalent to anomaly-prediction ability, and the baselines (Weighted XGBoost, Autoencoder) are not shown to use the same thresholding protocol. Consequently, the abstract's conclusion that traditional models often match or outperform TSFMs in anomaly prediction is not supported by the evidence as presented. A unified protocol (e.g., fixed thresholds defined on training data only, with precision/recall/F1 on anomaly flags) is needed.","section":"Appendix C.4, D; Tables 2–4"},{"comment":"The text states that Weighted XGB consistently yields lower MSE, RMSE, and MAE than the Autoencoder, but the table shows Weighted XGB with MAE 0.93 versus Autoencoder's 0.20, while the RMSE is 0.28 versus 0.54. The MAE value is internally inconsistent with both the adjacent RMSE and the stated claim; this entry must be corrected and the surrounding claims re-verified.","section":"Table 4, SMD row"},{"comment":"The MSL dataset is stated to contain 66,709 records, but every MSL row in Tables 2–4 reports #Train = 56K and #Test = 73K, which already sums to 129K and exceeds the total, with no validation split reported. Similar train/test inconsistencies appear elsewhere (e.g., Table 2's FF TimeGPT row has #Train 779K and #Val 194K but no #Test). These discrepancies undermine confidence in the quantitative results and must be rectified.","section":"Appendix A and Tables 2–4"},{"comment":"The claim that fine-tuning yields only marginal improvements is not consistently supported by the tables. For example, on SMD, MOIRAI's MSE worsens from 0.011 (zero-shot) to 1.3 (fine-tuned), and on ETTh1 it is unchanged; the paper does not report variance or significance, so the generalization that fine-tuning gives only marginal improvements is not supported by the presented numbers.","section":"Tables 2 and 3"}],"minor_comments":[{"comment":"Table 1 classifies MOIRAI as 'Encoder-Only', but the text describes a decoder-only transformer; please align these descriptions.","section":"Table 1 and MOIRAI section"},{"comment":"Appendix C.5 states that Chronos-tiny has 8M parameters, while Table 1 gives Chronos (tiny) a max model size of 710M; clarify which variant was actually used and correct the inconsistency.","section":"Appendix C.5 and Table 1"},{"comment":"The #Test entry is missing for the Future Factories TimeGPT row; include it or explicitly state that no test split was used in that configuration.","section":"Table 2, FF TimeGPT row"},{"comment":"The statement that 'we replicate the conditions of their original papers' is not supported by a complete list of hyperparameters; provide context lengths, batch sizes, and checkpoints for every model-dataset combination, as some details are only partially reported in Appendix C.","section":"Analysis Procedure"},{"comment":"Several reference entries have formatting issues, for example 'Z ¨ufle' in Zhang et al. 2023; a final proofread of the bibliography is needed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's GitHub artifact is a strength, but the core anomaly-prediction comparison needs re-analysis: either a unified classification-based protocol with training-only thresholds, or a softened conclusion that limits claims to forecasting error. The data inconsistencies in Tables 2–4 should be corrected before any further review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is one of the first systematic comparisons of five time-series foundation models (TimeGPT, FPT, Time-MOE, MOIRAI, Chronos) against weighted XGBoost and autoencoders for anomaly detection and prediction across five datasets, including low-pattern manufacturing data. That is genuinely useful. Second, the paper's central claim — that traditional models often match or beat TSFMs on anomaly work — is directionally plausible but not fully established by the numbers, because the anomaly-prediction columns mix incompatible protocols.\n\nWhat it does well: the anomaly detection results are fairly direct, with TimeGPT and FPT scored via precision/recall/F1 against the same baseline classifiers (XGB, autoencoder) on the same labels. The computational cost comparison, though rough, makes a fair point that large foundation models are far slower. The manuscript is honest about TSFMs not being designed for anomaly prediction, and the appendix describes the forecasting-based adaptations.\n\nThe soft spots are real. For anomaly prediction, TimeGPT/FPT/Time-MOE/Chronos are evaluated by forecasting raw sensor values and reporting MSE/RMSE/MAE; MOIRAI instead forecasts the binary anomaly label. Those are different tasks. The appendix says thresholds are used to convert forecasts to anomaly flags, but no classification metric (precision/recall/F1) is reported for these models, so the 'prediction' columns measure forecast error, not anomaly-prediction ability. The stress-test note is right about this. Second, the feature subsets are not aligned: TimeGPT uses 3 features on FF, FPT 20, baselines 20, etc. Third, Table 4 shows Weighted XGB with MAE 0.93 on SMD versus autoencoder's 0.20, while the text claims XGB is better on all metrics. Fourth, MSL train/test counts (56K/73K) sum beyond the dataset's total records (66,709). These are exactly the kind of numbers that make a benchmark hard to trust.\n\nThe central direction may still hold — simple supervised baselines often do beat generic pretrained models on narrow tasks. But a reader cannot verify that from this manuscript as-is. The GitHub repo is promised but not checkable from the text.\n\nWho is it for? Practitioners weighing TSFMs for anomaly tasks, and researchers building better evaluation protocols for foundation models in time series. It deserves a serious referee, but the referee should demand a unified protocol (forecast-to-anomaly conversion with fixed thresholds evaluated by classification metrics, aligned feature sets, corrected tables, and released code) before the claims are accepted.\n\nMy recommendation: send it to peer review with major revision required. The question is important, the comparison is genuinely new, and the flaws are fixable.","headline":"A useful but sloppy benchmark: the central caution about TSFMs is plausible, but the anomaly-prediction numbers compare different tasks and the tables have contradictions.","tokens_in":15130,"tokens_out":2588,"would_cite":false,"duration_ms":23677,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that time-series foundation models, despite their forecasting strengths, generally do not beat traditional statistical and deep-learning baselines on anomaly detection and prediction, and are much more expensive.","keywords":["time series foundation models","anomaly detection","anomaly prediction","zero-shot forecasting","fine-tuning","computational cost","benchmarking","XGBoost"],"falsifier":"A re-run of the same five datasets in which TimeGPT, MOIRAI, or Chronos is given a purpose-built anomaly head or an anomaly-aware loss, with thresholds chosen only from training data, and then beats weighted XGBoost's Pulp F1 of 0.94 or its SMD F1 of 0.87 at comparable inference time, would directly contradict the paper's blanket conclusion.","tokens_in":14119,"feed_emoji":"📉","tokens_out":7628,"duration_ms":61836,"temperature":0.7,"pith_summary":"Time series foundation models (TSFMs) are marketed as general-purpose forecasting tools, but their usefulness for anomaly detection and prediction is largely untested. This paper benchmarks five TSFMs—TimeGPT, FPT, Time-MOE, MOIRAI, and Chronos—against a weighted XGBoost model and an autoencoder on five multivariate datasets, including rare-event manufacturing data and data without clear patterns. Its central finding is that the baselines often match or outperform the TSFMs on both anomaly detection and anomaly prediction, while being far cheaper to run. Fine-tuning the TSFMs yields only marginal gains. If the finding generalizes, organizations should treat TSFM superiority as a hypothesis to test, not an assumption.","feed_headline":"Simple models beat time-series foundation models on anomalies","feed_subtitle":"A five-dataset benchmark finds weighted XGBoost and autoencoders match or exceed TSFM accuracy at far lower compute.","key_machinery":"The evaluation design is the paper's central mechanism. It selects TSFMs only if they are explicitly designed and trained for anomaly detection or forecasting on multivariate data, then applies each model under its original hyperparameters, replicates zero-shot and fine-tuned settings, and maps forecast outputs to anomaly labels through fixed feature thresholds before comparing all models on identical train/test splits. The baselines—weighted XGBoost and an autoencoder—carry the argument because they are trained from scratch on the same data and provide the accuracy-efficiency point of comparison.","core_discovery":"The paper's core claim is that TSFMs, while competitive in pure forecasting, are not specialized for anomaly work and lose to simpler models when adapted to it. Only TimeGPT and FPT have built-in anomaly-detection behavior among the five reference TSFMs, and none is designed for anomaly prediction, so the authors adapt next-step forecasting outputs into anomaly predictions by comparing forecasts against predefined safe thresholds or by setting the anomaly label as the forecasting target. Under that adaptation, weighted XGBoost and an autoencoder achieve equal or better precision, recall, F1, and forecasting error on the Pulp, Future Factories, SMD, MSL, and ETTh1 datasets, with inference times under a minute to a few minutes versus tens of minutes for several TSFMs. The TSFMs also fail to show consistent zero-shot or few-shot advantages and require high computational resources.","pith_inferences":["Beyond the paper, the negative result may reflect a task mismatch rather than a fundamental ceiling on TSFMs; an anomaly-aware pretraining objective or a classification head on top of TSFM embeddings could close or reverse the gap.","Beyond the paper, the forecast-to-threshold procedure needs externally defined feature thresholds, which are rarely available in new deployments; a learned-threshold version is a concrete next experiment that would test whether the TSFM disadvantage persists.","Beyond the paper, the compute comparison is in inference minutes only; a full lifecycle cost including pretraining energy and fine-tuning would likely tilt the balance even further toward the baselines."],"forward_implications":["Anomaly-detection practitioners should benchmark simple statistical and deep-learning baselines before adopting a TSFM, because the paper finds the baselines match or exceed TSFM accuracy on five datasets.","Fine-tuning a TSFM is not an automatic win for anomaly work: the paper reports only marginal zero-shot-to-finetuned improvements across TimeGPT, MOIRAI, and Chronos.","The absence of a TSFM purpose-built for anomaly prediction means that forecast-to-threshold conversion is the standard route, and that route is less accurate and slower than the baselines in this study.","Computational budget matters: inference times for FPT and TimeGPT reach tens of minutes, while the baselines finish in under a minute on the same tasks, changing the practical cost equation even where accuracy is close."],"supporting_citations":[{"why":"Supplies TimeGPT, the only closed TSFM in the study, used for zero-shot anomaly detection and forecasting; its undisclosed pretraining data motivates the transparency concern.","marker":"Garza and Mergenthaler-Canseco 2023"},{"why":"Supplies FPT, the frozen-pretrained LLM-adapted transformer used for anomaly detection and forecasting.","marker":"Zhou et al. 2023"},{"why":"Supplies Time-MOE, the mixture-of-experts forecasting TSFM used for zero-shot anomaly prediction.","marker":"Shi et al. 2024"},{"why":"Supplies MOIRAI, the forecasting TSFM fine-tuned with the anomaly label as target for anomaly prediction.","marker":"Woo et al. 2024"},{"why":"Supplies Chronos, the tokenized time-series TSFM used for forecasting-based anomaly prediction.","marker":"Ansari et al. 2024"},{"why":"Supplies the Pulp and paper manufacturing rare-event dataset used in all comparisons.","marker":"Ranjan et al. 2018"},{"why":"Supplies the Future Factories assembly-pipeline dataset used in all comparisons.","marker":"Harik et al. 2024"},{"why":"Supplies the MSL spacecraft telemetry dataset used in all comparisons.","marker":"Hundman et al. 2018"},{"why":"Supplies the SMD server-machine dataset used in all comparisons.","marker":"Su et al. 2019"},{"why":"Supplies the ETTh1 electricity transformer temperature dataset used for forecasting comparisons.","marker":"Wu et al. 2022"}],"fun_headline_variants":["Simple models match or beat TSFMs for anomaly detection","Time-series foundation models underperform on anomaly tasks","XGBoost and autoencoders top TSFMs in anomaly benchmarks","TSFMs no better than simpler models on anomaly detection","Benchmark: classic models equal TSFMs on anomalies"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion depends on treating thresholded forecasts—or forecasting the anomaly label—as a faithful use of each TSFM's anomaly-prediction ability, equivalent to a supervised classifier.","fun_headline_variants_meta":{"raw":{"variants":["Simple models match or beat TSFMs for anomaly detection","Time-series foundation models underperform on anomaly tasks","XGBoost and autoencoders top TSFMs in anomaly benchmarks","TSFMs no better than simpler models on anomaly detection","Benchmark: classic models equal TSFMs on anomalies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001191,"raw_usage":{"total_tokens":4887,"prompt_tokens":889,"completion_tokens":3998,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":3915}},"tokens_in":505,"tokens_out":3998,"duration_ms":25829,"temperature":1.0,"reasoning_tokens":3915,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:44:09.306634+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A re-run of the same five datasets in which TimeGPT, MOIRAI, or Chronos is given a purpose-built anomaly head or an anomaly-aware loss, with thresholds chosen only from training data, and then beats weighted XGBoost's Pulp F1 of 0.94 or its SMD F1 of 0.87 at comparable inference time, would directly contradict the paper's blanket conclusion.","supporting_citations":[{"cited_title":"Dataset: Rare Event Classification in Multivariate Time Series","cited_arxiv_id":"1809.10717","evidence_quote":"Supplies the Pulp and paper manufacturing rare-event dataset used in all comparisons."}],"review_version":1}