{"id":"4a860263-9e93-4c64-8690-b91fca55dbde","arxiv_id":"2505.13521","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Zero-shot CHRONOS forecasts are competitive for 5 and 10 year mortality horizons, TimesFM underperforms, and a mortality-trained random forest achieves the lowest SMAPE overall.","lead":"This paper tests whether general-purpose zero-shot forecasting models can predict mortality rates without fine-tuning. CHRONOS does well at short horizons, TimesFM does not, and a simple random forest trained on mortality data wins overall.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 6 admits a training/validation overlap for mortality-trained models; this leakage may inflate RF and fine-tuned CHRONOS results at the 20-year horizon.","rationale":"The reader's weakest assumption concerns whether mortality data appear in the pretraining corpora of TimesFM and CHRONOS. That is a legitimate but unverifiable, external concern. A more immediate threat is the paper's own Section 6 admission of possible overlap between training and validation for the mortality-trained models. Because the validation period is the last 5/10/20 years and the training cutoff appears to be a global last-20-years boundary, countries whose series end before the most recent year have part of their 20-year validation window in training. This would directly inflate RandomForest and CHRONOSSmallFinetuned at the 20-year horizon, undermining the abstract's main claims. The reader's verdict is CONDITIONAL, and this concern reinforces that condition: the authors must provide code or a reproducible split and re-run without overlap. I therefore keep the verdict unchanged.","tokens_in":13788,"tokens_out":9609,"duration_ms":85178,"concrete_test":"Reconstruct the exact train/validation split from the paper's data (Table 1 and Section 6) and check, for each of the 50 files, whether any year in the last 5, 10, or 20 years of that file also appears in the training set. If overlaps exist, retrain LSTM, RandomForest, and CHRONOSSmallFinetuned on a strict per-series split (train on years strictly before the final 20 years of that series) and recompute Tables 4 and 6; if RandomForest and CHRONOSSmallFinetuned no longer top the ranking, the headline results are artifacts of leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparison between zero-shot foundation models and mortality-trained models (RandomForest, LSTM, CHRONOSSmallFinetuned) is undercut by a potential training/validation overlap that the authors explicitly acknowledge. Section 6 states: \"For pretraining, we used historical data spanning all countries, excluding the last 20 years. As the training and validation periods differ to some degree for different countries, there may be partial overlap in time between these periods.\" If \"excluding the last 20 years\" means a global cutoff (e.g., all data before 2004, since the latest data are 2023), then for any country whose final year is before 2023, the last 20 validation years extend earlier than the cutoff and overlap with training. For example, Israel (final year 2016): the 20-year validation window is 1997-2016, while training includes 1983-2003, so 7 of the 20 validation years were seen during training. The effect is largest for short-history countries, exactly where the paper reports RF and the fine-tuned CHRONOS excel (Figure 12). This leakage would inflate the performance of all three mortality-trained models, directly biasing the abstract's claims that \"A Random Forest model... achieved the best overall performance\" and \"Fine-tuning CHRONOS... significantly improved long-term accuracy.\" The paper calls it \"a small degree,\" but the overlap can cover a substantial fraction of the 20-year validation window for countries ending before 2023. Because the code is not provided, the actual split cannot be verified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates zero-shot time-series foundation models (TimesFM and CHRONOS in three sizes) against benchmark methods (ARIMA, exponential smoothing, LSTM, Random Forest, and Lee-Carter variants) for forecasting age-specific mortality rates from the Human Mortality Database. It uses 50 HMD data files, 111 age groups, horizons of 5, 10, and 20 years, and SMAPE-based evaluation with Wilcoxon signed-rank tests and practical-significance thresholds. The main reported findings are that a Random Forest trained on mortality data performs best overall, CHRONOS is competitive in the short term and outperforms traditional methods, TimesFM underperforms, and fine-tuning CHRONOS on mortality data improves long-term accuracy.","tokens_in":14098,"tokens_out":6734,"duration_ms":68794,"significance":"If the results survive a corrected evaluation, the paper is a useful applied benchmark for zero-shot mortality forecasting. Its strengths include the breadth of the test bed (50 populations, 111 age groups, three horizons), the use of public data and public pretrained models, and the transparent reporting of median SMAPE, Wilcoxon tests, and practical-significance comparisons. The practical takeaway—that a simple lag-based Random Forest is competitive with, and often better than, general zero-shot foundation models for this domain, and that CHRONOS is the stronger of the two tested foundation models—is relevant to actuaries, demographers, and public-health planners. However, the current manuscript does not yet support its headline claims because the mortality-trained models were evaluated under a training/validation split that the authors themselves acknowledge can overlap in time; this directly affects the comparisons that drive the abstract. The reported results are also based on a single validation window per horizon, which limits the generality of the rankings.","major_comments":[{"comment":"The acknowledged temporal overlap between the training data of LSTM, RandomForest, and CHRONOSSmallFinetuned and the validation windows is load-bearing for the paper's central claims. Section 6 states that pretraining used \"historical data spanning all countries, excluding the last 20 years\" and then admits \"there may be partial overlap in time between these periods.\" Under the natural reading that the exclusion is a global cutoff (e.g., all data before 2004, because the most recent data are from 2023), the last 20 years of validation extend into the training interval for countries whose series end before 2023. For example, for Israel (last year 2016; Table 1) the 20-year validation window is 1997-2016 and training includes 1983-2003, so 7 validation years were seen in training; for Russia (last year 2014) the overlap is 9 years; for Ukraine (last year 2013) it is 10 years; and for NZL_MA (last year 2008) it is 15 years. These are exactly the short-history countries in the Q1 category where Figure 12 shows RandomForest and fine-tuned CHRONOS with their largest advantages at the 20-year horizon. Calling the leakage \"a small degree\" is therefore inaccurate for these populations, and Tables 6-7 plus the abstract's claims that \"A Random Forest model... achieved the best overall performance\" and \"Fine-tuning CHRONOS... significantly improved long-term accuracy\" are directly affected. The authors should state the exact cutoff, quantify the overlap for every country, and re-run the mortality-trained models under a split that guarantees no temporal overlap, or report the sensitivity of the rankings to an overlap-free split.","section":"6"},{"comment":"The single validation window is another load-bearing limitation for the comparative claims. Section 8.4 admits that \"in each case only the last available period was used for validation\" and that different results might have been obtained for earlier periods. The headline rankings in Tables 2, 4, and 6 are therefore based on exactly one realization of history for each horizon. Given well-documented structural breaks in mortality (pandemics, changes in data collection, geopolitical events), the relative ordering of models, especially at the 20-year horizon, may be period-specific. The authors should either add rolling-origin or multiple-window validation or explicitly restrict the conclusions to \"the most recent period\" rather than general claims about model performance. At minimum, the abstract and conclusions should be softened to reflect this conditional scope.","section":"8.4"},{"comment":"The \"zero-shot\" characterization rests on an unverified assumption. Section 3 says \"According to the documentations, mortality rate series were not used during the training\" of TimesFM and CHRONOS. The entire comparison between zero-shot and mortality-trained models depends on the pretraining corpora being disjoint from the mortality domain; if mortality or closely related life-table data were present in the pretraining data, the zero-shot framing would be distorted. This is a correctness-risk concern rather than an observed error, but the paper should at least label it as an assumption that is not independently verified. A concrete test, such as checking whether the foundation models' nearest training series resemble mortality series, or a clear statement that contamination cannot be ruled out from public documentation, would make the claim appropriately cautious.","section":"3"}],"minor_comments":[{"comment":"The text introducing Tables 2, 4, and 6 refers to \"the three methods,\" but the tables report 13 methods; this should be corrected to \"thirteen methods.\"","section":"7.1"},{"comment":"Because the 50 data files include multiple files for France, Germany, the United Kingdom, and New Zealand, the effective number of independent populations is smaller than 50; the paper should note this and discuss its implication for the paired Wilcoxon tests, whose effective sample size is also smaller than 5550 independent series.","section":"5"},{"comment":"The forecasting mechanics of the LSTM and Random Forest are underspecified for horizons beyond their 16-step or 16-lag windows; the paper should state whether 5-, 10-, and 20-year forecasts are generated recursively and how the initial conditioning is handled.","section":"4"},{"comment":"The phrase \"excluding the last 20 years\" is ambiguous: it could mean a per-country cutoff or a global cutoff. The authors should specify the exact split in one sentence, since this ambiguity is directly related to the leakage concern in Major Comment 1.","section":"6"},{"comment":"Wilcoxon p-values are reported as 0.00; they should be reported as p < 0.001, and the paper should state whether any multiple-comparison correction was applied (the practical-significance threshold mitigates this concern, but the reporting should be explicit).","section":"7"},{"comment":"There are minor typos, including \"decoposition\" in Section 4 and \"with with\" in Section 5; these should be corrected.","section":"5"}],"recommendation":"major_revision","confidential_remarks":"The leakage issue is serious but fixable: the authors need to clarify the training/validation split and rerun the mortality-trained models on an overlap-free design. If that is not possible within the revision, the claims about RandomForest and fine-tuned CHRONOS being best at the 20-year horizon should be substantially weakened or removed. The paper's applied benchmark value is real, but the abstract currently overstates the support for its main conclusions. The authors should also be encouraged to release the exact training/validation split and the fine-tuning configuration to make the comparison reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid empirical benchmark—CHRONOS is competitive on short horizons, Random Forest wins overall, fine-tuned CHRONOS helps long-term, and TimesFM underperforms. The authors ran a genuinely broad evaluation: 50 populations, 111 age groups, three horizons. That breadth is real value, and the paper reports what it did honestly, including a limitations section that names the leakage risk and the single-window validation.\n\nThe new piece is the application of TimesFM and CHRONOS to mortality data and the head-to-head against Lee-Carter and ARIMA. That hasn't been done at this scale before. The finding that a simple domain-trained RF beats off-the-shelf foundation models is consistent with the broader time-series literature, but it's a useful data point for practitioners.\n\nThe main soft spot is the one the authors flag in Section 6. They say they pretrained on 'historical data spanning all countries, excluding the last 20 years' and then admit 'there may be partial overlap' between training and validation periods. That phrasing doesn't tell us whether the cutoff is per-country or global. If it's global, then for countries with earlier end dates, a substantial chunk of the 20-year validation window can appear in training—exactly where RF and the fine-tuned CHRONOS show their biggest advantage. This directly affects the paper's headline claims. The authors call it 'small degree'; it may not be small for short-history countries. Without the code or a precise description of the split, a reader can't resolve this. That's a load-bearing flaw, not a cosmetic one.\n\nAlso worth noting: the zero-shot characterization rests on the models' documentation rather than a verified data audit, and the evaluation uses only the last window of each horizon, so the results could be period-sensitive. Lack of code is the biggest practical issue for a benchmark paper.\n\nWho's this for? Actuaries, demographers, and applied forecasters who want to know whether foundation models are ready for mortality work. It deserves a serious referee—the questions it asks are relevant and answerable. But the leak issue and missing code need to be addressed before the main claims can be fully trusted.","headline":"A broad, honest empirical benchmark of zero-shot foundation models for mortality forecasting, but the acknowledged train/validation overlap and missing code temper its headline claims.","tokens_in":14606,"tokens_out":4154,"would_cite":false,"duration_ms":37805,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Random Forest beat zero-shot AI models at forecasting mortality rates.","keywords":["zero-shot time series forecasting","mortality rate forecasting","foundation models","CHRONOS","TimesFM","Lee-Carter model","Random Forest","SMAPE"],"falsifier":"A concrete check is to examine the released training data or documentation of CHRONOS and TimesFM for any series from the Human Mortality Database; if any HMD-derived mortality series appears in the pretraining corpus, the zero-shot claim fails. Independently, rerunning the full benchmark with an earlier validation window, such as forecasting 1980-2000 from data through 1979, would reveal whether the reported ranking is an artifact of testing only on the most recent period.","tokens_in":13618,"feed_emoji":"📉","tokens_out":3870,"duration_ms":33307,"temperature":0.7,"pith_summary":"The paper asks whether pre-trained zero-shot time-series foundation models can forecast mortality rates without any mortality-specific training. It finds that the answer depends sharply on the model: CHRONOS delivers competitive 5-year forecasts, beating ARIMA and the Lee-Carter model, while TimesFM consistently underperforms. Fine-tuning CHRONOS on mortality data substantially improves its long-term accuracy, and a simple Random Forest trained on lagged mortality rates achieves the best overall performance. The study thus shows that zero-shot forecasting has promise, but is not yet a replacement for domain-trained models.","feed_headline":"A Random Forest beat zero-shot AI at forecasting mortality","feed_subtitle":"CHRONOS tops ARIMA and Lee-Carter at 5 years, but a mortality-trained Random Forest wins overall.","key_machinery":"The load-bearing mechanism is the evaluation design itself: a large panel of 50 national and sub-national life tables from the Human Mortality Database, 111 age groups grouped into five age bands, four country-income and four data-length strata, three horizons (5, 10, and 20 years), and SMAPE as a scale-free error metric. Pairwise Wilcoxon signed-rank tests decide whether differences are statistically significant, and a 5-point median difference is required for practical significance. This design is what lets the paper rank thirteen forecasting methods and claim that the differences between zero-shot, fine-tuned, and mortality-trained models are real.","core_discovery":"On the paper's own terms, the central discovery is a performance ranking that shifts with horizon. For 5-year forecasts, the zero-shot CHRONOS models (8M to 710M parameters) rank among the best, second only to Random Forest, and far ahead of Lee-Carter and ARIMA; at 20 years, fine-tuned CHRONOS moves to second place, and both zero-shot foundation models fall behind all mortality-trained models. TimesFM never escapes the bottom half. The ranking is established by median symmetric MAPE over 50 Human Mortality Database populations times 111 age groups, with Wilcoxon signed-rank tests and a 5-percentage-point threshold for practical significance.","pith_inferences":["If CHRONOS's pretraining corpus turns out to contain Human Mortality Database series, the 'zero-shot' comparison collapses; the authors' assumption rests only on documentation, so an audit of pretraining data is a cheap, decisive check.","The paper's validation uses only the most recent 5-, 10-, and 20-year window; re-running the benchmark with a historical holdout (e.g., predicting the 1980s from data up to 1980) would test whether the ranking reflects genuine generalisation or period-specific trends.","A directly testable extension is to fine-tune TimesFM the same way as CHRONOS: if TimesFM still underperforms, the gap is architectural; if it improves dramatically, the problem was adaptation, not model quality.","The strong Random Forest baseline suggests that autoregressive tree ensembles on pooled populations should become the default benchmark against which future mortality foundation models are judged."],"forward_implications":["For short-horizon (5-year) mortality forecasting, zero-shot CHRONOS is a viable off-the-shelf alternative to Lee-Carter and ARIMA.","Fine-tuning a foundation model on domain data is the key to long-horizon accuracy: fine-tuned CHRONOS gains 5 or more SMAPE points at 20 years over its zero-shot version.","The best overall mortality forecasts come from a Random Forest trained on lagged mortality rates across all populations, not from any foundation model tested.","TimesFM's poor performance suggests that not all zero-shot models transfer to low-frequency demographic series, and model choice matters more than the zero-shot label.","Traditional demographic benchmarks such as Lee-Carter are consistently beaten by machine-learning models, including zero-shot ones at short horizons."],"supporting_citations":[{"why":"Describes CHRONOS, the zero-shot pretraining framework whose models are the central objects evaluated.","marker":"[1]"},{"why":"Describes TimesFM, the second zero-shot model that underperforms in the study.","marker":"[8]"},{"why":"Introduces the Lee-Carter model, the traditional mortality forecasting benchmark that zero-shot CHRONOS beats at short horizons.","marker":"[17]"},{"why":"Provides all mortality data from the Human Mortality Database used across 50 populations.","marker":"[9]"},{"why":"Introduces Random Forest, the method that achieves the best overall forecasting performance.","marker":"[5]"},{"why":"Defines ARIMA, a traditional baseline model compared throughout the study.","marker":"[4]"},{"why":"Provides the AutoARIMA implementation used to fit ARIMA benchmarks automatically.","marker":"[34]"}],"fun_headline_variants":["Random Forest beats zero-shot AI in global mortality forecasts","CHRONOS zero-shot matches ARIMA at 5 years, Random Forest beats all","Mortality forecasting: domain-trained Random Forest outperforms zero-shot AI","Zero-shot mortality forecasts: CHRONOS strong, TimesFM weak, RF best"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that CHRONOS and TimesFM are truly zero-shot relies on the paper's stated assumption, taken from the models' documentation, that mortality-rate series were not part of their pretraining data; if that is wrong, the zero-shot versus domain-trained comparison is unfair.","fun_headline_variants_meta":{"raw":{"variants":["Random Forest beats zero-shot AI in global mortality forecasts","CHRONOS zero-shot matches ARIMA at 5 years, Random Forest beats all","Mortality forecasting: domain-trained Random Forest outperforms zero-shot AI","Zero-shot mortality forecasts: CHRONOS strong, TimesFM weak, RF best"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000418,"raw_usage":{"total_tokens":2094,"prompt_tokens":824,"completion_tokens":1270,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":1191}},"tokens_in":440,"tokens_out":1270,"duration_ms":10278,"temperature":1.0,"reasoning_tokens":1191,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:42:20.071957+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to examine the released training data or documentation of CHRONOS and TimesFM for any series from the Human Mortality Database; if any HMD-derived mortality series appears in the pretraining corpus, the zero-shot claim fails. Independently, rerunning the full benchmark with an earlier validation window, such as forecasting 1980-2000 from data through 1979, would reveal whether the reported ranking is an artifact of testing only on the most recent period.","supporting_citations":[{"cited_title":"Modeling and forecasting US mortality","cited_arxiv_id":null,"evidence_quote":"Introduces the Lee-Carter model, the traditional mortality forecasting benchmark that zero-shot CHRONOS beats at short horizons."},{"cited_title":"Human Mortality Database","cited_arxiv_id":null,"evidence_quote":"Provides all mortality data from the Human Mortality Database used across 50 populations."},{"cited_title":"Random forests","cited_arxiv_id":null,"evidence_quote":"Introduces Random Forest, the method that achieves the best overall forecasting performance."},{"cited_title":"Time series analysis: forecasting and control","cited_arxiv_id":null,"evidence_quote":"Defines ARIMA, a traditional baseline model compared throughout the study."},{"cited_title":"Smith et al","cited_arxiv_id":null,"evidence_quote":"Provides the AutoARIMA implementation used to fit ARIMA benchmarks automatically."}],"review_version":1}