{"id":"abc95353-bc02-458d-baf3-4e15c79725ea","arxiv_id":"2505.12738","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A frozen-LLM framework with dual-branch token alignment and spatio-temporal prompts beats prior epidemic forecasting models on four COVID-19 datasets.","lead":"A new framework called EpiLLM adapts large language models to forecast COVID-19 cases by jointly predicting human mobility and infections across regions. It reports better accuracy than 14 existing epidemic-forecasting models on four European country datasets, and claims performance improves with model size.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The mobility branch is asserted to be the core of EpiLLM, yet its forecast accuracy is never measured; the reported gains could be riding on an unvalidated component.","rationale":"The reader's weakest-assumption analysis identified the same load-bearing concern: Appendix D itself declares human mobility prediction to be core, yet the paper never measures it against ground truth. My stress-test confirms that this is not a cosmetic reporting gap but a premise on which the framework's claimed advantage rests. The direct-forecast results in Table 1 are internally consistent and the improvement over strong baselines such as ATMGNN is nontrivial, so I see no reason to reject the empirical comparison outright. However, the co-prediction mechanism is unverified, and Table 2 contains no non-EpiLLM baselines, so the multi-step part of the claim is even less supported. The proposed oracle-mobility test would settle whether the mobility branch's forecast quality actually drives the results: if ground-truth mobility does not help beyond the model's own predictions, then the co-prediction story is wrong even if the numbers stand; if it does help, the paper must report mobility accuracy to substantiate its central mechanism. This maintains the reader's CONDITIONAL verdict rather than converting it to acceptance or rejection: the empirical claim is plausible and potentially reproducible, but a core component of the architecture remains unvalidated. No independent machine-checked proof or full reproducible evaluation is provided, so the conditional stance is appropriate.","tokens_in":15755,"tokens_out":8660,"duration_ms":98399,"concrete_test":"On the same held-out test periods used in Table 2 (and Table 1 if those runs roll out mobility), compute the mobility branch's forecast error: report normalized RMSE/MAE of \\hat{M}_{T+1:T+h} against the true Facebook mobility matrices M_{T+1:T+h} in the same aggregation units used for training. Then rerun EpiLLM inference exactly as in the paper, but replace the predicted adjacency \\hat{A}_{T+i} with oracle ground-truth A_{T+i} at each step (Eq. 10). If the oracle variant improves case-forecast RMSE over the predicted-mobility variant by more than a few percent, the mobility branch's accuracy is genuinely load-bearing and must be reported and compared; if oracle and predicted variants are statistically indistinguishable, then co-prediction of mobility is not what drives the reported gains, and the framework's stated mechanism should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's design makes the mobility branch load-bearing: Eq. (9)-(10) feed predicted mobility matrices into the epidemic branch during inference, and Appendix D states that human mobility prediction 'constitutes the core component of our framework, as its performance directly determines whether models integrating external human mobility knowledge can achieve effective multi-step forecasting.' Nowhere, however, is the mobility branch's prediction compared with held-out ground-truth mobility. Table 5 ablates Graph2MLP, Adj2Aver, and Adj2Last against EpiLLM, but those variants only show that removing or degrading mobility input hurts; they do not show that the predicted mobility is accurate, nor do they isolate the contribution of mobility forecast quality from the frozen LLM's autoregressive prior. If the mobility branch produces poor forecasts, the co-prediction mechanism asserted in the abstract and Section 5.2 is not doing the claimed work, and the comparison reduces to an LLM forecaster with extra noisy inputs. If it produces good forecasts, the paper still has not supplied the evidence needed to support the stated mechanism. Because the success or failure of the mobility branch is decisive for the framework's central selling point, and because the ground-truth mobility matrices for the held-out windows are available in the same dataset, this is a resolvable but currently unverified premise. The same concern applies to direct-forecast settings if they roll out mobility as Eq. (9) indicates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EpiLLM, a framework that adapts frozen large language models to spatio-temporal epidemic forecasting. It uses a dual-branch design: an epidemic branch tokenizes infection-case histories with a GNN and a mobility branch tokenizes human-mobility matrices with an MLP, then an autoregressive next-token-prediction objective aligns both with an LLM. Spatio-temporal prompt learning adds learnable direction-aware edges and time-gating weights. The method is evaluated on COVID-19 datasets for England, France, Italy, and Spain for direct horizons of 3 and 7 days and multi-step horizons of 6 and 14 days. The authors report consistent direct-forecasting improvements over fourteen baselines, with EpiLLM-GEMMA3 achieving the best RMSE/MAE in most settings and up to 30.38% RMSE improvement on Spain, plus ablations, scaling experiments, prompt visualizations, and parameter-efficiency statistics.","tokens_in":15952,"tokens_out":5115,"duration_ms":59334,"significance":"If the results hold, the paper would demonstrate that a frozen LLM with light trainable projectors and adapters can outperform dedicated spatio-temporal epidemic models, which is a meaningful and practically attractive contribution. The code release, the explicit autoregressive reformulation, and the prompt-visualization analysis are strengths. I found no evidence of circularity: evaluation is on held-out future days and the hand-set prompt initializations do not encode test labels. However, the central mechanism and the multi-step claims currently rest on unverified assumptions about the mobility branch and on absent baseline comparisons, so the contribution is promising but not yet fully established.","major_comments":[{"comment":"The mobility branch is load-bearing but its forecast accuracy is never evaluated against ground truth. The paper states in Appendix D that \"Human mobility prediction constitutes the core component of our framework,\" and Eqs. (9)-(10) feed predicted mobility matrices into the epidemic branch during inference. Yet no table or figure reports held-out mobility-prediction error. Table 5 only shows that removing or approximating mobility input hurts performance; it neither establishes that the predicted mobility is accurate nor isolates the mobility branch's contribution from the frozen LLM's autoregressive prior. Please add a direct evaluation of mobility forecasts (e.g., MAE/RMSE of mobility matrices on held-out test periods) and, ideally, an oracle-mobility variant that uses ground-truth mobility during multi-step inference.","section":"Appendix D, Eq. (9)-(10), Table 5"},{"comment":"The multi-step forecasting results contain no baselines, only the three EpiLLM backbones. The text claims that \"Other baselines fail to perform multi-step forecasting due to the absence of future mobility prediction,\" but this is an assertion rather than a demonstrated result. Standard baselines such as ATMGNN, MPNN+LSTM, and MGNN can be run in recursive or direct multi-step modes, and an LLM-based case-only variant could be included as a control. Without such comparisons, the multi-step capability claim is unsupported.","section":"Table 2, Section 5.2"},{"comment":"The claim that \"Experimental results have passed the statistical significance tests\" is not substantiated. No test name, test statistic, p-value, confidence interval, or standard deviation is reported anywhere. Since the paper averages over 10 runs, error bars and a paired significance test across runs or regions should be reported; otherwise the word \"significantly\" in the abstract and Section 5.2 is not supported.","section":"Table 1 caption, Section 5.1"}],"minor_comments":[{"comment":"The England 14-day RMSE entry \"44.887\" appears to have an extra digit compared with the other entries; please make the decimal precision consistent.","section":"Table 5"},{"comment":"The ablation figure is difficult to read at the current resolution; a numerical table of the ablation results would make the comparisons verifiable.","section":"Figure 2"},{"comment":"The statement that EpiLLM-GEMMA3 demonstrates superior multi-step generation capability is contradicted by Table 2, where EpiLLM-DeepSeekR1 has lower RMSE on England 6-day and Spain 6-day; please qualify or correct the claim.","section":"Section 5.2, Table 2"},{"comment":"The notation H_{1:T} = MLP(M_{1:T}) with M_t in R^{N x N} should clarify how the MLP handles the matrix-valued input and whether spatial structure is flattened or treated graph-wise.","section":"Eq. (4)"},{"comment":"There are minor typos, e.g., \"futher forecasting\" in the Introduction and \"countires\" in Appendix A; a proofread would be helpful.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The direct-forecasting results are internally consistent and the improvements over the best baseline are non-trivial, but the paper's central selling point is the co-prediction of mobility, and that component is never validated. The multi-step table's lack of baselines and the unverified significance claim reinforce the need for revision before the results can be relied upon. I would also encourage the authors to compare against recent LLM-based time-series forecasters such as Time-LLM and Chronos, which are cited in related work but not included in the experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent adaptation of the AutoTimes/Time-LLM frozen-LLM recipe to epidemic forecasting, with a genuine new twist in the dual-branch design that co-predicts human mobility alongside case counts. The direct-forecasting numbers are consistently better than 14 baselines across four COVID datasets, so there is likely something real here. The problem is that the mobility branch—which the authors themselves call the core—is never evaluated against ground truth, and the paper's significance claims are not supported by any error bars or test detail.\n\nWhat's new and good: EpiLLM couples the epidemic branch (GNN on infection cases) with a mobility branch (MLP on mobility matrices), fuses them through autoregressive next-token prediction in a frozen LLM, and adds direction-aware prompt edges plus time gating. That combination is new. The engineering is clean, the parameter-efficiency numbers are useful, and the direct-forecasting table (Table 1) shows EpiLLM ahead of strong spatio-temporal baselines like ATMGNN on all four countries for both 3- and 7-day horizons, with RMSE improvements up to 30%. That is a meaningful result if it holds up.\n\nWhere it gets soft: the mobility branch is load-bearing. Eqs. (9)–(10) feed predicted mobility into the epidemic branch at inference, and Appendix D states that mobility prediction is the core component. Yet the paper never compares predicted mobility against held-out ground truth. The ablations in Figure 2/Table 5 show that removing or degrading mobility hurts, but that only shows the model uses the input; it does not show the predicted mobility is accurate. The mechanism the paper advertises—co-prediction enabling multi-step forecasting—is asserted, not demonstrated. This is a resolvable gap, because the ground-truth mobility is in the same dataset, but it is currently unverified.\n\nSecond, the paper says the results \"passed statistical significance tests\" but gives no test names, p-values, or error bars. The reader has to take that on faith. Third, Table 2 (multi-step) contains only EpiLLM variants—no baselines—so the multi-step superiority claim is unsupported. Fourth, the scaling-behavior claim rests on two datasets and includes a counterexample (DeepSeekR1-7B worse than the smaller Gemma-4B), which weakens the \"scaling\" phrasing.\n\nNone of this is fatal. The direct-forecasting results are consistent and the design is sensible. The paper is honest about its assumptions, even when it doesn't verify them, and there is no circularity: the model is trained on past windows and evaluated on held-out future days.\n\nWho it's for: anyone working on LLM-based time-series forecasting or on applying frozen LLMs to spatio-temporal data. It deserves a serious referee, but a revision should require (1) mobility-forecast accuracy against ground truth, (2) proper significance testing or confidence intervals, and (3) baseline comparisons for multi-step forecasting. I'd engage with it again after those are addressed.","headline":"A solid frozen-LLM epidemic forecaster whose main claimed mechanism—mobility co-prediction—is never directly validated, and whose significance claims lack statistical backing.","tokens_in":16547,"tokens_out":2248,"would_cite":false,"duration_ms":25554,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a frozen LLM, fed infection cases and human mobility as token sequences, outperforms dedicated spatio-temporal epidemic models on real-world COVID-19 forecasting.","keywords":["large language models","epidemic forecasting","spatio-temporal modeling","autoregressive forecasting","prompt learning","human mobility","COVID-19","next-token prediction"],"falsifier":"Run EpiLLM with ground-truth future mobility fed into the epidemic branch instead of predicted mobility; if 14-day RMSE does not worsen compared with the predicted-mobility version, the mobility branch is not contributing as claimed. Alternatively, compute RMSE and MAE of the predicted mobility matrices against actual future mobility at 3- and 7-day horizons.","tokens_in":15525,"feed_emoji":"🦠","tokens_out":8669,"duration_ms":81857,"temperature":0.7,"pith_summary":"This paper argues that epidemic forecasting can be treated as a language-model task: convert regional infection counts and inter-region human mobility into token sequences, and let a frozen large language model predict the next tokens. The proposed framework, EpiLLM, uses a dual-branch projector to align both signals into the LLM's embedding space, then performs autoregressive next-token prediction with learnable spatio-temporal prompts. On real-world COVID-19 datasets from England, France, Italy, and Spain, the paper reports that the best EpiLLM variant achieves the lowest RMSE and MAE for 3- and 7-day direct forecasts, with RMSE improvements over the strongest baseline up to 30.38%. If correct, this would mean a frozen general-purpose LLM, with only lightweight adapters trained, can outperform specialized epidemic models and improve further as the backbone scales.","feed_headline":"Frozen LLM out-forecasts dedicated COVID-19 models","feed_subtitle":"EpiLLM turns case counts and human movement into next-token predictions, beating baselines on four European datasets.","key_machinery":"The load-bearing mechanism is dual-branch token alignment combined with autoregressive next-token prediction. A graph neural network maps infection-case features to tokens $Z_{1:T}$, an MLP maps mobility matrices to tokens $H_{1:T}$, frozen LLM layers predict the next patch from previous tokens, and decoupled adapters map predictions back to the original feature space. Around this, spatio-temporal prompt learning adds direction-aware forward and backward edges and learnable time-gating weights $\\gamma_k$ over a token window, allowing the model to weight recent time steps more heavily. The mobility branch's predicted adjacency matrix is then fed into the epidemic branch for multi-step forecasting.","core_discovery":"The central claim is that spatio-temporal epidemic forecasting can be reformulated as next-token prediction over tokens derived from infection cases and human mobility, and that this reformulation lets a frozen LLM outperform dedicated epidemic forecasting models. The framework jointly models an epidemic branch and a mobility branch; at inference it first predicts future mobility structure, then uses those predicted mobility graphs to forecast future cases iteratively for arbitrary horizons. The paper reports direct-forecast superiority on all four datasets, successful 6- and 14-day multi-step forecasts, and scaling behavior in which larger LLM backbones generally yield lower RMSE while the fraction of trainable parameters falls to as low as 0.03%.","pith_inferences":["If the mobility branch forecasts future movement accurately, the same dual-branch recipe could transfer to other mobility-driven contagions by retraining only the lightweight projectors and adapters on new regional data.","The reported scaling trend implies a testable corollary: on longer epidemic records, larger frozen backbones should show growing relative gains over small ones, which a reader could verify before adopting the method.","A comparison against an oracle variant that receives ground-truth future mobility instead of predicted mobility would isolate how much of the multi-step gain comes from mobility prediction; the paper does not report this comparison.","The learned prompt weights suggest the model discovers a recency prior, so transferring trained prompt parameters to a new outbreak region may accelerate adaptation; that transferability is not demonstrated in the paper."],"forward_implications":["On all four COVID-19 datasets, the best EpiLLM variant records the lowest RMSE and MAE for 3- and 7-day direct forecasts; on Spain the RMSE improvement over the strongest baseline is 30.38%.","Because the framework generates mobility jointly with cases, it can produce 6- and 14-day multi-step forecasts, which the comparison models cannot do without future mobility input.","Forecast quality improves with LLM parameter scale within the GPT2 and GEMMA3 families, while the trainable-parameter ratio drops to between 0.03% and 0.58%, so larger frozen backbones give better forecasts at modest added training cost.","Learned prompt weights rise toward the current time step and forward edges dominate backward edges, consistent with the temporal-directionality prior used to initialize them.","Replacing the LLM backbone with trainable MLP, RNN, or Transformer blocks degrades performance, indicating that the pre-trained autoregressive next-token mechanism carries part of the forecasting ability."],"supporting_citations":[{"why":"Supplies the pandemic forecasting problem setup and the mobility-network-based MPNN baseline that EpiLLM extends and compares against.","marker":"[15]"},{"why":"Provides the autoregressive LLM time-series forecasting pipeline that EpiLLM adapts for epidemic tokens.","marker":"[22]"},{"why":"Supplies the temporal multiresolution graph baseline and the A VG, A VG_WINDOW, and LAST_DAY statistical benchmarks used in the comparison.","marker":"[27]"},{"why":"Presents the ATMGNN spatio-temporal transformer-GNN baseline that EpiLLM must beat in direct forecasting.","marker":"[43]"},{"why":"Gives the temporal-directionality prior used to initialize and interpret the learnable forward and backward prompt edges.","marker":"[44]"},{"why":"Presents the GEMMA3 backbone and its reduced error accumulation in long-sequence generation, which the paper cites for multi-step forecasting gains.","marker":"[47]"},{"why":"Establishes autoregressive next-token prediction as the training objective behind LLM generalization, motivating the task reformulation.","marker":"[28]"},{"why":"Supplies the prompt-learning methodology that EpiLLM extends to spatio-temporal structured data.","marker":"[30]"}],"fun_headline_variants":["Epidemic forecasting via LLM next-token prediction","Frozen LLM beats dedicated COVID-19 models","LLM forecaster uses mobility to predict case counts","Scaling LLMs improves epidemic forecast accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the mobility branch can forecast future human movement accurately enough to help infection forecasting; the paper asserts this but never measures mobility prediction error against actual mobility data.","fun_headline_variants_meta":{"raw":{"variants":["Epidemic forecasting via LLM next-token prediction","Frozen LLM beats dedicated COVID-19 models","LLM forecaster uses mobility to predict case counts","Scaling LLMs improves epidemic forecast accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00037,"raw_usage":{"total_tokens":1944,"prompt_tokens":869,"completion_tokens":1075,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":1013}},"tokens_in":485,"tokens_out":1075,"duration_ms":10607,"temperature":1.0,"reasoning_tokens":1013,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:27:33.763871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EpiLLM with ground-truth future mobility fed into the epidemic branch instead of predicted mobility; if 14-day RMSE does not worsen compared with the predicted-mobility version, the mobility branch is not contributing as claimed. Alternatively, compute RMSE and MAE of the predicted mobility matrices against actual future mobility at 3- and 7-day horizons.","supporting_citations":[{"cited_title":"Autotimes: Au- toregressive time series forecasters via large language models","cited_arxiv_id":null,"evidence_quote":"Provides the autoregressive LLM time-series forecasting pipeline that EpiLLM adapts for epidemic tokens."},{"cited_title":"Temporal multires- olution graph neural networks for epidemic prediction","cited_arxiv_id":null,"evidence_quote":"Supplies the temporal multiresolution graph baseline and the A VG, A VG_WINDOW, and LAST_DAY statistical benchmarks used in the comparison."},{"cited_title":"Predicting COVID-19 pandemic by spatio-temporal graph neural networks: A New Zealand's study","cited_arxiv_id":"2305.07731","evidence_quote":"Presents the ATMGNN spatio-temporal transformer-GNN baseline that EpiLLM must beat in direct forecasting."},{"cited_title":"What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning, pages 22964–22984","cited_arxiv_id":null,"evidence_quote":"Establishes autoregressive next-token prediction as the training objective behind LLM generalization, motivating the task reformulation."}],"review_version":1}