{"id":"c914e4bd-77b7-4d7a-a2a4-29a12fc353ef","arxiv_id":"2506.05752","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A parallel-stream LSTM with a social-connectivity-based spatial feature (SPH) is claimed to beat CDC Forecast Hub ensembles for state-level COVID-19 hospitalization forecasts.","lead":"This paper builds a deep learning model that forecasts daily COVID-19 hospitalizations in each U.S. state up to 28 days ahead, using a new feature that tracks hospitalizations in socially connected states. The authors report consistent gains over the CDC's official forecasting ensembles, especially during the Omicron surge.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation of Hub baselines may use the raw target while SLSTM is scored on the 7-day smoothed series, so the reported 27–69 hospitalization gains may reflect target smoothing rather than model skill.","rationale":"The paper's most defensible contribution is the SPH ablation (Table 1 and Fig. 5): comparing SLSTM with and without SPH on the same smoothed target and same architecture shows consistent error reduction and lower ensemble variance, so the claim that the spatiotemporal feature helps is plausible. The central quantitative claim against the Hub, however, is insecure because of the target mismatch. The reader's weakest assumption pinpoints exactly this issue, and the manuscript's own §3.5 sentence ('we selected smoothed hospitalizations as the evaluation target instead of raw values') makes the problem explicit rather than an artifact of review. The fix is straightforward: score all models on the identical transformed series, or report both raw- and smoothed-scale results, and add uncertainty quantification for the differences. Because this condition is testable from the released code and the rest of the evaluation design (walk-forward training, spatial cross-validation, quantile-loss evaluation) is sound, the verdict remains CONDITIONAL rather than REJECT. No concerns about author integrity or internal consistency are raised.","tokens_in":30865,"tokens_out":6212,"duration_ms":56533,"concrete_test":"Using the released repository (https://github.com/geohai/covid-lstm-hosp), inspect the evaluation script to determine the target series used for each model. Then recompute the Omicron daily MAE (Table C3) with both SLSTM and COVIDhub-4week ensemble forecasts scored against the same smoothed ground truth: apply the 7-day trailing average used in §3.5 to the Hub ensemble point forecasts and compute MAE against the smoothed observations; also compute both models' MAE against the raw daily series (for SLSTM, use its raw-scale predictions if available, or retrain on raw targets). If the day-7/14/21/28 differences fall materially below the reported 27/42/54/69 (e.g., toward the inter-week variability in Table C2), the central outperformance claim is not robust to the evaluation target.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim—outperformance of COVIDhub ensembles by 27–69 hospitalizations per state during Omicron (abstract; Table 3; Table C3)—depends on both models being scored on the same target series. Section 3.5 explicitly states: 'we selected smoothed hospitalizations as the evaluation target instead of raw values', and the model is trained on 7-day rolling averages of cases and hospitalizations. The Hub ensembles, however, were designed and evaluated operationally on raw daily incident hospitalizations (Cramer et al. 2022a). The paper never states that the Hub forecasts were transformed to the smoothed scale (e.g., by applying the same 7-day trailing average to their predicted trajectories) before computing MAE/WIS in Tables 3 and C3. Nor does Section 3.6 define y_i,t in Eq. (7) as smoothed or raw. A 7-day average reduces the volatility of a time series; scoring SLSTM on this smoother target while scoring Hub models on raw data inflates the apparent advantage. The SPH ablation (Table 1) is internal and compares models on the same target, so it is not affected by this mismatch; the load-bearing concern is specifically the cross-model comparison against the Hub.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SLSTM, a parallel-stream LSTM architecture for forecasting daily state-level COVID-19 hospitalizations in the United States. Two LSTM branches process 7-day and 28-day histories and are fused with a learnable weight; the model also uses Social Proximity to Hospitalizations (SPH), a spatial feature built from Meta's Social Connectedness Index and hospitalization rates in socially connected states. A multi-horizon ensemble combines 7-, 14-, and 28-day output members. The model is evaluated on Delta and Omicron waves against COVIDhub baselines and ensembles, reporting consistent improvements, with the Omicron headline claim of 27, 42, 54, and 69 hospitalizations per state at 7-, 14-, 21-, and 28-day horizons. The authors also report an ablation showing SPH improves performance and reduces ensemble variance.","tokens_in":31126,"tokens_out":5550,"duration_ms":61062,"significance":"The manuscript makes a concrete, falsifiable claim: a spatially informed LSTM with SPH improves hospitalization forecasts over the COVID-19 Forecast Hub ensembles, and the ablation is internally sound. The open-source code release and the explicit evaluation on two surge periods are strengths. The circularity concern sometimes raised about SPH does not apply: SPH at time t uses observed hospitalization rates in connected states while the target is future hospitalizations in the focal state. If the head-to-head comparison is confirmed on a common target scale, the paper would be a useful contribution to operational epidemic forecasting and to the use of social-connectivity data in spatial epidemiology.","major_comments":[{"comment":"The headline comparison against the COVIDhub ensembles may be evaluated on different target series. Section 3.5 states: \"we selected smoothed hospitalizations as the evaluation target instead of raw values,\" and the model is trained on 7-day rolling averages. The Hub ensembles, however, were designed and operationally evaluated on raw daily incident hospitalizations. The paper does not state whether the Hub forecast trajectories were transformed to the smoothed scale (for example, by applying the same 7-day trailing average) before computing MAE and WIS in Tables 3 and C3, and Equation (7) does not specify whether y_i,t is smoothed or raw. Since smoothing reduces volatility, scoring SLSTM on a smoothed target while scoring Hub models on raw data would inflate the apparent advantage. The authors must either confirm that all baselines were re-evaluated on the same smoothed target or rerun the comparison on a common target series.","section":"§3.5, §3.6, Tables 3 and C3"},{"comment":"The superiority claims rest on average errors over only 10 Omicron and 15 Delta forecast dates, but no uncertainty quantification is provided. The 27–69 hospitalization differences are reported as point estimates without confidence intervals or paired significance tests. Because these averages are taken over a small number of weekly forecasts, the authors should add a paired test across forecast dates (e.g., Wilcoxon signed-rank or Diebold-Mariano) or bootstrap confidence intervals to establish whether the improvements are statistically reliable rather than driven by one or two forecast weeks.","section":"§4.3, Tables 3 and C3"},{"comment":"The paper does not specify whether the 7-day rolling average applied to cases, hospitalizations, and the evaluation target is trailing or centered. If the average is centered, the smoothed target at horizon k would incorporate future observations relative to the forecast date, which would constitute lookahead leakage. Please state explicitly that a trailing (past-7-day) average is used, or correct the procedure accordingly.","section":"§3.5"}],"minor_comments":[{"comment":"The abstract says \"surpasses the ensemble by 27, 42, 54, and 69 hospitalizations,\" but Table 3 reports two ensemble models and the specific numbers correspond to the COVIDhub-4 week ensemble. Please specify which ensemble is meant in the abstract.","section":"Abstract and §4.3"},{"comment":"The text refers to \"four subgroups of such 7-day forecast output models,\" but the description indicates four individual 7-day models with different seeds, not subgroups. Please rephrase for clarity.","section":"§3.4"},{"comment":"Please clarify whether the set C in the SPH denominator excludes the focal state i, and whether the row-normalization is computed over all other 50 states or over a subset of connected states.","section":"Equation (4), §3.1"},{"comment":"The paper does not explicitly describe the inverse transform from the MinMax-scaled rates back to hospitalization counts used in the reported MAE and WIS tables. Since the code is released, this is probably available, but an explicit equation or a pointer to the relevant code block would aid reproducibility.","section":"§3.1 and §3.5"}],"recommendation":"major_revision","confidential_remarks":"The key risk is the target-scale mismatch in the head-to-head comparison. If the Hub baselines were scored on raw data while SLSTM was scored on smoothed data, the abstract's 27–69 hospitalization improvements would be overstated, potentially by a large margin. The authors should be asked to document exactly how Hub forecasts were transformed before scoring. Adding uncertainty quantification would also strengthen the claims substantially."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper has one genuinely solid contribution and one big loose end. The solid part: the SPH feature—a Social Connectedness Index-weighted lag of hospitalization rates in connected states—is new for hospitalization forecasting, and the ablation (Table 1, Table C1) shows it helping beyond a no-spatial LSTM and actually hurting when SPC is added, which is a sensible and interesting story. The multi-horizon ensemble and the open code are also useful. I read the SPH ablation as internally valid: the feature is a spatial lag of current hospitalizations, the target is future hospitalizations, so it is not circular, and the learnable weight and hyperparameters are normal model fitting. That said, the evidence for SPH is limited to the Omicron period, so the generality claim is weaker than the authors suggest.\n\nThe loose end is the headline comparison against the COVIDhub ensembles. The paper explicitly says it selected smoothed hospitalizations as the evaluation target instead of raw values (Section 3.5). That is fine for the SLSTM model, which is trained on the smoothed series. But I cannot find any statement that the Hub forecasts—which were designed and operationally scored on raw daily hospitalization reports—were transformed to the smoothed scale before computing the MAE/WIS in Tables 3 and C3. If they were not, the reported 27, 42, 54, and 69 hospitalization gains are partly an artifact of scoring one model on a 7-day smoothed target and the others on raw daily data. That is the load-bearing claim of the abstract, and it needs to be verified. The fix is straightforward: recompute the comparison with matched targets, or state clearly that both sides were smoothed. This is not a fatal flaw—the SPH ablation stands on its own—but the paper should not claim superior performance over the Hub until this is settled.\n\nOther soft spots: no significance tests, and the ensemble ablation only compares the multi-horizon strategy to its own 28-day direct model, not to other standard multi-step strategies. These are minor relative to the target-matching issue.\n\nWho is this for? Epidemiologic forecasters and geospatial machine learning researchers. It deserves a serious referee because the feature is reusable and the evaluation problem is fixable. I would engage with it, but only after the matched evaluation is done.","headline":"Solid, reusable SPH feature and a well-executed internal ablation; the headline Hub comparison is on a possibly mismatched target and needs a matched-target re-run before the outperformance claim is taken seriously.","tokens_in":31659,"tokens_out":2086,"would_cite":false,"duration_ms":24814,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A parallel-stream LSTM with a social-connectivity spatiotemporal feature forecasts 28-day COVID-19 hospitalizations more accurately than COVID-19 Forecast Hub ensembles, with per-state gains of 27 to 69 during Omicron.","keywords":["COVID-19","hospitalization forecasting","LSTM","spatiotemporal feature","Social Connectedness Index","spatial spillover","quantile regression","multi-horizon ensemble"],"falsifier":"Re-score the stored COVIDhub-4 week ensemble forecasts on the same 7-day smoothed state-level hospitalization series used for SLSTM, matching the smoothing window and evaluation weeks, then compare MAE and WIS to SLSTM's reported numbers; if the ensemble's smoothed-target error meets or beats SLSTM's, the claimed margin disappears.","tokens_in":30663,"feed_emoji":"🏥","tokens_out":8204,"duration_ms":73706,"temperature":0.7,"pith_summary":"This paper tries to establish that a deep-learning model can forecast state-level COVID-19 hospitalizations up to 28 days ahead more accurately than the official forecasting ensembles, if it is given a feature that summarizes how hospitalizations in socially connected states are moving. The feature, called Social Proximity to Hospitalizations (SPH), is a weighted average of other states' hospitalization rates using Facebook friendship-connection weights, and it is intended to capture interstate transmission. The model, SLSTM, is a pair of stacked LSTM networks that read 7-day and 28-day input windows in parallel and fuse them with a learnable weight, then combine 7-, 14-, and 28-day output models into a 15-member ensemble. On the $\\Delta$ and Omicron surges, the paper reports average per-state improvements over the COVID-19 Forecast Hub ensembles that grow with horizon, reaching 69 hospitalizations per state at the 28-day horizon during Omicron. The claim matters because hospitalization forecasts are used for staffing and resource allocation, and the Omicron surge was precisely the period when existing models struggled most.","feed_headline":"Social-connection data beat COVID forecast ensembles","feed_subtitle":"Adding a social-connectivity feature to an LSTM cuts 28-day hospitalization forecast error by up to 69 per state.","key_machinery":"The load-bearing object is SPH, the Social Proximity to Hospitalizations feature, defined as the row-normalized Social Connectedness Index-weighted sum of hospitalization rates in connected states. It converts a static matrix of interstate friendship strengths into a daily time series that carries spatial spillover into each state's LSTM input. The other pieces are the parallel-stream architecture, with two stacked LSTM networks processing 7-day and 28-day windows fused by a learnable scalar weight, the multi-horizon ensemble of 15 members with staggered 7-day, 14-day, and 28-day output horizons aggregated by median, and quantile loss with 23 probability levels for probabilistic forecasts. Together these let the model balance short- and long-term temporal dependence, smooth over stochastic training runs, and emit prediction intervals evaluated by the Weighted Interval Score.","core_discovery":"The paper's central claim is that spatiotemporal information derived from social connectivity can be turned into a scalar feature that materially improves deep-learning hospitalization forecasts. SPH is computed as a row-normalized, Social Connectedness Index-weighted sum of hospitalization rates in all connected states, so each state's model input carries a signal of what is happening in states whose populations interact with it. The paper argues that adding SPH to a parallel-stream LSTM, called SLSTM, reduces average 28-day MAE by roughly 363 hospitalizations per state compared with the same model without SPH, reduces forecast variance across ensemble members, and beats the COVID-19 Forecast Hub baseline and both Hub ensemble models on MAE, MAPE, RMSE, and WIS during both the $\\Delta$ and Omicron evaluation windows. The paper also claims that combining 7-day, 14-day, and 28-day output models by taking the median across 15 members improves on a single 28-day direct model by about 167 hospitalizations per state on average.","pith_inferences":["The comparison to Hub ensembles is only fully fair if those ensembles are re-scored on the same 7-day smoothed hospitalization series; the paper does not state that this was done, so the reported margins may partly reflect a smoothing advantage.","Because the Social Connectedness Index is static and freely available, SPH could be built with other connectivity proxies such as migration or cell-phone mobility where Facebook data are unavailable; a direct test would be to permute the connectivity weights and see whether the gains survive.","The multi-horizon median ensembling idea likely generalizes to any high-noise multi-step forecasting task, independent of epidemiology.","A more direct test of the mechanism would compare SPH against an equally informative non-spatial feature, like a lagged national average, to check whether spatial structure rather than extra information is what drives the improvement."],"forward_implications":["With SPH, the model improves average 28-day MAE by about 363 hospitalizations per state versus the same model without SPH, and the spatial model's variance across ensemble members is consistently lower.","Using a multi-horizon ensemble instead of a single 28-day direct output improves average MAE by 167 hospitalizations per state over the Omicron evaluation period.","SLSTM outperforms the COVID-19 Forecast Hub ensembles on both Delta and Omicron, with the Omicron advantage growing from roughly 15 hospitalizations per state on day 2 to about 64 per state on day 23.","SLSTM's 95% prediction intervals contain 90.2% of actual values, versus 68.6% and 70.2% for the two Hub ensembles, and its WIS is lowest in both waves.","The same framework is presented as transferable to influenza-style forecasting hubs and to other multi-step time-series problems such as air quality, weather, and energy demand."],"supporting_citations":[{"why":"Defines the Social Connectedness Index and shows it measures the strength of social ties between regions, which the paper uses to build SPH.","marker":"Bailey et al. 2018"},{"why":"Introduces the Social Proximity to Cases metric, which the paper extends to hospitalizations as SPH.","marker":"Kuchler et al. 2022"},{"why":"Defines the COVID-19 Forecast Hub dataset, the state-level 28-day hospitalization forecasting target, and the benchmark ensembles.","marker":"Cramer et al. 2022a"},{"why":"Provides the LSTM architecture on which the parallel-stream SLSTM stacks are built.","marker":"Hochreiter and Schmidhuber 1997"},{"why":"Establishes the interval-forecast evaluation and weighted median ensemble approach used for the Hub comparisons.","marker":"Bracher et al. 2021"},{"why":"Supplies the WIS evaluation protocol for probabilistic forecasts used to score SLSTM and baselines.","marker":"Cramer et al. 2022b"},{"why":"Shows SCI-derived features are competitive with mobility data for COVID-19 case forecasting, motivating the spatial feature here.","marker":"Vahedi et al. 2021"},{"why":"Demonstrates spatiotemporal machine learning for county-level case forecasting, the approach this paper extends to state-level hospitalizations.","marker":"Lucas et al. 2023"},{"why":"Provides the scoringutils WIS implementation used to compute the reported probabilistic scores.","marker":"Bosse et al. 2023"},{"why":"Documents the difficulty of COVID-19 case forecasting in the U.S., used to motivate the need for better hospitalization forecasts.","marker":"Lopez et al. 2024"}],"fun_headline_variants":["Social proximity cuts COVID forecast errors per state","Social ties sharpen LSTM hospital forecasts","Social connectedness tops COVID forecast ensembles","Spatiotemporal social data boosts COVID forecasts","Social signals reduce COVID hospitalization forecast error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim assumes the comparison models were scored on the same smoothed 7-day hospitalization series that SLSTM was trained and evaluated on, whereas the paper states only that it chose smoothed hospitalizations as its evaluation target and does not say the Hub forecasts were re-scored on that same series.","fun_headline_variants_meta":{"raw":{"variants":["Social proximity cuts COVID forecast errors per state","Social ties sharpen LSTM hospital forecasts","Social connectedness tops COVID forecast ensembles","Spatiotemporal social data boosts COVID forecasts","Social signals reduce COVID hospitalization forecast error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1637,"prompt_tokens":1001,"completion_tokens":636,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":572}},"tokens_in":617,"tokens_out":636,"duration_ms":7405,"temperature":1.0,"reasoning_tokens":572,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:12:34.472471+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the stored COVIDhub-4 week ensemble forecasts on the same 7-day smoothed state-level hospitalization series used for SLSTM, matching the smoothing window and evaluation weeks, then compare MAE and WIS to SLSTM's reported numbers; if the ensemble's smoothed-target error meets or beats SLSTM's, the claimed margin disappears.","supporting_citations":[],"review_version":1}