{"id":"64095937-4b81-482e-a984-04967cc0a511","arxiv_id":"2507.10014","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A graph neural network with a Transformer is applied to Valley Fever forecasting in Arizona, but the evaluation lacks baselines and shows unexplained relative errors above 1.","lead":"This paper applies a graph neural network plus Transformer to forecast weekly Valley Fever cases in Maricopa County, Arizona, using environmental and air quality data. It reports mean absolute percentage errors of 13% to 23% for 2- to 16-week forecasts, but provides no baseline comparison and reports unexplained relative errors above 1.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported Root Relative Squared Error (7.55–1.47) contradicts the claimed forecasting accuracy; if computed in the standard way, the model beats neither the test mean nor likely naive baselines, so the central claim is unsupported until this is resolved.","rationale":"The reader's formal weakest_assumption concerns graph construction, and that is a real limitation (linear Pearson correlations, 0.05 threshold, 6-week max lag), but the strongest single threat to the central claim is internal: the reported RSE numbers are not merely unexplained, they are numerically inconsistent with the reported MAPE/MAE and, under the standard formula, show a model that loses to the mean predictor. That directly attacks the claim of effective forecasting, whereas the graph concern attacks mechanism attribution. I agree with the reader's REJECT direction and add a sharper, testable failure mode. The paper does provide a repository, which is a credit, but until the RSE issue and baseline comparisons are resolved the MAPE values are uninterpretable; the 49-week train/test gap and the feature-count inconsistency in Table 1 (23 listed features vs '19 variables') reinforce the need for a corrected and baseline-validated revision.","tokens_in":11898,"tokens_out":4342,"duration_ms":49260,"concrete_test":"Using the released GitHub repository, recompute Table 2 from the saved predictions: compute RSE with denominator Σ(y_t−ȳ_test)², and add persistence (ŷ_t = y_{t−h}), seasonal-naive (same epi-week previous year), and ARIMA baselines on the same weeks 900–991. If RSE remains >1 or any baseline beats the model on MAPE, the central claim fails; if the reported RSE values prove to be typos or a nonstandard denominator, the corrected metrics should be reported with baselines before any claim of effectiveness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 reports RSE = 7.55, 2.25, 1.73, 1.47 for the 2/4/8/16-week horizons while claiming MAPE 0.13–0.23. Under the usual definition RSE = sqrt(Σ(y−ŷ)² / Σ(y−ȳ)²) with ȳ the test-period mean, RSE > 1 means the model is worse than always predicting ȳ. For the 2-week row, MSE = 1018.83 gives RMSE ≈ 31.9; an RSE of 7.55 implies the mean predictor's RMSE is only ~4.2, which is inconsistent with MAPE = 0.13 and MAE = 24.35 on the same count scale unless the variance of the target is implausibly small or the metrics were computed on different data. The paper gives no definition of RSE, no baseline comparisons, and no raw prediction curves or error decomposition. Since the headline claim is 'effectively models Valley Fever trends' and provides 'early warning up to four months', the possibility that the model is worse than a trivial mean or persistence baseline is directly load-bearing. The RSE anomaly is compounded by an unexplained train/test gap: training through week 850 and testing on weeks 900–991 leaves 49 weeks unaccounted for, so the test period may not be contiguous with training.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a GNN-Transformer hybrid for forecasting weekly Valley Fever incidence in Maricopa County, Arizona. It constructs a variable graph from Pearson correlations thresholded at 0.05, includes lagged features up to six weeks, prunes inputs to the top 10% via a learned feature gate, applies GATv2 layers, and uses a Transformer encoder-decoder. The model is evaluated on forecast horizons of 2, 4, 8, and 16 weeks over epi-weeks 900-991, reporting MAPE between 0.13 and 0.23, MAE between 24.35 and 42.71, MSE between 1018.83 and 3075.13, and RSE between 7.55 and 1.47. The abstract and conclusion claim that the model effectively models Valley Fever trends, establishes the first successful graph-based deep learning approach for the disease, and provides early warning up to four months ahead.","tokens_in":12219,"tokens_out":5903,"duration_ms":66105,"significance":"The problem is practically relevant, and the authors provide code and data, a fixed-model walk-forward evaluation, and a 100-seed feature-importance stability analysis, all of which are commendable. If the reported accuracy were valid and supported by baselines, a variable-graph GNN could be a useful tool for public-health early warning. However, the evidence as presented does not support the headline claims: the RSE values contradict the other reported metrics under the standard definition, no baseline comparisons or error bars are provided, a 49-week gap exists between the training and test windows, and the claimed dynamic lag learning is not implemented. These issues are load-bearing because the paper's central contribution is the empirical forecasting claim.","major_comments":[{"comment":"The RSE column is undefined, and the reported values contradict the other metrics under the standard definition. If RSE = sqrt(Σ(y-ŷ)² / Σ(y-ȳ)²), then every reported value exceeds 1, meaning the model is worse than always predicting the test-period mean. For the 2-week row, MSE=1018.83 gives RMSE≈31.9, so RSE=7.55 implies a mean-predictor RMSE of only ≈4.2, which is inconsistent with MAE=24.35 and MAPE=0.13 unless the metrics were computed on different scales or different samples. The paper must define RSE, report the test-period mean and variance, and provide at least mean, persistence, and standard time-series baselines; without this, the central claim of accurate forecasting is unsupported.","section":"Section 4.5, Table 2"},{"comment":"The train/test split is under-specified. Training is described as including all weeks up to week 850, while testing is conducted on weeks 900-991, leaving 49 weeks (851-899) unaccounted for. If those weeks were excluded, the test period is not a simple continuation of the training window and the out-of-sample protocol changes; if they were used, the description is incorrect. This matters for the validity of the claimed out-of-sample performance, and the authors should explain why these weeks are missing or correct the description.","section":"Section 4.3"},{"comment":"The method does not learn lag structures dynamically. Lagged features are generated only up to a fixed maximum of 6 weeks in Section 4.2, and the graph is built once from Pearson correlations thresholded at 0.05 in Section 3.1; no mechanism adapts the lag window or edge structure during training. The statement in Section 2.3 that graph architectures can learn optimal lag structures dynamically, and the conclusion's implication that the model captures critical delays through learned lagged effects, are therefore not supported by the implemented architecture. The authors should either implement an adaptive lag mechanism or explicitly temper these claims.","section":"Sections 3.1 and 4.2"},{"comment":"Forecast metrics are reported as point estimates from a single 92-week test window with no error bars or significance tests. The 100-seed analysis in Section 4.4 evaluates feature-importance stability only and does not quantify prediction uncertainty. The overlapping 16-week samples in Figure 4 (weeks 900-915, 901-916, 902-917, 903-918) are not independent, so the visual confirmation is weaker than it appears. The authors should report mean and standard deviation across seeds or bootstrap confidence intervals and test whether differences from baselines are statistically significant.","section":"Sections 4.4 and 4.5"},{"comment":"There is no ablation isolating the contribution of the graph components. The paper does not compare the full model against ablated versions such as a Transformer without the graph module, a GAT without the feature gate, or a simple MLP using the same lagged features. Without such comparisons, the claimed benefits of the graph structure and feature selection cannot be evaluated, and the title claim of a 'Graph Neural Network approach' being responsible for the results is not established.","section":"Section 4.5"}],"minor_comments":[{"comment":"The affiliation line lists 'Hao Yana'; this is likely a typo for 'Hao Yan' and should be corrected.","section":"Author list"},{"comment":"The reverse-differencing formula is written for h = 0, ..., H-1, which would reconstruct the target at time t for h=0; clarify whether the forecast horizon begins at t+1 and adjust the indexing accordingly.","section":"Equation (9)"},{"comment":"The sentence 'A forecasting model was trained on data from the in-sample period ending one epidemiological week prior to the rolling test set' is ambiguous about whether the model is retrained for each rolling window, while the next sentence says model weights remained fixed; state explicitly which protocol was used.","section":"Section 4.5"},{"comment":"The four displayed 16-week samples are overlapping and therefore not independent evidence; the text should acknowledge this and avoid implying they are four separate validation checks.","section":"Figure 4"},{"comment":"The reference for the 2024 Arizona case count cites a non-peer-reviewed travel website (Vax-Before-Travel); the Arizona Department of Health Services or CDC primary data should be cited instead.","section":"References"},{"comment":"Grant DMS-1615879 is a National Science Foundation grant number, not an NIH grant; the acknowledgments should be corrected.","section":"Acknowledgments"}],"recommendation":"reject","confidential_remarks":"The RSE inconsistency in Table 2 is severe enough that I cannot recommend revision in its current form. Even if the RSE was mislabeled, the absence of any baseline comparison and the unexplained 49-week gap mean the central empirical claim is not currently supported. If the authors can show that the RSE is an artifact and provide proper baselines and ablations, a resubmission could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a genuine first application of a graph neural network to coccidioidomycosis forecasting, with code and data on GitHub, a 100-seed feature-stability analysis, and a sensible environmental feature set. But the paper as written should not be accepted. The reported Root Relative Squared Errors are not consistent with the other metrics, and without baselines the MAPE claims mean little.\n\nWhat is new and credible: the GATv2 + feature gate + Transformer combination is new for Valley Fever. The graph uses correlation threshold 0.05 and six-week lags, the feature gate retains the top 10% of features, and they evaluate 2-, 4-, 8-, and 16-week horizons. The feature importance analysis across 100 seeds is a legitimate piece of work, and the GitHub repo is a real reproducibility plus.\n\nWhere it falls down: Table 2 reports RSE = 7.55 at the 2-week horizon alongside MSE = 1018.83 and MAE = 24.35. Under the standard definition RSE = sqrt(MSE_model / MSE_mean), so RSE > 1 means the model is worse than predicting the test-period mean. At 2 weeks, RSE 7.55 implies the mean predictor has RMSE around 4.2 cases, which is implausible when MAPE = 0.13 and MAE = 24.35 on the same scale. The authors never define RSE, never compare against persistence, seasonal means, ARIMA, or even a trivial mean forecast. The central claim that the model 'effectively models Valley Fever trends' is therefore unsupported as presented.\n\nThere are also fixable but real problems: an unexplained 49-week gap between training through week 850 and testing on weeks 900-991; a fixed six-week maximum lag that contradicts the phrase 'dynamic lag learning'; and an arithmetic slip where the text says 19 features but the table lists 23. These reinforce the impression that the evaluation needs a careful rewrite.\n\nBottom line: the RSE issue is load-bearing, but the paper deserves a serious referee rather than a desk reject. The application is real, the data work is there, and the flaws are addressable. A revised version with baselines, error bars, a defined RSE, and a contiguous train/test split could change my verdict.","headline":"First GNN for Valley Fever is a real domain application, but the headline numbers are not credible until the RSE anomaly and missing baselines are resolved.","tokens_in":12743,"tokens_out":3261,"would_cite":false,"duration_ms":36221,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","92D30"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims the first graph neural network for Valley Fever forecasting, with weekly incidence errors of 13% at two weeks and 23% at sixteen weeks in Maricopa County.","keywords":["Valley fever","coccidioidomycosis","graph neural networks","multivariate time series forecasting","public health early warning","environmental epidemiology","Transformer","feature selection"],"falsifier":"Run the same walk-forward protocol with all graph edges set to zero (identity adjacency) and without the feature gate. If the MAPE at 2 and 16 weeks does not degrade materially, the graph structure is not carrying the claimed signal; if a simple baseline such as last year's same-week count beats the 16-week MAPE of 0.23 on weeks 900-991, the early-warning claim fails.","tokens_in":11673,"feed_emoji":"🦠","tokens_out":12171,"duration_ms":111622,"temperature":0.7,"pith_summary":"This paper tries to establish that a graph neural network combined with a Transformer can forecast weekly Valley Fever cases in Maricopa County, Arizona, up to sixteen weeks ahead, with a mean absolute percentage error of 13 percent at two weeks and 23 percent at sixteen weeks. The authors build a graph whose nodes are environmental and surveillance variables, edges weighted by absolute Pearson correlation above a 0.05 threshold, and let a graph attention module prune to the top 10 percent of features before a Transformer models their time dynamics. If correct, the model would give public-health agencies an automated early-warning signal that accounts for delayed environmental effects without manual feature selection. The paper also claims this is the first successful use of graph-based deep learning for coccidioidomycosis.","feed_headline":"Graph AI forecasts Valley Fever up to four months ahead","feed_subtitle":"First GNN for the fungal disease keeps 2-week MAPE at 13%, 16-week at 23%, and flags rising cases early.","key_machinery":"The engine is a correlation graph over variables: each node is an original or up-to-six-week-lagged feature, and the edge weight between series u and v is $|\\rho_{uv}|$ when that absolute Pearson correlation is at least 0.05, otherwise zero. A trainable Feature Gate keeps the top 10 percent of node gates on every forward pass, pruning 90 percent of variables. GATv2 layers then compute source-target attention weights $\\alpha_{ij}$ over each node's correlated neighborhood, and a Transformer encoder-decoder with positional encodings reads the resulting embeddings to produce multi-step forecasts of differenced case counts, which a reverse-differencing layer converts back to case numbers.","core_discovery":"The central discovery is that relational structure among environmental predictors can carry Valley Fever forecasting: the GATv2-on-correlation-graph plus Transformer encoder-decoder reports MAPE of 0.13, 0.16, 0.21, and 0.23 for 2-, 4-, 8-, and 16-week horizons over test weeks 900-991, after training only through week 850. The feature gate selected the same 15 of roughly 190 variables in the top 10 percent across all four horizons, with 20-inch soil temperature maximum at lag 0 ranked first everywhere, followed by minimum relative humidity and six-week-lagged PM10. The paper interprets the stable selection and visual inspection of rolling 16-week samples as evidence that the model captures both short-term fluctuations and longer-term epidemiological trends, including the sharp rise around week 906.","pith_inferences":["Editorial extension: a direct ablation that zeros all graph edges would reveal whether the graph structure, rather than the Transformer alone, is responsible for the forecast skill; the paper does not report such a comparison.","Editorial extension: the paper tests only Maricopa County, so whether the same graph recipe transfers to Pima County or other endemic regions remains open; that transfer test is the natural next step.","Editorial extension: the 0.05 correlation threshold and six-week lag cap are modeling choices, not tested results; sweeping both would show whether the identified features are stable or artifacts of the cutoff.","Editorial extension: because no ARIMA or LSTM baseline appears in the experiments, \"first successful implementation\" is a novelty claim rather than a superiority claim; a head-to-head would put the MAPE values in context."],"forward_implications":["At the 16-week horizon the model keeps MAPE at 0.23, so a health department could use it as a directional early-warning system, spotting rising or falling trends up to four months before cases arrive.","Because 15 features rank in the top 10 percent at every horizon, the model identifies soil temperature, humidity, and PM10 as stable environmental drivers worth monitoring across seasons.","The automatic feature gate reduces the input to 10 percent of variables, so the pipeline can be applied to new data without manual feature selection.","The walk-forward evaluation protocol, with training ending at week 850 and testing on weeks 900-991 with fixed weights, provides an out-of-sample check across horizons, though it does not retrain the model during the test period."],"supporting_citations":[{"why":"Establishes that seasonal precipitation predicts Valley Fever incidence in Arizona, grounding the environmental-input design.","marker":"Tamerius and Comrie, 2011"},{"why":"Identifies climate factors behind coccidioidomycosis seasonality and outbreaks, motivating the weather and soil variables.","marker":"Comrie, 2005"},{"why":"Quantifies precipitation, heat, and drought effects on incidence, supplying the ecological rationale for lagged environmental predictors.","marker":"Head et al., 2022"},{"why":"Documents climate-driven expansion of the endemic region and rising case counts, framing the forecasting need.","marker":"Gorris et al., 2019"},{"why":"Provides the graph neural network formalism and terminology adopted for variable-relation modeling.","marker":"Wu et al., 2020"},{"why":"Shows GNNs applied to epidemic forecasting, the line of work this paper extends to Valley Fever.","marker":"Kapoor et al., 2020"},{"why":"Reviews GNN epidemic models and variable-centric graph construction, supporting the architecture's design choices.","marker":"Liu et al., 2024"}],"fun_headline_variants":["GNN predicts Valley Fever up to 16 weeks ahead","Graph AI forecasts Valley Fever using soil and air data","Valley Fever forecast with graph neural networks","New GNN model flags Valley Fever risk months early","AI graph model sees Valley Fever trends months ahead"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that absolute Pearson correlations above 0.05, computed on original and six-week-lagged variables, capture the environmentally relevant dependencies; if the true relationships are nonlinear, nonstationary, or operate on longer lags, the network may be fitting noise or partial signals despite acceptable-looking test error.","fun_headline_variants_meta":{"raw":{"variants":["GNN predicts Valley Fever up to 16 weeks ahead","Graph AI forecasts Valley Fever using soil and air data","Valley Fever forecast with graph neural networks","New GNN model flags Valley Fever risk months early","AI graph model sees Valley Fever trends months ahead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1660,"prompt_tokens":860,"completion_tokens":800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":726}},"tokens_in":476,"tokens_out":800,"duration_ms":9494,"temperature":1.0,"reasoning_tokens":726,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:42:10.382157+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same walk-forward protocol with all graph edges set to zero (identity adjacency) and without the feature gate. If the MAPE at 2 and 16 weeks does not degrade materially, the graph structure is not carrying the claimed signal; if a simple baseline such as last year's same-week count beats the 16-week MAPE of 0.23 on weeks 900-991, the early-warning claim fails.","supporting_citations":[{"cited_title":"Coccidioidomycosis incidence in arizona predicted by seasonal precipitation","cited_arxiv_id":null,"evidence_quote":"Establishes that seasonal precipitation predicts Valley Fever incidence in Arizona, grounding the environmental-input design."},{"cited_title":"Climate factors influencing coccidioidomycosis seasonality and outbreaks","cited_arxiv_id":null,"evidence_quote":"Identifies climate factors behind coccidioidomycosis seasonality and outbreaks, motivating the weather and soil variables."},{"cited_title":"Effects of precipitation, heat, and drought on incidence and expansion of coccidioidomycosis in western usa: a longitudinal surveillance study","cited_arxiv_id":null,"evidence_quote":"Quantifies precipitation, heat, and drought effects on incidence, supplying the ecological rationale for lagged environmental predictors."},{"cited_title":"Expansion of coccidioidomycosis endemic regions in the united states in response to climate change","cited_arxiv_id":null,"evidence_quote":"Documents climate-driven expansion of the endemic region and rising case counts, framing the forecasting need."}],"review_version":1}