{"id":"b3207599-7266-4403-923a-58bcd2ecb14a","arxiv_id":"2501.14980","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Frequency-tuned state space models (S4D-FT) give slightly better median NSE and KGE than LSTM for rainfall-runoff prediction at 531 US watersheds.","lead":"A state space model called S4D-FT, previously used for long-sequence machine learning, was trained to predict daily streamflow from weather and landscape data at 531 US watersheds. It beat the standard LSTM model on median accuracy metrics, though it performed worse on low-flow volume bias and total volume bias.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Manual tuning of S4D-FT without a reported validation split is the load-bearing gap: the small NSE/KGE gains over LSTM may be test-set selection bias rather than an architectural SSM advantage.","rationale":"The paper is a useful, reproducible application: the 531-CAMELS standard setup, eight-member ensembles, public code and data, and LSTM numbers that match prior literature are real strengths. The S4D-FT construction itself is not the issue; the issue is the evidential link from configuration to architectural conclusion. The reported median gains (NSE +0.02, KGE +0.01) are small enough that a modest amount of test-set selection among manual trial-and-error runs could explain them. The authors disclose the trial-and-error tuning in Section 2, so this is not a hidden flaw, but the absence of any validation split means the core comparison is not yet airtight. My proposed test is operational: rerun the S4D-FT tuning under a pre-registered validation split and compare with LSTM on the untouched test period. If the advantage survives, the conditional verdict can be upgraded; if it disappears, the claim should be downgraded. I therefore keep the reader's conditional verdict.","tokens_in":14975,"tokens_out":6948,"duration_ms":59866,"concrete_test":"Check whether the released repository (github.com/WESTENR-OU/S4D_rainfall_runoff_simulations) contains any train/validation split before the test period is accessed. If it does not, run a nested tuning experiment: use water years 1999–2005 for training and 2006–2008 as validation, optimize cfi, cfr, and the other S4D-FT hyperparameters on validation median NSE only, freeze the selected settings, then evaluate on the untouched 1989–1999 test period with the same eight seeds. Compare the resulting median NSE/KGE with a matched LSTM trained under the identical split. If the validation-tuned S4D-FT no longer beats LSTM by more than the ensemble standard error, the reported advantage is attributable to selection on the test period.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—S4D-FT outperforms LSTM and 'sets a new community-based standard'—rests on Table 1, where S4D-FT beats LSTM by median NSE 0.74 vs. 0.72 and KGE 0.75 vs. 0.74. Section 2 states that S4D and S4D-FT hyperparameters are 'manually tunned through a trial-and-error process,' with only final values provided in Tables S2 and S3. No validation split or selection criterion is reported for the S4D-FT-specific settings cfi=10 and cfr=10. If these were chosen by inspecting the 1989–1999 test period (the same period used for the LSTM benchmark), the observed gain could be test-set selection bias rather than an architectural advantage of state-space models. Since the paper's conclusion is explicitly architectural ('challenges the dominance of LSTM'), this is the load-bearing uncertainty. The phrase 'across diverse regions' is also stronger than Table S6, which shows S4D-FT improves NSE in only 68.7% and KGE in only 54.4% of watersheds and is worse on FLV and PBias, but the missing validation protocol is the decisive issue to pin down.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Frequency Tuned Diagonal State Space Sequence (S4D-FT), a state-space model, for daily rainfall-runoff simulation and benchmarks it across 531 CAMELS watersheds in the contiguous United States. Using a standard training period (1999–2008) and test period (1989–1999), the authors compare S4D-FT against an LSTM baseline, a basic S4D, and several previously published models (MC-LSTM, Transformers, Sac-SMA) using six statistical metrics. They report that S4D-FT achieves better median NSE and KGE than LSTM, present spatial skill-score maps showing regional variation, and offer an attribution analysis linking S4D-FT's relative performance to hydrologic signatures such as flow magnitude and event frequency. The paper concludes that S4D-FT outperforms LSTM and 'sets a new community-based standard' for CONUS-wide rainfall-runoff simulations with deep learning.","tokens_in":15209,"tokens_out":4951,"duration_ms":44492,"significance":"If the central claim holds, this would be the first demonstration that a state-space model outperforms the LSTM benchmark on the standard large-sample CAMELS rainfall-runoff task, a result of genuine interest to both hydrology and the broader deep-learning time-series community. The paper uses the standard CAMELS benchmark, an eight-member ensemble, public data, and a public code repository, which are strengths. The attribution analysis, while correlational, is a useful step toward understanding where a new architecture helps or hurts. However, the headline gains are small and per-watershed results are mixed, and the manual hyperparameter tuning protocol is not documented with a validation split; these issues currently prevent the strong 'new standard' conclusion from being fully supported.","major_comments":[{"comment":"The central claim that S4D-FT outperforms LSTM is not fully supported because the S4D and S4D-FT hyperparameters were 'manually tunned through a trial-and-error process' without any reported validation split or selection criterion. In particular, the S4D-FT-specific frequency-tuning scales cfi=10 and cfr=10 may have been chosen based on the 1989–1999 test period, which is the same period used for the benchmark comparison. If so, the small observed gains in Table 1 (median NSE 0.74 vs. 0.72; KGE 0.75 vs. 0.74) could reflect test-set selection bias rather than an architectural advantage of state-space models. The authors should report the validation period used for hyperparameter selection, state the selection criterion, and ideally re-evaluate S4D-FT with fixed hyperparameters selected on a held-out validation split before assessing test performance.","section":"Section 2, Tables S2–S3"},{"comment":"The conclusion that S4D-FT 'outperforms the LSTM model across diverse regions' is stronger than the reported evidence. The median gains are small (NSE +0.02, KGE +0.01), and Table S6 shows that S4D-FT improves NSE in only 68.7% and KGE in only 54.4% of watersheds, while FHV, FLV, and PBias are improved in roughly half the watersheds or fewer. No significance tests or confidence intervals are provided for the median differences or for the fraction of improved watersheds. The authors should add paired statistical tests (e.g., Wilcoxon signed-rank or bootstrap) and should either temper the 'new community-based standard' claim in Section 5 or define explicitly that it refers to median performance rather than consistently improved simulation quality.","section":"Table 1 and Table S6"},{"comment":"The attribution analysis is plausible but correlational: the conclusion that S4D-FT excels in snowmelt-driven and intermittent watersheds and underperforms in high-flow pluvio-nival watersheds is based on percentage differences and pairwise correlations for two broad groups, with only two example hydrographs. These regime classifications are not actually tested against the eight hydrologic signatures in a multivariate way, and no uncertainty is reported for the correlation coefficients. Since this attribution is a central part of the paper's interpretation (though secondary to the headline benchmark claim), the authors should either add a multivariate or conditional analysis (e.g., regression of skill scores on signatures) or soften the causal-sounding language.","section":"Section 4, Figure 2"}],"minor_comments":[{"comment":"The dimensions of matrix D are incorrect: for u(t) in C^m and y(t) in C^p, D should be in C^{p x m}, not C^{m x p}.","section":"Equation (1)"},{"comment":"The manuscript contains typos, including 'Sacramental Soil Moisture Accounting' (should be 'Sacramento') and 'manually tunned' (should be 'manually tuned').","section":"Abstract and Section 1"},{"comment":"The tuning parameters cfi and cfr are not defined in the main text; readers have to infer their meaning from the supplementary table.","section":"Section 2.1 and Table S3"},{"comment":"The reference 'Naiman et al.' appears without a year, and the Hochreiter 1997 citation is incomplete; these should be corrected.","section":"References"},{"comment":"The metric names FLV and FL V are used inconsistently, and 'Supplementary' versus 'Appendix' labels should be harmonized.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a timely question and follows a standard large-sample benchmark with public data and code, which I regard as strengths. The decisive issue is the missing validation protocol for the manually tuned S4D-FT hyperparameters; this can potentially be fixed by re-running with a proper validation split or by reporting the validation-based selection criterion. I therefore see this as a major-revision case rather than a rejection. I also note that the S4D-FT method is from the authors' own recent ICLR paper, which is appropriately cited; I do not see a circularity problem in the benchmark comparison itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. First, it is the first application of a state space model (S4D-FT) to the standard CAMELS rainfall-runoff benchmark, with public code and an honest report of a wide set of metrics. Second, the main claim—that S4D-FT \"sets a new community-based standard\" and \"outperforms LSTM\"—is plausible but not fully pinned down, because the S4D-FT hyperparameters were manually tuned with no validation split reported, and the gains over LSTM are small and mixed across metrics.\n\nWhat is genuinely new is the benchmark result itself: on 531 watersheds, median NSE goes 0.72 to 0.74 and KGE 0.74 to 0.75. That is a modest step, and the paper does a solid job of contextualizing it. The evaluation follows the established protocol, uses an eight-member ensemble, and reports FLV, PBias, and FHV alongside NSE and KGE. The regional analysis linking relative performance to hydrologic signatures (snowmelt versus pluvio-nival regimes) is useful, and the paper honestly notes in the discussion that the explanation of S4D-FT's success remains superficial and the model is a black box. That kind of self-assessment counts.\n\nThe soft spots. The tuning protocol is the load-bearing gap. The text says S4D and S4D-FT hyperparameters were \"manually tunned through a trial-and-error process,\" with no validation split or selection criterion. If cfi=10 and cfr=10 were selected by inspecting the same 1989-1999 test period used for the LSTM comparison, then part of the observed gain could be selection bias. There is no evidence they cheated, but the methods section needs to close that hole. Second, there are no significance tests across watersheds; the paper reports ensemble standard deviations but not a paired comparison, so we cannot say the median difference is distinguishable from noise at the watershed level. Third, the conclusion overstates the evidence: only 68.7% of watersheds improve on NSE and 54.4% on KGE, and FLV and PBias are worse. The body is more careful than the abstract and conclusion.\n\nBottom line: this is a competent, useful benchmark paper with a real transparency problem in the tuning. The central claim might well be true, but as reported it does not quite justify \"new community-based standard.\" It deserves a serious referee. I would like to see a revision that documents the validation protocol, adds a paired significance test, and softens the conclusion.","headline":"A useful first SSM benchmark for rainfall-runoff, but the manual tuning protocol and missing significance tests leave the 'outperforms LSTM' claim one step short of fully supported.","tokens_in":15819,"tokens_out":3217,"would_cite":true,"duration_ms":28879,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a state space model, S4D-FT, outperforms the LSTM benchmark for rainfall-runoff simulation across 531 US watersheds, challenging the long-standing LSTM dominance in hydrology.","keywords":["state space models","S4D-FT","rainfall-runoff simulation","LSTM benchmark","CAMELS","hydrologic deep learning","frequency tuning","streamflow prediction"],"falsifier":"Retrain S4D-FT and LSTM under identical conditions with a fixed validation period (for example, the last year of the training interval) used for early stopping and hyperparameter selection, then compare NSE and KGE on the held-out test period; if S4D-FT's advantage disappears or reverses, the published comparison is not evidence of an architectural benefit.","tokens_in":14735,"feed_emoji":"💧","tokens_out":5430,"duration_ms":41564,"temperature":0.7,"pith_summary":"This paper tries to establish that a state space model architecture, the Frequency Tuned Diagonal State Space Sequence model (S4D-FT), can outperform the long-standing LSTM benchmark for rainfall-runoff simulation across the contiguous United States. If correct, it breaks the pattern in which LSTM has remained the best deep-learning model for this task despite newer architectures such as Transformers, and it gives hydrologists a new benchmark and a faster, parallelizable modeling alternative. The evidence is a CONUS-wide comparison on 531 CAMELS watersheds under the community's standard training and testing setup, where S4D-FT achieves higher median NSE (0.74 vs 0.72) and KGE (0.75 vs 0.74) than LSTM. The model wins in most regions but lags in high-flow, high-volume watersheds such as the Mid-Atlantic and Pacific Northwest.","feed_headline":"State-space model edges out LSTM on 531 watersheds","feed_subtitle":"In the standard CONUS benchmark, S4D-FT lifts median NSE to 0.74 and KGE to 0.75.","key_machinery":"The load-bearing object is S4D-FT, a diagonal state space sequence model whose imaginary part of the state matrix A is rescaled at initialization by a frequency-tuning hyperparameter (set to cfi=cfr=10), correcting the spectral bias of the base S4D model. The model operates as a deep stack of continuous-time linear time-invariant systems, discretized with a trainable time step, trained in a sequence-to-one mode with a 365-day context. The comparison is carried inside the community's standard setup: 531 unimpaired CONUS watersheds, NLDAS meteorology plus 27 static attributes, training in 1999–2008 and testing in 1989–1999.","core_discovery":"Using a standardized setup of 531 CAMELS watersheds, 32 input variables, an eight-member ensemble, and a 365-day look-back window, S4D-FT reaches the best median NSE, KGE, Pearson-r, and FHV among all compared models, including LSTM, MC-LSTM, Transformers, basic S4D, and Sac-SMA. Frequency tuning is what makes the difference: basic S4D lands near Transformer level and below LSTM, while S4D-FT surpasses LSTM. The regional analysis shows the advantage concentrates in snowmelt-driven and intermittent-streamflow watersheds, while LSTM retains an edge where daily mean and peak flows are large, which the authors trace to S4D-FT's weaker improvement on high-flow volume bias (FHV).","pith_inferences":["Editorial extension: the trial-and-error hyperparameter selection should be checked with a proper validation split before treating the benchmark as settled; this is a direct test of the paper's conclusion.","Editorial extension: if the architectural advantage holds, other linear-time state space families such as Mamba or S5 would be natural candidates for the same hydrologic benchmark.","Editorial extension: combining S4D-FT with mass-conserving structure, along the lines the paper discusses for MC-LSTM, could address its weaker high-flow volume bias and improve KGE.","Editorial extension: the reported regional skill patterns suggest a practical model-selection rule—use S4D-FT in snowmelt and intermittent basins and LSTM in high-volume pluvio-nival basins—though the paper stops short of recommending it."],"forward_implications":["S4D-FT becomes the new reference deep-learning model for CONUS-wide rainfall-runoff simulation, with median NSE 0.74 and KGE 0.75.","Hydrologists can expect SSM-based models to be competitive with LSTM on long-range hydrologic sequences while training and inferring in parallel.","In watersheds with frequent, prolonged high- and low-flow events and smaller flow magnitudes—snowmelt and intermittent regimes—S4D-FT is the safer choice.","In pluvio-nival watersheds with large flow volumes, LSTM remains competitive, so high-flow simulation is the remaining gap for SSMs."],"supporting_citations":[{"why":"Supplies the LSTM implementation, hyperparameters, and training setup that this paper must beat.","marker":"[Kratzert et al., 2019a,b]"},{"why":"Provides the CAMELS dataset of catchment attributes and meteorology used for all 531 watersheds.","marker":"[Addor et al., 2017]"},{"why":"Documents the large-sample watershed data set underlying the study's forcing and streamflow data.","marker":"[Newman et al., 2015]"},{"why":"Provides the standard evaluation setup and the Sac-SMA, LSTM, and MC-LSTM benchmark statistics adopted for comparison.","marker":"[Frame et al., 2022]"},{"why":"Supplies Transformer benchmark results and the standardized simulation protocol the paper follows.","marker":"[Liu et al., 2024]"},{"why":"Introduces the frequency-tuned S4D (S4D-FT) model that is the paper's central contribution.","marker":"[Yu et al., 2025a]"},{"why":"Defines the basic S4D model and its diagonal parameterization that S4D-FT modifies.","marker":"[Gu et al., 2022]"},{"why":"Documents the Sac-SMA physically based model used as the traditional baseline.","marker":"[Anderson & McDonnell, 2005]"}],"fun_headline_variants":["S4D-FT state-space model tops LSTM on 531 watersheds","State-space model dethrones LSTM for rainfall-runoff","Frequency-tuned SSM outperforms LSTM on streamflow","S4D-FT challenges LSTM dominance in hydrology","State-space model beats LSTM on rainfall-runoff benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that S4D-FT's hand-tuned hyperparameters, selected by trial and error with no reported validation split, were not chosen using test-period performance; if that premise fails, the reported advantage over LSTM could be a tuning artifact rather than a property of the architecture.","fun_headline_variants_meta":{"raw":{"variants":["S4D-FT state-space model tops LSTM on 531 watersheds","State-space model dethrones LSTM for rainfall-runoff","Frequency-tuned SSM outperforms LSTM on streamflow","S4D-FT challenges LSTM dominance in hydrology","State-space model beats LSTM on rainfall-runoff benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000674,"raw_usage":{"total_tokens":3042,"prompt_tokens":895,"completion_tokens":2147,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":2058}},"tokens_in":511,"tokens_out":2147,"duration_ms":15842,"temperature":1.0,"reasoning_tokens":2058,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:43:49.851993+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain S4D-FT and LSTM under identical conditions with a fixed validation period (for example, the last year of the training interval) used for early stopping and hyperparameter selection, then compare NSE and KGE on the held-out test period; if S4D-FT's advantage disappears or reverses, the published comparison is not evidence of an architectural benefit.","supporting_citations":[{"cited_title":"The camels data set: catchment attributes and meteorology for large-sample studies","cited_arxiv_id":null,"evidence_quote":"Provides the CAMELS dataset of catchment attributes and meteorology used for all 531 watersheds."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the large-sample watershed data set underlying the study's forcing and streamflow data."},{"cited_title":"Deep learning rainfall--runoff predictions of extreme events","cited_arxiv_id":null,"evidence_quote":"Provides the standard evaluation setup and the Sac-SMA, LSTM, and MC-LSTM benchmark statistics adopted for comparison."},{"cited_title":"Probing the limit of hydrologic predictability with the transformer network","cited_arxiv_id":null,"evidence_quote":"Supplies Transformer benchmark results and the standardized simulation protocol the paper follows."},{"cited_title":"Sacramento soil moisture accounting model (sac-sma)","cited_arxiv_id":null,"evidence_quote":"Documents the Sac-SMA physically based model used as the traditional baseline."}],"review_version":1}