{"id":"da48025b-9839-43a4-9786-7b223e8775bb","arxiv_id":"2504.12559","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning a globally pre-trained LSTM on data from individual basins raises mean NSE by 14% and mean KGE by 15% on 159 Caravan basins, with the largest gains in the basins the global model handles worst.","lead":"This paper tests whether fine-tuning a globally trained flood forecasting model on data from individual river basins improves local forecasts. It reports roughly 14 to 15 percent gains in common hydrology skill scores, with the biggest gains in basins where the global model performed worst.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning gains in Table 1 rest on an undocumented train/validation/test split; without a demonstrated disjoint test period, the +14%/+15% NSE/KGE improvements may reflect exposure to evaluation-period data rather than local-data generalization.","rationale":"The reader's weakest assumption is exactly the concern I find most load-bearing: the paper compares fine-tuned and pre-trained models on a test period that is never described. I agree that this is the condition on which the central claim depends, and the manuscript's silence on the split makes the empirical result unverifiable as written. The reader's CONDITIONAL verdict is appropriate: the issue is addressable by inspecting or rerunning the public code, but it is not resolved in the paper. I do not see a stronger alternative concern. The regression-to-the-mean issue in Figure 2 is real but secondary; it affects the claim that underperforming basins benefit most, whereas the split problem affects the primary claim that fine-tuning helps at all. I also note the internal inconsistency between 159 and 147 basins, and the mismatched standard errors between Table 1 and Appendix C, but these do not threaten the central claim as directly. Because the code and data are public, the decisive check is straightforward, which supports CONDITIONAL rather than REJECT. My read does not change the reader's verdict.","tokens_in":8733,"tokens_out":3179,"duration_ms":35341,"concrete_test":"Inspect the public repository's configuration files to extract the exact date ranges used for training, validation, and testing for both the pre-trained and fine-tuned models. If no strict chronological split is present, rerun the fine-tuning pipeline for the 159 basins with a fixed split (e.g., train before 2016, validate 2016-2017, test 2018-2020) and recompute the mean NSE/KGE deltas in Table 1; if the gains shrink substantially or reverse, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sections 2.2 and 3.1 report that per-basin hyperparameters are selected by optimizing validation NSE, and Table 1 then compares the mean NSE/KGE of pre-trained versus fine-tuned models. The paper never states the date ranges of the training, validation, and test periods, nor whether the fine-tuning training/validation data are temporally disjoint from the evaluation period. Because the LSTM uses a 365-day lookback and streamflow is strongly autocorrelated, even modest overlap between fine-tuning training data and the evaluation window can inflate reported skill: the model would have seen the observed flows it is being asked to predict. The Table 1 footnote references a 'pre-defined training set' only for the single-basin baseline, not for the fine-tuned models, so the reader cannot infer that a clean test split was used. This is the load-bearing condition for the central empirical claim: without a demonstrated held-out test period, the +0.068 mean NSE and +0.079 mean KGE improvements are not evidence that local data adds skill beyond the global pre-trained model. The public repository could resolve this, but the manuscript itself does not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a methodology for fine-tuning a globally pre-trained LSTM streamflow model on data from individual basins, using the Caravan dataset. It compares the fine-tuned model against the pre-trained model and against single-basin-trained models on a random sample of 159 basins, reporting that fine-tuning increases mean NSE from 0.473 to 0.541 and mean KGE from 0.520 to 0.599. The paper also reports a negative correlation between pre-trained skill and fine-tuning improvement, and provides a step-by-step guide for forecasters. The stated goal is to enable national agencies to take ownership of global ML flood-forecast models by adapting them with local data.","tokens_in":8992,"tokens_out":5176,"duration_ms":47913,"significance":"If the results hold, this work is practically significant: it offers a low-compute path for local agencies to improve global ML flood forecasts, addresses data-ownership concerns, and provides open-source code and a pre-trained model. The finding that fine-tuning helps most in basins where the global model performs worst is also useful for prioritizing deployment. The paper's strengths are its clear practical framing, the open-source release, and the multi-seed (8 seeds) experimental design. However, the central empirical claim depends entirely on a correctly held-out evaluation period, which the manuscript does not describe; the reported results also contain internal inconsistencies that must be resolved before the quantitative claims can be accepted.","major_comments":[{"comment":"The manuscript never states the temporal boundaries of the training, validation, and test periods. Section 2.2 says per-basin hyperparameters are selected by optimizing validation NSE, but it does not say whether the evaluation period used for the NSE/KGE scores in Table 1 is disjoint from the fine-tuning training and validation periods. Given the 365-day input lookback and the strong autocorrelation of daily streamflow, any overlap between fine-tuning training data and the evaluation window can inflate skill because the model would have seen the observed flows it is asked to predict. The Table 1 footnote mentions a 'pre-defined training set' only for the single-basin baseline, not for the fine-tuned models. The authors must provide the exact date ranges for the training/validation/test split and a statement that the evaluation period is entirely unseen during fine-tuning (including hyperparameter selection). This is the load-bearing condition for the paper's central claim.","section":"Section 2.2 and Section 3.1, Table 1"},{"comment":"There are internal inconsistencies in the reported results. The footnote to Table 1 states that the single-basin model was trained on only 147 of the 159 basins, while the pre-trained and fine-tuned models appear to be evaluated on all 159 basins, so the three-way comparison in Table 1 is not on a common set of basins. Additionally, the standard errors for the same quantities differ between Table 1 and Table 3: for the fine-tuned model, KGE mean is 0.599 ± 0.042 in Table 1 but 0.599 ± 0.026 in Table 3; NSE mean is 0.541 ± 0.017 vs 0.541 ± 0.014; median NSE is 0.625 vs 0.627; median KGE is 0.709 vs 0.711. These discrepancies need to be reconciled, or the text should explicitly state what each interval represents (e.g., across seeds or across basins).","section":"Table 1 footnote and Appendix C, Table 3"},{"comment":"The claim that fine-tuning provides a \"significant increase\" is not supported by any inferential statistic. The paper reports standard errors but no formal significance test (e.g., paired t-test or Wilcoxon signed-rank test across basins, or a test across the 8 seed instances). Given the large number of basins, the improvement may well be real, but the current use of the word \"significant\" is not backed by a stated test. The authors should add a formal test, or explicitly state that the improvement is consistent across all 8 seeds and all 159 basins, and clarify the aggregation procedure used to compute the reported means and intervals.","section":"Section 3.1"}],"minor_comments":[{"comment":"The text states that \"for every country in the dataset, there is a net positive improvement,\" but the figure appears to show only a subset of countries. Please clarify whether this statement refers to all countries in the 159-basin sample and label the figure accordingly.","section":"Figure 2 (right panel)"},{"comment":"The fraction of variance explained by the relationship between pre-trained skill and fine-tuning improvement is 11.1%; the text \"improvement from fine-tuning is particularly large in these\" overstates the strength of the relationship. Suggest tempering the language or reporting an effect-size measure.","section":"Section 3.2"},{"comment":"The text says \"recursive neural networks (RNNs)\" but should read \"recurrent neural networks.\"","section":"Section 2.2"},{"comment":"The phrase \"short-perm predictions\" appears to be a typo for \"short-term predictions.\"","section":"Abstract and Introduction"},{"comment":"The caption for Table 3 says \"mean ± std/√n\" but does not define n (whether n=159 basins or n=8 seeds). Also, the text says 1272 data points (159×8), but then says the results are averaged over the fine-tuning values for each basin; please clarify the exact averaging order.","section":"Appendix B.2 and Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a workshop paper (ICLR 2025 Climate Change workshop) with a practical focus. The missing validation/test split description is the chief obstacle to accepting the central empirical claim; the repository may already contain the split, but the manuscript must state it explicitly. The internal inconsistencies in Table 1 vs Appendix C should be resolved before review. The authors should be encouraged to provide a public summary of the train/validation/test date ranges for all 159 basins."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — quick read on arXiv:2504.12559. What is actually new: a quantified, multi-seed evaluation of fine-tuning a global LSTM on individual Caravan basins (mean NSE +0.068, KGE +0.079), and a straightforward workflow with open code and a pretrained model. The comparison to data assimilation is fair; the 'underperformers gain most' claim is plausible but weaker than the paper makes it sound.\n\nWhere the paper earns its keep: 8 random seeds, 159 basins, hyperparameter sweeps, public repo, and the guide in Appendix A is genuinely useful for national hydromet agencies. The finding that fine-tuning helps most where the global model struggles is consistent with intuition and roughly supported, though R^2=0.111 is a weak relationship and 22% of (model, basin) pairs get worse.\n\nSoft spots, in order of severity:\n1. The train/validation/test split is not described. The paper says hyperparameters were picked on validation NSE but never states the period or whether fine-tuning training is temporally disjoint from evaluation. Given 365-day lookback and autocorrelated streamflow, this is load-bearing. If the numbers in Table 1 come from the same period used for selection, the gains are optimistically biased. The stress-test note has this right.\n2. Internal inconsistencies: 147 vs 159 basins; Fine-tuned KGE standard error is 0.042 in Table 1 but 0.026 in Appendix C; NSE 0.017 vs 0.014. Easy to fix, but a referee needs to know which is right and how basin count affects the mean.\n3. No significance tests. With 1272 data points and only 159 basins, some statement about within-basin variance vs across-basin variance is needed.\n4. The regression-to-mean caveat: basins with low pretrained scores have more room to improve. The authors don't control for this before attributing the pattern to local data.\n\nNone of these necessarily kills the paper. The central claim—local fine-tuning adds skill—is plausible and the code is public, so a motivated reviewer can check the split. But as submitted, the headline improvement is conditional on an undocumented split, so the abstract overstates it.\n\nWho this is for: hydrologists working on operational ML forecasting, and agencies deciding whether to adopt this workflow. It deserves a serious referee—send it out, but with a request to fix the split description, reconcile the numbers, and soften the causal language about underperforming basins. I'd be happy to review a revision.","headline":"Useful, honest workshop paper on fine-tuning global LSTM flood forecasts per basin; the central numbers likely hold but the paper must document the evaluation split before those numbers can be trusted.","tokens_in":9516,"tokens_out":3325,"would_cite":true,"duration_ms":31393,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a global LSTM on local basin data raises flood-forecast skill, with the biggest gains where the global model struggles.","keywords":["streamflow forecasting","flood prediction","fine-tuning","transfer learning","LSTM","global hydrology","local data","Nash-Sutcliffe efficiency"],"falsifier":"Evaluate the fine-tuned and pre-trained models on a fixed test period that is explicitly disjoint from every validation decision, including the 50-trial hyperparameter sweep for each basin; if the mean NSE and KGE gains shrink to zero or reverse on that split, the paper's central claim fails.","tokens_in":8493,"feed_emoji":"🌊","tokens_out":6224,"duration_ms":60459,"temperature":0.7,"pith_summary":"The paper tries to establish that fine-tuning a globally pre-trained LSTM on data from individual river basins improves streamflow forecast accuracy, and that the largest gains appear exactly where the global model performs worst. This matters because national forecasters often have local data but cannot train global models; fine-tuning offers them a path to take ownership of a state-of-the-art model. The reported mean Nash-Sutcliffe efficiency rises from 0.473 to 0.541 (14%) and mean Kling-Gupta efficiency from 0.520 to 0.599 (15%) across 159 sampled basins, averaged over 8 pretrained model instances. The paper also confirms that single-basin training from scratch is worse than both global pretraining and fine-tuning. If the results hold, local agencies could deploy high-skill flood forecasts using their own proprietary data.","feed_headline":"Fine-tuned local data lifts flood forecasts by 14%","feed_subtitle":"Adapting a global flood model to individual basins helps most where the global model struggles.","key_machinery":"The load-bearing mechanism is transfer learning by parameter fine-tuning of a 260K-parameter LSTM that ingests 365 days of atmospheric and catchment variables and outputs next-day streamflow. Global pretraining on 6,375 basins learns shared rainfall-runoff dynamics; fine-tuning then updates the network weights (either all weights or only the output head) on a single basin's record, with per-basin hyperparameter selection by validation NSE. A comparison against single-basin models trained from scratch isolates the value of the pretrained initialization.","core_discovery":"On its own terms, the paper's claim is that a globally pre-trained LSTM retains transferable skill that per-basin fine-tuning can sharpen: across 159 randomly sampled basins and 8 pretrained model instances, mean Nash-Sutcliffe efficiency rises from 0.473 to 0.541 and mean Kling-Gupta efficiency from 0.520 to 0.599, gains of 14% and 15%. The distribution of gains is left-skewed: median gains are smaller (NSE +0.042, KGE +0.026), and improvement correlates negatively with pretrained skill, explaining 11.1% of the variance, so the weakest basins benefit most; every country in the sample has a net positive change. Fine-tuning is not guaranteed to help: 22% of base-model/basin pairs degrade, and 14% of basins still degrade on average. The same setup also shows that training an LSTM from scratch on a single basin (mean NSE 0.358) is worse than both the global model and the fine-tuned model, which the paper interprets as confirming that large-sample pretraining sets the right initialization.","pith_inferences":["A natural next test is whether fine-tuning on local data and near-real-time data assimilation are additive; the paper compares the two approaches but does not combine them.","The negative correlation between pretrained skill and fine-tuning gain suggests fine-tuning may act partly as bias correction for regions where global forcing or regional hydrology is poorly represented; one could test this by conditioning gains on aridity, basin area, and data quality.","For operational adoption, agencies will need a protocol for re-fine-tuning as new observations arrive, since the paper's per-basin hyperparameter sweeps are expensive; a cheaper fixed-schedule fine-tune would be a testable simplification.","Because 14% of basins still regress on average, a practical deployment rule such as only fine-tuning when validation NSE improves over the pretrained model would make the method safer; the paper does not propose such a rule."],"forward_implications":["National hydromet agencies can adapt a global pretrained model to their own basins without sharing proprietary data or training from scratch, lowering the compute and data barrier to operational ML flood forecasting.","The same fine-tuning recipe likely transfers to other deep-learning hydrology architectures, since the paper notes the principle is model-agnostic.","Basins underserved by global models, those with low pretrained skill, stand to gain the most, so fine-tuning could help close the flood-forecasting gap in developing regions.","Because fine-tuning needs only historical data rather than near-real-time assimilation, it offers a cheaper route to skill gains than data assimilation, which previously produced an 8% median NSE improvement on 539 US basins."],"supporting_citations":[{"why":"Establishes that LSTM streamflow models perform best when trained on large, diverse datasets; supplies the global-training premise the paper builds on.","marker":"Kratzert et al., 2019a"},{"why":"Argues against training an LSTM on a single basin; supplies the baseline intuition and the single-basin comparison the fine-tuned model must beat.","marker":"Kratzert et al., 2024"},{"why":"Provides the global dataset of 6,375 basins used for pretraining and basin sampling.","marker":"Kratzert et al., 2023"},{"why":"Supplies the open-source training library used to pretrain and fine-tune the models.","marker":"Kratzert et al., 2022"},{"why":"Supplies the base LSTM model configuration the paper follows.","marker":"Roi-Cohen and Morin, 2024"},{"why":"Demonstrates global LSTM flood forecasting at scale; the operational context and benchmark for the pretrained model.","marker":"Nearing et al., 2024"},{"why":"Reports the 8% median NSE improvement from data assimilation that serves as the comparison point for fine-tuning's gain.","marker":"Nearing et al., 2022"},{"why":"Provides the decomposition of mean squared error and NSE that motivates reporting bias-related metrics.","marker":"Gupta et al., 2009"},{"why":"Frames NSE and KGE as complementary skill metrics and justifies the paper's metric choice.","marker":"Knoben et al., 2019"}],"fun_headline_variants":["Fine-tuning global flood models with local data boosts accuracy 14%","Local fine-tuning lifts flood forecasts, especially where global model lags","Fine-tuned flood AI: local data sharpens global forecasts by 14%","Per-basin fine-tuning improves flood forecasts, aiding weakest watersheds","Global flood model gets local upgrade: 14% better with fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the test-period scores in Table 1 are computed on data that was not used for the per-basin hyperparameter sweeps; the paper never describes the validation/test split, and its footnote mentions only a 'pre-defined training set'.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning global flood models with local data boosts accuracy 14%","Local fine-tuning lifts flood forecasts, especially where global model lags","Fine-tuned flood AI: local data sharpens global forecasts by 14%","Per-basin fine-tuning improves flood forecasts, aiding weakest watersheds","Global flood model gets local upgrade: 14% better with fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000676,"raw_usage":{"total_tokens":3091,"prompt_tokens":977,"completion_tokens":2114,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":2019}},"tokens_in":593,"tokens_out":2114,"duration_ms":16564,"temperature":1.0,"reasoning_tokens":2019,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:28:41.893848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the fine-tuned and pre-trained models on a fixed test period that is explicitly disjoint from every validation decision, including the 50-trial hyperparameter sweep for each basin; if the mean NSE and KGE gains shrink to zero or reverse on that split, the paper's central claim fails.","supporting_citations":[],"review_version":1}