{"id":"111d020f-7411-4181-b17c-a75ee51ea663","arxiv_id":"2506.13786","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"EBMBag+, a bagging ensemble with block bootstrap and error-weighted averaging, forecasts 2021 U.S. state diabetes prevalence with MAE 0.41 and R2 0.91 on 51 states.","lead":"This paper builds a combined U.S. state-level dataset and a modified bagging regression model, called EBMBag+, to forecast diabetes prevalence for 2021. The model reports the lowest error among seven baselines, but the test set has only 51 states and the margin is small.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set lag selection likely inflates ERMBag+ gains; validation-based lag choice is required before the headline holds.","rationale":"The reader's verdict already requires validation-based lag selection, and my independent reading identifies the same protocol as the most load-bearing issue. The paper's strongest claim is specifically about being best at Lag=2, but the lag is chosen by inspecting test-set performance, so the reported metrics are in-sample maxima rather than honest forecasts. The performance gap over the closest baseline is small enough that choosing among nine lag configurations on 51 test points could plausibly account for it. The interpolation of 2021 features is also concerning, but it affects all models similarly and is secondary to the selection bias. The algorithm itself is not internally inconsistent; the weakness is in the evaluation protocol. Since the reader already conditioned acceptance on validation-based lag selection, I do not change the verdict.","tokens_in":10430,"tokens_out":8655,"duration_ms":90779,"concrete_test":"Run a leave-one-year-out validation over 2012-2020 (with each year's lag restricted to available history) to select the Lag for each model, then evaluate the selected models on 2021; also report fixed-Lag=2 results for all models. If ERMBag+ is not best under both protocols, the headline 'best at Lag=2' is a test-set selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2 and Table 2 report ERMBag+ as best at Lag=2, but the Lag for every model is chosen as its best value on the same 2021 test set (51 state-year observations). The headline metrics (MAE=0.41, RMSE=0.53, R2=0.91) are therefore the maximum over nine lag configurations of in-sample test performance, not an unbiased out-of-sample result. The comparison also mixes models at different lags (BDTree/LSBoost at Lag=1, NN at Lag=4, SVMReg at Lag=6, LSTM at Lag=9), so it is not a single forecasting task. With only 51 test points, selecting the best of 9 correlated configurations can shift rankings by more than the reported margins (ERMBag+ vs ERMBag: ΔMAE=0.03, ΔRMSE=0.07, ΔR2=0.03). No confidence intervals or significance tests are reported. If Lag is selected by temporal validation instead of test performance, the claimed superiority may disappear; this is the central load-bearing threat to the claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper integrates four U.S. government datasets (chronic disease indicators, population, housing, and economy) into a 90-feature state-year panel from 2011 to 2021, and proposes ERMBag+, a bagging ensemble of decision trees with stratified block bootstrap, early stopping, and inverse-RMSE weighted aggregation. It evaluates ERMBag+ against six baselines (SVMReg, BDTree, LSBoost, NN, LSTM, ERMBag) for one-step-ahead prediction of state-level diabetes prevalence in 2021, and reports that ERMBag+ achieves the best metrics (MAE = 0.41, RMSE = 0.53, MAPE = 4.01, R² = 0.91).","tokens_in":10642,"tokens_out":2328,"duration_ms":27202,"significance":"If the empirical comparison were valid, the paper would make a useful applied contribution by demonstrating a reproducible data-engineering pipeline for state-level health forecasting and a modestly improved bagging variant. The data integration across four public sources and the explicit time-series framing are constructive. However, the central claim of superiority is currently supported only by a single test year with 51 observations, a test-set-based selection of lag hyperparameters, and no uncertainty quantification; the methodological novelty of ERMBag+ over standard bagging is incremental, and the evaluation protocol needs substantial rework before the headline result can be accepted.","major_comments":[{"comment":"The reported performance of every model is the best result over nine lag configurations, with the lag chosen on the same 51-point 2021 test set used for the final comparison. This is test-set selection: the headline ERMBag+ figures (MAE = 0.41, RMSE = 0.53, R² = 0.91) are the maximum over Lag = 1,...,9 of in-sample test performance, not an unbiased out-of-sample estimate. Because the margins over ERMBag are small (ΔMAE = 0.03, ΔRMSE = 0.07), and 51 points cannot reliably discriminate among nine correlated configurations, this protocol could plausibly manufacture the reported ranking. The authors should select Lag and any other hyperparameters by temporal (rolling-origin) validation on 2011-2020, then evaluate once on 2021, or report results for all lags without per-model selection.","section":"Section 5.2 and Table 2"},{"comment":"The test set contains only 51 observations, and no confidence intervals, standard deviations across repeated runs, or significance tests are provided. NN, LSTM, and the bagging ensembles are stochastic, so single-run comparisons are not sufficient to support the claim that ERMBag+ outperforms ERMBag and NN. Additionally, several hyperparameters of ERMBag+ are not specified: the number of base learners M, the block size B, the early-stopping threshold, and the validation split used to compute RMSE weights in Step 3 and Step 4 of Algorithm 1. Without these details, the contribution of each proposed enhancement cannot be isolated, and the experiments are not reproducible.","section":"Section 5.2 and Algorithm 1"},{"comment":"The 2021 values for population and housing features are not observed but are obtained by linear interpolation (population from 2011-2020 trends and the 2021 Census; housing from 2010 and 2020 decennial data). Since 2021 is exactly the test year, every model prediction rests on interpolated inputs; if the interpolation error is correlated with diabetes prevalence or with state-level trends, the evaluation could be systematically biased. The paper should either validate the interpolation against held-out observed years, or repeat the experiment using an earlier year with fully observed inputs as the test period.","section":"Section 3.1"}],"minor_comments":[{"comment":"The abstract reports an R² of 0.9 while Section 5.2 reports R² = 0.91; the inconsistent number should be reconciled.","section":"Abstract and Section 5.2"},{"comment":"The model name is written as 'EBMBag+' in most of the paper but 'ERMBag+' in the abstract and in Table 2; the naming should be made consistent.","section":"Abstract and Section 4.4"},{"comment":"The 'stratified block bootstrap' is not fully defined: it is not specified which variables are used for stratification, how the non-overlapping time blocks are formed when lagged features are present, or how the block size B is chosen. A precise algorithmic description is needed.","section":"Section 4.4, Algorithm 1"},{"comment":"Figure 3 is described as showing average prediction performance, but no error bars or dispersion measures are provided; for stochastic models this makes the visual comparisons uninformative.","section":"Section 5.2, Figure 3"},{"comment":"The statement that ERMBag+ outperforms baselines by '3.4% to 61.1%' is not backed by an explicit calculation in the paper and should be derived from the metric formulas or replaced by directly reported relative differences.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The main barrier is evaluation validity rather than novelty or scope. The paper fits an applied ML / health-informatics venue, but the test-set lag selection and lack of uncertainty quantification need to be fixed before the headline claim can be considered sound. I would not recommend rejection because the issues are addressable within the manuscript's scope: a validation-based model selection protocol and repeated-run experiments with interval estimates would substantially strengthen the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the integrated state-level diabetes dataset and the specific bagging variant called EBMBag+. The data integration is real work: 90 features across age, race, gender, housing, economy, and 71 chronic-disease indicators for 51 states, 2011–2021. The method is a reasonable combination of known pieces: stratified block bootstrap, early stopping, and inverse-RMSE weighted averaging on decision trees. I buy that as a modest contribution for applied public-health forecasting.\n\nWhat the paper does not do is establish that the improvement over plain ERMBag is real. The headline numbers (MAE 0.41 vs 0.44, RMSE 0.53 vs 0.60, R2 0.91 vs 0.88) come from Table 2, where each model's lag is chosen as its best on the 2021 test set. That is test-set selection, and with 51 test points the difference between 0.41 and 0.44 is well within what noise can produce. The comparison also mixes models at different lags, so it is not a single forecasting task. No confidence intervals, no repeated runs, no significance test. Early stopping weights depend on a validation split that is never described. Hyperparameters for the base models are largely unreported. The 2021 inputs are partly linearly interpolated; that is disclosed, but it means every 2021 prediction rests on constructed values.\n\nI don't think the central idea is wrong. The bagging variant is plausible, and the dataset is a useful resource if they release it. But the evidence as presented is not enough to support \"ERMBag+ outperforms all baselines\" as a general claim. The stress-test note lands: validation-based lag selection is the load-bearing fix.\n\nWho benefits: applied ML people in public health who want a ready-made state-level feature set and a baseline to beat. It is not a methods paper. It deserves a serious referee rather than a desk reject, but the referee should require nested lag selection, error bars or repeated runs, a persistence/ARIMA baseline, and code release. My own verdict would be conditional on those changes.","headline":"Useful integrated dataset and a plausible bagging tweak, but the headline accuracy gains rest on test-set lag selection and need error bars before they hold.","tokens_in":11147,"tokens_out":2617,"would_cite":false,"duration_ms":27754,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An enhanced bagging ensemble with stratified block bootstrap and inverse-RMSE weighting beats six baselines for U.S. state-level diabetes prevalence in 2021","keywords":["diabetes prevalence","bagging ensemble regression","time series forecasting","stratified block bootstrap","inverse-RMSE weighting","health data integration","state-level prediction","machine learning regression"],"falsifier":"Replace the interpolated 2021 population and housing values with the actual 2021 Census counts and rerun the full comparison with Lag chosen by cross-validation on 2011 to 2020 data; if the reported MAE = 0.41 and $R^2 = 0.91$ persist, the central claim is supported, while if the gap over ERMBag narrows sharply, the reported advantage depends on interpolation and test-set tuning.","tokens_in":10244,"feed_emoji":"🩺","tokens_out":7687,"duration_ms":72038,"temperature":0.7,"pith_summary":"The paper tries to establish that a modified bagging ensemble regression model, called ERMBag+ in the main text, predicts next-year diabetes prevalence for U.S. states more accurately than six standard machine learning baselines. It builds a 90-feature dataset by integrating four U.S. public data sources spanning 2011 to 2021, then converts the panel into supervised time-series examples using lagged predictors. The model's two main changes to ordinary bagging are stratified block bootstrap sampling, which preserves temporal ordering and subgroup distributions, and weighted aggregation in which each tree's vote counts inversely to its validation RMSE. On the held-out year 2021 across 51 states, the paper reports ERMBag+ achieves MAE = 0.41, RMSE = 0.53, MAPE = 4.01, and $R^2 = 0.91$, the best among all models compared. If correct, this gives public-health planners a practical recipe for state-level diabetes forecasting from routinely collected public data.","feed_headline":"Bagging variant predicts diabetes prevalence with R-squared 0.91","feed_subtitle":"Stratified block bootstrap plus inverse-RMSE weighting beats six baselines on 51 U.S. states for 2021.","key_machinery":"The load-bearing mechanism is the ERMBag+ training procedure: stratified block bootstrap sampling for time-series data, early stopping per decision tree, and weighted aggregation with $w_i = 1/\\mathrm{RMSE}(f_i)$ normalized over the ensemble. Lag-window construction turns the panel of 89 annual features into supervised examples with $89 \\times (l+1) + l$ variables, so that at the selected Lag=2 each example covers the current and two preceding years. The block bootstrap is supposed to preserve temporal autocorrelation while retaining demographic and economic subgroup proportions, and the inverse-RMSE weights let better-validated trees dominate the final forecast.","core_discovery":"On the paper's own terms, the central discovery is that replacing plain bootstrap resampling with stratified block bootstrap and replacing equal averaging with inverse-RMSE weighted averaging turns an ordinary bagged decision-tree regressor into the most accurate model for the 2021 diabetes-prevalence forecasts. The reported winner is ERMBag+ at Lag=2, where each training example concatenates 89 predictors from the current and two previous years plus the two previous target values, yielding $89 \\times (l+1) + l = 269$ variables per example. Its reported error metrics are MAE = 0.41, RMSE = 0.53, MAPE = 4.01, and $R^2 = 0.91$, compared with the runner-up ERMBag at MAE = 0.44, RMSE = 0.60, MAPE = 4.38, and $R^2 = 0.88$. The paper attributes the gain to better preservation of temporal dependencies and to giving more weight to trees that validate well.","pith_inferences":["The paper reports each model's best Lag chosen from the test-year results; a stricter protocol would tune Lag inside the training period only, and rerunning the comparison that way could shrink but would not necessarily erase the reported gap.","The 71 chronic-disease indicators likely carry much of the predictive signal, since they include direct health-behavior and comorbidity measures; an ablation by feature category would test whether demographic, housing, and economic features add accuracy beyond those indicators.","An $R^2$ of 0.91 with MAPE of 4.01 implies average errors near 0.4 percentage points, which can still be material in states with low diabetes prevalence; reporting per-state error bounds would sharpen the policy use of the forecasts.","The same stratified block bootstrap plus inverse-RMSE weighting recipe could be transferred to other annual state-level health outcomes, such as obesity or kidney disease, using the same public data pipeline."],"forward_implications":["ERMBag+ at Lag=2 yields a one-year-ahead state-level forecast with reported MAE of 0.41 percentage points and $R^2 = 0.91$, the best accuracy among all models the paper compares.","Because ERMBag+ differs from the ERMBag baseline by stratified block bootstrap, early stopping, and inverse-RMSE weighting, the reported gap isolates the benefit of those modifications.","The lag analysis shows that the best window length is model-dependent, with Lag=1 for BDTree and LSBoost, Lag=2 for the bagging variants, Lag=6 for SVMReg, and Lag=9 for LSTM, so practitioners should treat lag as a hyperparameter.","The integrated 90-feature dataset is assembled from public annual sources, so the same preprocessing pipeline can be rerun as CDC, Census, BLS, and BEA data are updated."],"supporting_citations":[{"why":"Supplies the 71 chronic-disease indicator features from the CDC's U.S. Chronic Disease Indicators dataset.","marker":"[6]"},{"why":"Provides the NCI census-tract population data used to construct demographic features for 2011 to 2020.","marker":"[21]"},{"why":"Provides the 2021 Census population estimates used, with linear interpolation, for the 2021 input features.","marker":"[28]"},{"why":"Supplies the 2010 and 2020 Census housing counts used to interpolate housing features for 2011 to 2021.","marker":"[26]"},{"why":"Provides the Bureau of Labor Statistics employment data included among the economic features.","marker":"[4]"},{"why":"Defines the bagging ensemble regression method that ERMBag+ extends.","marker":"[31]"},{"why":"Defines the LSBoost gradient boosting baseline used for comparison.","marker":"[10]"},{"why":"Defines the LSTM baseline used for comparison in the time-series forecasting task.","marker":"[18]"}],"fun_headline_variants":["Bagging with weighted resampling hits R2 0.91 for diabetes","Stratified block bootstrap bagging outdoes six baselines","Inverse-RMSE bagging improves diabetes prevalence forecasts","Enhanced bagging regressor wins diabetes prediction with R2 0.91","Weighted bagging ensemble tops time-series diabetes forecasting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 2021 population and housing features are accurate enough despite being linearly interpolated rather than observed, and that selecting each model's best Lag from the 2021 test results does not inflate the reported advantage.","fun_headline_variants_meta":{"raw":{"variants":["Bagging with weighted resampling hits R2 0.91 for diabetes","Stratified block bootstrap bagging outdoes six baselines","Inverse-RMSE bagging improves diabetes prevalence forecasts","Enhanced bagging regressor wins diabetes prediction with R2 0.91","Weighted bagging ensemble tops time-series diabetes forecasting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1384,"prompt_tokens":933,"completion_tokens":451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":362}},"tokens_in":549,"tokens_out":451,"duration_ms":4486,"temperature":1.0,"reasoning_tokens":362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:49:22.777638+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the interpolated 2021 population and housing values with the actual 2021 Census counts and rerun the full comparison with Lag chosen by cross-validation on 2011 to 2020 data; if the reported MAE = 0.41 and $R^2 = 0.91$ persist, the central claim is supported, while if the gap over ERMBag narrows sharply, the reported advantage depends on interpolation and test-set tuning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 71 chronic-disease indicator features from the CDC's U.S. Chronic Disease Indicators dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NCI census-tract population data used to construct demographic features for 2011 to 2020."},{"cited_title":"Census Authors: State population by characteristics: 2020- 2023.https://www.census.gov/data/tables/time-series/demo/popest/ 2020s-state-detail.html(2024), [Accessed 01-June-2024]","cited_arxiv_id":null,"evidence_quote":"Provides the 2021 Census population estimates used, with linear interpolation, for the 2021 input features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 2010 and 2020 Census housing counts used to interpolate housing features for 2011 to 2021."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Bureau of Labor Statistics employment data included among the economic features."},{"cited_title":"IEEE Transactions on Artificial Intelligence5, 1728 (2023)","cited_arxiv_id":null,"evidence_quote":"Defines the bagging ensemble regression method that ERMBag+ extends."},{"cited_title":"An- nals of Statistics pp","cited_arxiv_id":null,"evidence_quote":"Defines the LSBoost gradient boosting baseline used for comparison."},{"cited_title":"Expert Systems with Applications177, 114905 (2021)","cited_arxiv_id":null,"evidence_quote":"Defines the LSTM baseline used for comparison in the time-series forecasting task."}],"review_version":1}