{"id":"87df4493-21a4-4aa7-98fd-b4245eb68b3d","arxiv_id":"2507.11902","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across 30 imbalanced regression datasets, resampling strategies improve most models, with GN, RO, and WERCS most often best, and no universally dominant strategy.","lead":"This study tests six resampling strategies for regression problems in which rare target values matter most, using 30 datasets, six learning models, and two imbalance-aware metrics. It finds that rebalancing usually helps, that Gaussian Noise, Random Over-sampling, and WERCS are most often the best, and that no single strategy wins everywhere.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The F1-score results are not reproducible because the utility parameter p in Eq. (10) is never reported, and the strategy rankings (e.g., RO as best under F1) may depend on it.","rationale":"The reader's formal weakest_assumption is the reliability of the automatic relevance function phi, which is indeed foundational because both resampling and both metrics depend on it. I agree that this is a deep assumption, but it is a known and openly acknowledged limitation of the imbalanced-regression framework, and the paper explicitly calls for new relevance functions in future work. The more immediately load-bearing concern for the specific central claim is the unreported utility parameter p: it affects the F1-score, which is one of the two metrics used to justify the set {GN, RO, WERCS}, and it is a concrete, internally verifiable omission rather than a field-level assumption. The reader's rationale also lists this p issue as the first reason for the conditional verdict, so my concern aligns with the reader's overall judgment even though the formal weakest_assumption field points elsewhere. Because the conditional verdict already captures this issue, no adjustment to the verdict is needed. A single computational check with alternative p values would settle whether the F1-based rankings are robust, thereby confirming or refining the central claim.","tokens_in":25871,"tokens_out":6643,"duration_ms":72068,"concrete_test":"Using the released GitHub code, recompute the F1-score results (Tables 6 and 8, and the Nemenyi diagrams in Figure 6) with p set to 0, 0.5, and 1, holding all other experimental settings fixed, and report the p value that is actually used in the code/default configuration. If the relative ranking of RO, GN, and None changes across these p values, then the abstract's claim that RO and GN are among the best strategies under F1-score is conditional on an unreported hyperparameter, and the paper should report the chosen p and re-state the conclusion accordingly. If the rankings are invariant to p, the concern is resolved by simply reporting p in the experimental setup.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the best strategies include GN, RO, and WERCS is supported by two metrics: F1-score and SERA. The F1-score branch (Tables 6 and 8, Figure 6) depends on the utility function U^p_phi defined in Eqs. (4) and (10), where Eq. (10) introduces the free parameter p to weight the relevance of the predicted value phi(hat y) against the relevance of the true value phi(y). The manuscript never states the value of p used in the experiments, nor whether it was tuned or fixed. Since the abstract and Lessons Learned present RO as a top strategy largely on the strength of the F1-score results (Table 6: RO wins 62 times vs. 19.7 for None), an unreported p makes that portion of the evidence non-reproducible. If p were chosen differently (e.g., p=0 vs. p=1), the utility values and hence the F1-based ranking of resampling strategies could change, potentially removing RO from the set of best strategies. This is a concrete internal omission, distinct from the acknowledged dependence of the whole study on the chosen relevance function; the relevance-function issue is a stated limitation of the field, whereas the missing p is an unstated experimental condition that a reader cannot verify from the paper alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper surveys and empirically evaluates six resampling strategies (SmoteR, Random Over-sampling, Random Under-sampling, Introduction of Gaussian Noise, SMOGN, and WERCS) for imbalanced regression, together with six regression models and two imbalanced-aware evaluation metrics (utility-based F1-score and SERA). It proposes a taxonomy of imbalanced regression approaches, reports win counts and Friedman/Nemenyi statistical analyses over 30 datasets, and derives practical lessons: resampling is usually beneficial, the best strategy depends on dataset, model, and metric, and dataset characteristics such as size and imbalance ratio affect performance. The authors provide code and data on GitHub.","tokens_in":26123,"tokens_out":6755,"duration_ms":78165,"significance":"If the findings hold, this is a useful and reasonably comprehensive benchmark for an under-studied problem, and the practical guidance (resample, but choose the strategy conditionally) is actionable. The paper's strengths include the multi-dataset experimental design, the use of appropriate non-parametric comparisons (Friedman and Nemenyi), the explicit treatment of both local and global metrics, and the public release of code and data. The main weakness is that one load-bearing experimental condition — the utility parameter p in Eq. (10) — is never reported, which makes the F1-score branch of the evidence non-reproducible. The dependence of the whole study on the pchip-based relevance function is a known limitation of the field and is acknowledged by the authors; I do not treat that as a flaw of this manuscript.","major_comments":[{"comment":"The utility parameter p is never specified. The F1-score results in Tables 6 and 8 and Figure 6 are computed through the utility function U^p_phi in Eq. (4), which depends on the weighted relevance phi_p in Eq. (10), but the manuscript does not state the value of p used, nor whether it was fixed or tuned. Since the headline ranking of RO as the best strategy under F1 rests on this branch of the evidence, the F1 results are not reproducible as written. Please report p (and the beta value in Eq. 16), and ideally include a sensitivity analysis over p to show whether the strategy rankings change.","section":"Section 4.3, Eq. (10), Tables 6 and 8"},{"comment":"The claim that dataset size, number of rare cases, number of attributes, and imbalance ratio 'significantly influence' predictive performance is not supported by any statistical test reported in the paper; the evidence consists of visual inspection of line/scatter plots. Because RQ5 is one of the stated contributions, please either add appropriate statistical analyses (e.g., correlation or regression on per-dataset best F1 or on strategy ranks) or soften the wording to descriptive statements such as 'appear to be associated with' or 'show an association with'.","section":"Section 5, RQ5, Figures 9-13"},{"comment":"The SMOGN strategy did not complete on the california, heat, and wine-quality datasets (Tables 8 and 9), but the Friedman and Nemenyi analyses do not state how missing values were handled. Please clarify whether the statistical tests and average ranks in Tables 10-11 use complete cases only or some imputation, since this affects interpretation of the SMOGN results and of the overall statistical comparisons.","section":"Section 5, Tables 10-11 and Figures 6-7"}],"minor_comments":[{"comment":"There is a typo: 'wich uses metrics' should be 'which uses metrics'.","section":"Abstract"},{"comment":"The paragraph beginning 'By observing the score by rows...' is repeated verbatim twice; please remove the duplicate.","section":"Section 5, after Tables 6 and 7"},{"comment":"The stated range '0 <= beta <= 1' is non-standard for the F-beta formula; F1-score normally corresponds to beta = 1. Please state explicitly which beta was used to compute the reported F1-scores, or simplify the equation to the balanced F1 form.","section":"Eq. (16)"},{"comment":"The XG row sums to 30.1 and the grand total to 180.1, presumably due to rounding of 1/n tie scores; please make the displayed totals consistent with 30 datasets and 180 dataset-model combinations.","section":"Table 6"},{"comment":"The text says that datasets with fewer features exhibit superior performance, but later states that a higher number of attributes leads to better model performance. Please reconcile these statements, since they appear contradictory without further explanation of the different analyses.","section":"Section 5 and Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The missing value of p is a concrete reproducibility issue, but it is fixable by reporting the parameter and, ideally, a sensitivity analysis. The unsupported 'significantly influence' wording for dataset characteristics is also fixable. I do not see a need to reject; the study is useful and the central comparative methodology is sound. I did not inspect the GitHub repository; if the repository contains the missing p value, it should still be stated in the manuscript itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's substance is a large, clean empirical comparison of six resampling strategies across six regressors and 30 datasets, with code and data online. The main findings—resampling mostly helps, no single winner, best strategy depends on model/metric/dataset—are not new; they were in Branco et al. and Moniz et al. What is new is the breadth and the public release. That is worth having.\n\nThe good stuff: the experimental protocol is sound. They use 2x10 CV, nested CV for resampling hyperparameters, report win counts, and back them with Friedman plus Nemenyi tests. The taxonomy in Section 2 is a reasonable organizing device, even if it overlaps with existing categorizations. The discussion of each strategy's behavior (Section 3.7) is genuinely useful. The GitHub release makes the benchmark checkable.\n\nThe soft spots, in order of severity. First, the F1-score results are not reproducible as reported. Eq. (10) introduces a weighting parameter p in the utility function, and the paper never states its value or whether it was tuned/fixed. RO is presented as a top strategy largely on F1 grounds, so this is a load-bearing omission. A reader cannot verify the F1 tables without guessing p. This is an easy fix—report p and, ideally, a sensitivity check—but until then that half of the evidence is unverifiable. Second, the claims that dataset characteristics 'significantly influence' results are based on visual inspection of box plots, not significance tests. As written, 'significantly' is doing work it shouldn't. Either soften the language or run a proper test (e.g., correlation or grouped comparison). Third, the 'first comprehensive empirical analysis' framing should be qualified against Branco et al. [10], which the paper already cites and which did empirical comparisons; the claim survives only if 'comprehensive' is defined carefully (e.g., including SERA and WERCS). Minor: Section 5 repeats a paragraph verbatim twice, and the SG failures on three large datasets (marked 'not completed') are included as zero values in the tables, which biases the win counts against SG; those runs should be reported as missing, not zero.\n\nOverall: the central argument holds. The missing p is the only piece that undermines reproducibility, and it is fixable. This paper deserves a serious referee, not a desk reject. I'd send it out with a request to address p and the significance-language issue. Good candidate for a reading group too, but mainly as a reference benchmark.","headline":"A useful, broad empirical benchmark on resampling for imbalanced regression, held back by one missing experimental parameter and some overclaimed dataset findings—revisable, not fatal.","tokens_in":26680,"tokens_out":2261,"would_cite":true,"duration_ms":26003,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Resampling nearly always helps imbalanced regression, but the best strategy depends on the data, the model, and the metric.","keywords":["imbalanced regression","resampling strategies","relevance function","SERA metric","utility-based F1-score","random over-sampling","Gaussian noise","WERCS"],"falsifier":"Re-run the 30-dataset comparison with a domain-defined relevance function, for example the NO2 concentration thresholds the paper cites, in place of the pchip/quartile-boxplot one; if the ranking of strategies changes substantially, the reported winners are artifacts of that automatic relevance definition rather than properties of imbalanced regression.","tokens_in":1416,"feed_emoji":"📊","tokens_out":1592,"duration_ms":92834,"temperature":0.7,"pith_summary":"Imbalanced regression treats problems where extreme or rare target values matter most, and machine-learning models tend to ignore them because they are underrepresented. The paper compares six resampling strategies on 30 datasets with six regression models, using two metrics designed to reward accuracy on rare values. It tries to establish that resampling is beneficial for the vast majority of regression models, with Gaussian noise, random over-sampling, and WERCS as the strongest strategies, and that the best choice depends on the dataset, the model, and the metric. If this holds, practitioners should not skip resampling but also should not expect a single universal winner.","feed_headline":"Resampling beats no resampling for imbalanced regression","feed_subtitle":"Gaussian noise, random over-sampling, and WERCS lead, but the best choice depends on data, model, and metric.","key_machinery":"The load-bearing mechanism is the automatic relevance function $\\varphi(y)$, built by pchip interpolation over control points derived from quartile-based boxplot statistics of the target variable. This function assigns each target value a relevance score between 0 and 1, and a threshold $t_R$ splits the data into rare and normal examples. The same function drives both halves of the study: each resampling strategy uses it to decide which cases to duplicate, remove, or synthesize, and both evaluation metrics (the utility-based F1-score and the SERA metric) compute scores from it. The whole comparison is therefore a comparison of strategies under one particular definition of which target values count as rare.","core_discovery":"The paper's central claim is that resampling strategies improve imbalanced regression performance for the vast majority of models, and that the improvement is statistically significant when tested across multiple datasets. The best overall strategies are Gaussian noise, random over-sampling, and WERCS, while SmoteR, SMOGN, and random under-sampling are weaker in these experiments. No strategy wins everywhere: the winning combination shifts with the dataset and the learning algorithm, and the two evaluation metrics (utility-based F1-score and SERA) sometimes disagree. The paper also finds that the hardest problems are small datasets with few rare cases and high imbalance ratios, and that models perform better when the number of features is small.","pith_inferences":["Beyond the paper: because the same automatic relevance function both selects rare cases for resampling and scores the results, the reported ranking is conditional on that choice of $\\varphi$; adopting a domain-defined relevance function, such as regulatory thresholds, could reorder the winners.","Beyond the paper: the finding that dataset size and imbalance ratio drive difficulty suggests that a meta-learning approach recommending a resampling strategy from dataset characteristics could be feasible.","Beyond the paper: a direct test of the paper's conclusions would be to repeat the 30-dataset comparison with a different relevance threshold, for instance $t_R = 0.5$ or $0.9$, and check whether the Gaussian-noise, random-over-sampling, and WERCS advantage persists.","Beyond the paper: SERA's global, threshold-free evaluation combined with WERCS's threshold-free resampling suggests these two may be a particularly robust pairing when domain knowledge about the relevance threshold is absent."],"forward_implications":["Practitioners should not default to no resampling: for most regression models, applying Gaussian noise, random over-sampling, or WERCS improves imbalance-aware performance over the raw data.","There is no universal best strategy; resampling should be treated as a tunable component of the modeling pipeline, selected per dataset and per evaluation metric.","Strategies that inflate the training set, like random over-sampling (about 1421 percent growth), can be replaced by lighter ones like Gaussian noise and WERCS (about 1 to 3 percent growth) with comparable gains, saving training time.","Datasets that are small, have few rare cases, or have a high imbalance ratio are where resampling choices matter most, and default pipelines are most likely to fail there.","The two imbalance-aware evaluation lenses, F1-score and SERA, can disagree about which strategy wins, so reported performance should be metric-specific."],"supporting_citations":[{"why":"Supplies the relevance function used to label rare and normal target values, which drives every resampling strategy and both evaluation metrics.","marker":"[6]"},{"why":"Proposes the SERA metric and the extreme-value evaluation framework used to score models on rare target values.","marker":"[8]"},{"why":"Introduces WERCS, random over-sampling, and Gaussian noise strategies, and motivates the utility-based evaluation approach.","marker":"[10]"},{"why":"Introduces SmoteR and random under-sampling, two of the six compared strategies.","marker":"[9]"},{"why":"Introduces SMOGN, the hybrid resampling strategy included in the empirical comparison.","marker":"[5]"},{"why":"Defines precision, recall, and F1-score for regression, the local utility-based metric used here.","marker":"[16]"},{"why":"Provides the Friedman and Nemenyi statistical tests used to establish that resampling significantly improves performance.","marker":"[54]"}],"fun_headline_variants":["Resampling wins for imbalanced regression, but best strategy varies","Gaussian noise and oversampling lead imbalanced regression fixes","No single best resampling for imbalanced regression, survey finds","Resampling improves imbalanced regression for most models","Best imbalanced regression resampling depends on data and model"],"cache_read_input_tokens":28800,"weakest_assumption_plain":"The whole comparison rests on the automatic relevance function correctly marking which target values are rare and important, because that same function decides both what each resampling strategy changes and what both metrics reward.","fun_headline_variants_meta":{"raw":{"variants":["Resampling wins for imbalanced regression, but best strategy varies","Gaussian noise and oversampling lead imbalanced regression fixes","No single best resampling for imbalanced regression, survey finds","Resampling improves imbalanced regression for most models","Best imbalanced regression resampling depends on data and model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1470,"prompt_tokens":845,"completion_tokens":625,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":544}},"tokens_in":461,"tokens_out":625,"duration_ms":6108,"temperature":1.0,"reasoning_tokens":544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:58:34.244353+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 30-dataset comparison with a domain-defined relevance function, for example the NO2 concentration thresholds the paper cites, in place of the pchip/quartile-boxplot one; if the ranking of strategies changes substantially, the reported winners are artifacts of that automatic relevance definition rather than properties of imbalanced regression.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the relevance function used to label rare and normal target values, which drives every resampling strategy and both evaluation metrics."},{"cited_title":"Machine Learning 109(9), 1803–1835 (2020)","cited_arxiv_id":null,"evidence_quote":"Proposes the SERA metric and the extreme-value evaluation framework used to score models on rare target values."},{"cited_title":"Neurocomputing 343, 76–99 (2019)","cited_arxiv_id":null,"evidence_quote":"Introduces WERCS, random over-sampling, and Gaussian noise strategies, and motivates the utility-based evaluation approach."},{"cited_title":"In: Portuguese Conference on Artificial Intelligence, pp","cited_arxiv_id":null,"evidence_quote":"Introduces SmoteR and random under-sampling, two of the six compared strategies."},{"cited_title":"In: First International Workshop on Learning with Imbalanced Domains: Theory and Applications, vol","cited_arxiv_id":null,"evidence_quote":"Introduces SMOGN, the hybrid resampling strategy included in the empirical comparison."},{"cited_title":"In: International Conference on Discovery Science, pp","cited_arxiv_id":null,"evidence_quote":"Defines precision, recall, and F1-score for regression, the local utility-based metric used here."},{"cited_title":"Journal of Machine Learning Research 7(Jan), 1–30 (2006)","cited_arxiv_id":null,"evidence_quote":"Provides the Friedman and Nemenyi statistical tests used to establish that resampling significantly improves performance."}],"review_version":1}