{"id":"7e03ee90-c548-45d8-828a-99a0fd5a23d8","arxiv_id":"2607.20173","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An instance-hardness relevance function that labels examples by prediction difficulty improves oversampling for imbalanced regression, with modest empirical gains.","lead":"This paper proposes a new way to decide which examples are 'rare' in imbalanced regression: instead of looking only at how often target values occur, it measures how hard each example is for machine-learning models to predict. The method then uses those hard examples to drive oversampling, and the authors report modest gains in prediction error compared with older relevance-based sampling.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IH may be computed with the same models and folds used for evaluation; without a nested protocol, the reported improvements could be a selection artifact.","rationale":"The reader's weakest assumption partially overlaps with this concern, but focuses on whether 'hard' instances are stable versus noise/outliers. My concern is more specific and more damaging: the manuscript does not document whether IH is computed in a nested, fold-local manner or on the full dataset, and it uses the same model families for both hardness estimation and evaluation. This creates a leakage/selection-artifact risk that directly threatens every quantitative result, including the headline '70%' win rate. The correct response is not to reject outright, because the code may reveal a valid protocol, but to mark the paper unverified until the protocol is checked. The concrete test—code inspection plus a disjoint-model rerun—is decisive and cheap. If the check shows out-of-fold IH with a disjoint model family, the empirical claim would merit reconsideration; if not, the claim should be abandoned.","tokens_in":10005,"tokens_out":5312,"duration_ms":61105,"concrete_test":"Download the public repository and trace how IH values are assigned. Verify (a) whether IH is computed before the 2×5 CV split or within each training fold, and (b) whether the model(s) producing IH for a given validation instance were trained without that instance. Then rerun the main comparison (Tables III–IV) with IH computed from a disjoint, simple regressor family (e.g., k-NN) and a threshold chosen a priori. If the win rate drops below ~50% or Wilcoxon significances disappear, the reported advantage is an artifact; if it persists, the central claim is supported.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Section III Eq. (1) defines IH from prediction errors of a regressor set L, and Section IV states IH was 'instantiated using error-based measures computed from the regression models defined in the Regression Models subsection'—the same RF, BG, XGB, SVR, and MLP families later used as base learners in the 2×5 CV comparison. The manuscript never states whether IH scores are computed within each training fold (e.g., via internal cross-validation) or on the full dataset before resampling. If IH is computed on all instances, test target values leak into the choice of which instances are oversampled, invalidating the empirical comparisons in Tables III–V. Even if IH is computed fold-locally, using the same model families that are later evaluated induces a selection bias: the 'rare' set is defined by where those models err, so evaluating those same model families on oversampled data is partly an in-sample optimization. Table V's improvement on the 'InHaR-rare' subset is particularly expected under this protocol. The paper's claim that MAE/MSE are 'relevance-independent' addresses metric bias, not this selection/leakage channel. Since the released code is the only way to determine which situation holds, the central claim is currently unverifiable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InHaR, an Instance Hardness-based relevance function for imbalanced regression, replacing the traditional target-frequency relevance function ϕ by a difficulty-based score computed from prediction errors of a set of regressors (Eq. 1, Section III). Rare instances are then selected by thresholding IH values, and these instances guide Random Oversampling (RO) and Gaussian Noise (GN) resampling. Experiments use 2×5 cross-validation on 29 datasets and a synthetic bimodal dataset, comparing MAE and MSE against no resampling, the traditional relevance function, and the Distance-Based Relevance Function (DRF). The authors claim that InHaR correctly identifies rare regions under bimodal distributions and improves predictive performance in approximately 70% of the tests compared to traditional relevance-based approaches.","tokens_in":10409,"tokens_out":3696,"duration_ms":40481,"significance":"If the empirical claims hold, the paper offers a novel and potentially useful perspective on relevance in imbalanced regression by shifting from distributional rarity to learning difficulty. The use of relevance-independent metrics (MAE and MSE), the Wilcoxon signed-rank tests for paired comparisons, and the public release of code and data are positive features. However, the currently unverifiable protocol for computing IH, the circular validation in the bimodal experiment, and the inconsistent reporting of the headline '70%' result prevent the paper from being accepted in its present form. The central idea is promising, but the evidence needs to be substantially clarified and strengthened.","major_comments":[{"comment":"The manuscript never states whether IH values in Eq. (1) are computed within each training fold (e.g., by internal cross-validation) or on the full dataset before resampling. Section IV says IH was 'instantiated using error-based measures computed from the regression models defined in the Regression Models subsection' — the same model families (RF, BG, XGB, SVR, MLP) used later as base learners in the 2×5 CV. If IH is computed on all instances, test target values leak into the choice of which instances are oversampled, invalidating the comparisons in Tables III–V. Even if computed fold-locally, the overlap between IH regressors and evaluated regressors should be described and justified. This is a load-bearing methodological ambiguity and makes the central empirical claim currently unverifiable from the text alone.","section":"Section IV - Instance Hardness / Evaluation Methods"},{"comment":"The validation in the bimodal experiment is circular. Bins are labeled 'rare' when their median IH is in the top 30% (the 70th percentile), and the paper then claims that InHaR 'correctly identifies' rare regions (Abstract and Section V-B). No independent ground truth for rarity is provided; the method is shown to identify regions that it itself defines as rare. Furthermore, the 70th percentile was 'selected based on preliminary experiments' with no sensitivity analysis, as the paper itself acknowledges. This section should either provide an external benchmark (e.g., known under-sampled areas in the synthetic data) or be reframed as a descriptive demonstration rather than a validation.","section":"Section V-B - Bimodal Distributions"},{"comment":"The abstract states that 'in approximately 70% of the tests performed, the proposed approach achieved superior performance compared to the traditional approach.' This is not supported by Table III, where InHaR-RO and InHaR-GN together win 68/145 (47%) MAE pairs and 58/145 (40%) MSE pairs. Even if 'traditional approach' is intended to mean only the classical RO/GN (Table IV), the MAE win rate is 51/58 (88%) and the MSE rate is 40/58 (69%), not an overall 70%. The claim needs a precise definition of 'tests' and a consistent number.","section":"Abstract / Introduction and Table III"},{"comment":"The method relies on a fixed threshold τ=0.7 for IH (Section IV) and a 70th-percentile cutoff in the bimodal analysis (Section V-B), both chosen based on preliminary experiments. The conclusion explicitly notes that 'a systematic analysis of threshold sensitivity was not conducted.' Since the performance of InHaR is directly controlled by these hyperparameters, the reported results may not generalize. The authors should provide a sensitivity analysis or otherwise justify that the performance advantage is not an artifact of the chosen thresholds.","section":"Section IV / Conclusion - Threshold sensitivity"}],"minor_comments":[{"comment":"There is a typo: 'future work. .' should be 'future work.'","section":"Section V-B"},{"comment":"The row/column formatting makes the comparison hard to read. Consider providing a clearer table with pairwise wins/losses or a visual matrix to avoid misreading.","section":"Table III"},{"comment":"Figure 2 would benefit from a caption that explicitly states over how many datasets the correlation distribution is plotted and how the IH values were computed for this analysis (e.g., full-data vs. cross-validated predictions).","section":"Section V-A"},{"comment":"Reference [18] is a technical report; if a peer-reviewed version of the instance-hardness-for-regression measure exists, it should be cited instead.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a potentially valuable contribution, but the lack of clarity about the IH computation protocol is a serious issue that needs to be resolved before the empirical claims can be trusted. The authors have released code, so the protocol can likely be clarified or fixed with a nested evaluation. The circularity in the bimodal analysis and the inconsistency in the '70%' claim are also fixable. I recommend major revision rather than rejection, provided the authors can address these points adequately."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper has a new idea—using instance hardness as the relevance criterion for imbalanced regression—and it mostly runs a sensible evaluation, but the central empirical claim is currently unverifiable because the manuscript never says how IH is computed relative to the cross-validation folds. If IH is computed on the full dataset, test target values leak into the choice of which instances get oversampled and the comparisons in Tables III-V don't mean what they claim. Even if it's fold-local, using the same model families that are later evaluated is a milder form of selection bias. The code link may resolve this, but the paper as written does not.\n\nWhat's genuinely good: replacing statistical rarity with learning difficulty is more than a parameter tweak. The correlation analysis between IH and the Ribeiro relevance function is interesting; it shows they are related but not identical, which motivates the approach. Using MAE/MSE rather than SERA or similar is the right call for a paper that is changing the relevance definition. The Wilcoxon testing is appropriate, and the 29-dataset comparison is a reasonable amount of work.\n\nThe soft spots, in decreasing order: the fold-protocol ambiguity above is the big one. Then the abstract's 'approximately 70%' is contradicted by the paper's own Table III, which gives 68/145 (47%) for MAE and 58/145 (40%) for MSE. That matters because the abstract is what people will remember. The bimodal validation is circular: the paper defines rare bins as those with high median IH, then reports that those bins have high IH. Without an external ground-truth definition of rarity, 'correctly identifies' is meaningless. Finally, the threshold τ=0.7 and the 70th percentile are chosen after preliminary experiments, with no sensitivity analysis; the paper honestly acknowledges this in the conclusion, but it remains a major limitation.\n\nThis is a paper for the imbalanced-regression subcommunity. The idea is worth a serious referee, and the authors have a chance to fix things. But in its current form I would not cite it for the main empirical claim. My recommendation: send it to peer review with a request that the reviewers verify the IH computation protocol and ask for a sensitivity analysis and a corrected abstract.","headline":"InHaR is a genuinely new twist on relevance for imbalanced regression, but the empirical comparisons may be compromised by an unstated fold-handling protocol, and the abstract overstates the results.","tokens_in":10743,"tokens_out":4083,"would_cite":false,"duration_ms":37999,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Imbalanced regression should treat hard-to-predict instances as rare, not only infrequent target values.","keywords":["imbalanced regression","instance hardness","relevance function","resampling","random oversampling","Gaussian noise","bimodal distribution","rare instance identification"],"falsifier":"Run the bimodal experiment with a known generative mixture (e.g., two Gaussians with known weights) and define true rare regions by the mixture's low-density components; if InHaR's top-30% median-hardness bins do not align with those regions, or if density-based resampling beats it on MAE/MSE there, the identification claim fails. Simpler: compute IH with a regressor pool disjoint from the downstream model and see whether the resampling wins disappear.","tokens_in":9988,"feed_emoji":"🎯","tokens_out":6248,"duration_ms":56957,"temperature":0.7,"pith_summary":"This paper argues that for imbalanced regression, rarity should be measured by how hard an instance is for a learning algorithm, not by how infrequent its target value is. It proposes InHaR, a relevance function that scores each instance by the normalized prediction error of a pool of regressors, and uses a fixed threshold to label a subset as rare. Unlike the standard target-frequency relevance function, InHaR can flag hard instances inside dense regions, such as the valley between two modes of a bimodal target. When InHaR guides random oversampling and Gaussian-noise resampling, the paper reports improved MAE and MSE on 29 datasets plus a synthetic bimodal study, with most pairwise wins statistically significant. The point of caring: resampling decisions currently built on a density-only notion of rarity may be targeting the wrong instances.","feed_headline":"Resample hard cases, not rare targets, in imbalanced regression","feed_subtitle":"Hardness-guided resampling wins more pairwise tests than target-frequency relevance across 29 datasets.","key_machinery":"The load-bearing object is the Instance Hardness-based relevance function (InHaR), which replaces the target-value-to-relevance mapping phi(y) with a per-instance difficulty score: IH = 1 - average over a pool of regressors of exp(-relative squared error / gamma). The exponential-error form, normalized by gamma = mean(y^2), turns any regressor's prediction error into a [0,1] hardness; a fixed threshold tau = 0.7 then partitions the dataset into rare and normal sets for resampling. The identity does the work: it lets the attribute space and learning behavior enter the rarity decision, which density-only relevance cannot.","core_discovery":"The central claim is that learning-based instance hardness is a more useful notion of rarity for imbalanced regression than target marginal density. Traditional relevance functions assign fixed importance to extreme or low-frequency target values and fail in bimodal distributions, where the low-density region between modes is not genuinely rare. InHaR defines relevance directly as IH_L(x_i, y_i) = 1 - (1/|L|) sum_j exp(-d(y_i, h_j(x_i)) / gamma), with gamma the mean squared target, so instances that are consistently hard to predict get high relevance regardless of their frequency. The paper shows that IH and the traditional relevance function are only partially correlated, that InHaR flags r","pith_inferences":["A testable next step would be to calibrate the threshold tau per dataset or to replace the fixed 0.7 with a data-driven quantile; if the gains shrink under cross-validated selection, the fixed threshold is doing more work than the hardness principle.","The same hardness-based relevance could be plugged into synthetic oversampling variants beyond RO and GN; if IH identifies hard regions, synthetic oversampling should benefit at least as much as duplication.","A stronger validation of the bimodal claim would define true rarity from the known data-generating mixture rather than from IH itself; absent that, the 'correctly identifies' result is partly self-referential."],"forward_implications":["Relevance-based resampling need not be tied to target marginal density; any hardness measure can drive RO and GN while preserving the existing workflow.","In bimodal target distributions, InHaR can mark mid-distribution instances as rare, so resampling can focus on the hard valley rather than only the tails.","Because IH and the traditional relevance function are only partially correlated, the two definitions select different instances, so the choice of relevance criterion changes which regions a model ends up improving.","Using relevance-free metrics for comparison shows that hardness-guided resampling can improve normal-region performance rather than degrading it, unlike aggressive oversampling of statistical extremes.","The method generalizes across five regressor families, suggesting the hardness signal is not tied to one learning algorithm."],"fun_headline_variants":["Hardness beats rarity for imbalanced regression resampling","Why rarity fails: hardness drives imbalanced regression gains","InHaR: Resample hard instances, not just rare targets","For bimodal targets, hardness beats rarity in regression","Imbalanced regression? Focus on hardness, not frequency"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method assumes that an instance's prediction difficulty, measured on the same kinds of regressors that will be trained later, reveals which instances are worth duplicating during resampling—rather than reflecting noise or outliers that amplification would make worse.","fun_headline_variants_meta":{"raw":{"variants":["Hardness beats rarity for imbalanced regression resampling","Why rarity fails: hardness drives imbalanced regression gains","InHaR: Resample hard instances, not just rare targets","For bimodal targets, hardness beats rarity in regression","Imbalanced regression? Focus on hardness, not frequency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1153,"prompt_tokens":771,"completion_tokens":382,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":303}},"tokens_in":515,"tokens_out":382,"duration_ms":3966,"temperature":1.0,"reasoning_tokens":303,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T10:32:50.320180+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the bimodal experiment with a known generative mixture (e.g., two Gaussians with known weights) and define true rare regions by the mixture's low-density components; if InHaR's top-30% median-hardness bins do not align with those regions, or if density-based resampling beats it on MAE/MSE there, the identification claim fails. Simpler: compute IH with a regressor pool disjoint from the downstream model and see whether the resampling wins disappear.","supporting_citations":[],"review_version":1}