{"id":"de82c7d2-9311-40d2-951f-ebb0d681bfb2","arxiv_id":"2412.14527","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Two undersampling methods, MI-based stratified sampling and support point optimization, are applied to class-imbalanced classification and compared to random undersampling on two datasets.","lead":"This paper proposes two ways to shrink the majority class in imbalanced datasets before training a classifier: one uses mutual information to group similar examples, and the other uses support points to pick representative examples. The authors report higher balanced accuracy than random undersampling in two small tests, but the evaluation method makes the improvement hard to trust.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation protocol undersamples before the train/test split, so test sets are method-dependent and balanced; reported gains may be artifacts of test-set construction and single-split noise.","rationale":"Agree with the reader's weakest assumption. The central claim—higher balanced accuracy than traditional undersampling—rests entirely on Tables 1 and 5, and both tables are generated under the Section 5.1 protocol in which undersampling precedes the train/test split. This creates two intertwined problems. First, the test set is method-dependent: each undersampling method selects its own majority subset, and the test set is a random 20% of that subset. Thus the test distribution is not the original majority distribution; it is a balanced, already-filtered distribution. A method that deliberately selects majority points far from the decision boundary, or in low-density regions, can make the test set artificially easy. Second, the data size makes noise a serious issue: the breast-cancer dataset has only 272 samples, the resampled balanced set has roughly 162 points, and the test fold has roughly 32 points. A single 80/20 split with no replication cannot support a claimed over-30% improvement. The support-points results show only small differences (0.9225–0.9401), so they do not rescue the broad abstract claim. The paper also provides no code or data, which compounds the difficulty of adjudicating the protocol. A corrected evaluation with original train/test split before resampling would settle whether the methods genuinely generalize. The concern is not about disagreement with current consensus; it is an internal validity issue in the paper's own evaluation design.","tokens_in":8449,"tokens_out":5917,"duration_ms":53154,"concrete_test":"Re-run the mutual-information breast-cancer experiment (Section 6.1) with the corrected protocol: split the original 272-sample imbalanced data into 80% training and 20% held-out test (preserving class proportions) before any undersampling; apply random undersampling and MI-stratified undersampling only to the training fold; train the same four classifiers; evaluate balanced accuracy on the untouched original test fold. Repeat with at least 30 random seeds and report mean and standard deviation. If MI does not beat random undersampling by a margin exceeding the sampling noise, the claimed improvement is an evaluation artifact. The analogous check should be run on the credit-card support-points experiment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing flaw is the evaluation protocol in Section 5.1, step 3: the 80-20 split is applied after undersampling, so the test set is not a sample from the original imbalanced distribution. For each method, both training and test majority points are selected by that method; the test sets therefore differ across methods and are not independent of the sampling procedure. If the MI-stratified selection tends to pick majority points that are easier to separate from the minority class, the large gains in Table 1 (0.5455 to 0.8485 for logistic regression) can be an artifact of test-set construction rather than of genuine generalization. Compounding this, the balanced test set for the 272-sample breast cancer dataset is only about 32 points, and the experiments use a single split with no repeated seeds or error bars, so the reported over-30% improvement could be sampling noise. The same protocol is used for the support-points results in Section 6.2.3, although those gains are small. The central claim in the abstract therefore is not supported as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two undersampling strategies for imbalanced classification: a mutual-information-based stratified simple random sampling method and a support-points optimization method. The MI method computes a purported pairwise 'element-wise' mutual information matrix among majority-class samples, converts it into a dissimilarity matrix, clusters via K-means, and then applies stratified sampling. The support-points method minimizes the empirical energy distance between a reduced majority subset and the original majority distribution, mapping optimized points to nearest neighbors. Experiments compare balanced accuracy against random undersampling on a 272-sample breast cancer dataset (MI) and a 284,807-sample credit card fraud dataset (support points). The paper reports large MI gains for logistic regression, random forest, and SVC, and small gains or ties for support points across four classifiers.","tokens_in":8644,"tokens_out":4342,"duration_ms":40448,"significance":"If the empirical claims were valid, the support-points application would be a modest but reasonable extension of Mak and Joseph (2018) to class-imbalance problems, and the paper's quantitative representativeness checks (feature-wise statistics and KS tests) are appropriate. However, the central evaluation protocol is flawed: the train/test split occurs after undersampling, so the test sets are method-dependent and artificially balanced. The MI estimator is also not mathematically well-defined, as mutual information is a distributional quantity rather than a per-point score. The reported gains, especially the >30% logistic-regression improvement in Table 1, are therefore not interpretable as generalization improvements. The current manuscript does not establish its headline claim that the proposed methods outperform traditional undersampling.","major_comments":[{"comment":"The evaluation protocol is invalid for the central claim. The 80-20 train/test split is applied after the majority class has been undersampled, so the test set contains the same method-selected majority points used for training, is balanced, and differs across methods. Consequently, the reported balanced accuracies do not measure performance on the original imbalanced distribution; a sampling method that preferentially selects majority points that are easy to separate from the minority class can inflate test accuracy without improving genuine generalization. For example, the logistic-regression improvement from 0.5455 to 0.8485 in Table 1 is an artifact of this protocol. A correct protocol would split the original data into train and test first, then undersample only the training fold, and evaluate on the untouched test set.","section":"§5.1, Step 3; §6.1.2, Table 1; §6.2.3, Table 5"},{"comment":"The proposed 'element-wise' mutual information computation is not well defined. Mutual information I(X;Y) is a functional of the joint and marginal distributions of two random variables, not a per-data-point quantity. The text says the authors calculate MI between each x and y pair and store the values in a symmetric matrix, but for fixed observations x_i and y_j there is no joint distribution PXY(x_i, y_j) to estimate in the manner described. The algorithm therefore does not implement the formula in §4.1.1, and the stratification based on this heuristic is not justified by the stated information-theoretic rationale.","section":"§4.1.1 and §4.1.2"},{"comment":"The MI experiments use a single split with no repeated seeds, no cross-validation, and no uncertainty quantification. The balanced test set for the breast cancer dataset is only about 32 points after the 80-20 split, so the reported 'over 30%' improvement for logistic regression and 'over 20%' for random forest could be within split-to-split variability. The paper needs repeated trials or confidence intervals before such claims can be taken as evidence.","section":"§6.1.2, Table 1"},{"comment":"The support-points classification results show at most a 1.4-percentage-point improvement over random undersampling and one exact tie (XGBoost). Given that the same post-undersampling split protocol is used and no error bars are provided, the abstract's claim that support points 'outperform' traditional undersampling is not supported. The representativeness analysis in §6.2.2 is informative but reports only selected features and does not by itself establish classifier-level superiority.","section":"§6.2.3, Table 5; §6.2.2"}],"minor_comments":[{"comment":"There are numerous grammatical errors that impede readability, e.g., 'classification tasks machine learning' in the abstract and 'it impacted machine learning models' in §3; these should be corrected.","section":"Abstract and §1"},{"comment":"The text says the resampled data will be 'spitted' into an 80-20 split; this should be 'split'.","section":"§6.1.1"},{"comment":"The citation to Bhagat and Patil (2015) is incomplete and does not appear in the reference list; the authors should add the full reference.","section":"§2.2"},{"comment":"The energy distance formula is written without explicit expectation notation and the norm is undefined; specifying the norm and the underlying probability spaces would improve clarity.","section":"§4.2.1, Eq. (1)"},{"comment":"The claim that direct pairwise computations require over 600GB of memory is not explained; the complexity analysis in §4.2.4 also omits the cost of the clustering-based subsampling stage.","section":"§6.2.1"}],"recommendation":"reject","confidential_remarks":"The paper appears to be an early-stage project report rather than a journal-ready contribution. The evaluation protocol flaw is load-bearing and affects both experimental sections; the MI method is not a valid mutual-information estimator; and the support-points improvements are small and unquantified. I do not see a path to acceptance without substantially reworking the methodology and the experiments, which goes beyond standard revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe genuinely new piece is the application of support points to undersampling for class imbalance. Mak and Joseph introduced support points for UQ and Bayesian computation; using them to downsample the majority class is a sensible, principled idea, and the authors back it with feature-wise statistics and KS tests, which is more than most undersampling papers bother with. The MI-based stratified SRS is a new combination, but it rests on an ill-defined 'element-wise mutual information' between data points, and the O(n^2) cost limits it to tiny datasets.\n\nThe stress-test note is right about the load-bearing flaw. The 80-20 split in Section 5.1 happens after undersampling, so the test set is method-dependent and artificially balanced. For the breast cancer dataset, the balanced test set is only about 32 points, single split, no error bars, so the 0.5455 → 0.8485 jump for logistic regression could easily be noise or an artifact of how the test set was constructed. The support points results on the credit card data are more stable, but the gains over random undersampling are about 1–2%, which does not support the abstract's claim of outperforming traditional undersampling. The paper only compares against random undersampling, not against other cluster-based or k-means undersampling, and no code or data is provided.\n\nThat said, the paper is not a throwaway. The support points idea deserves a proper evaluation: split first, then undersample only the training set, repeat with several seeds, and compare to existing methods. The current experiments do not support the central claim, but the flaw is fixable and the core concept is worth engaging. It reads like a project report — rough writing, confused structure — but the thinking is honest.\n\nIf I were the editor, I would send this to review with the expectation of major revisions, not desk reject it. It is also a useful reading-group example of why evaluation protocol matters in imbalanced learning.\n\nRecommendation: worth a referee, but only with serious revisions.","headline":"Support points for undersampling is a principled new application, but the post-hoc split invalidates the reported gains.","tokens_in":9171,"tokens_out":3302,"would_cite":false,"duration_ms":29598,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Information-aware undersampling methods improve balanced accuracy by preserving the structure of the majority class.","keywords":["class imbalance","undersampling","mutual information","stratified sampling","support points","energy distance","balanced accuracy","imbalanced classification"],"falsifier":"Re-run the breast-cancer experiment with a held-out test set taken from the original 272 samples before any undersampling, then compare logistic regression balanced accuracy under random undersampling versus the mutual-information method; if random undersampling matches or beats the proposed method on that held-out set, the paper's central claim fails.","tokens_in":8230,"feed_emoji":"⚖️","tokens_out":7846,"duration_ms":59377,"temperature":0.7,"pith_summary":"This paper argues that undersampling does not have to be a blind random draw: it can be a statistically informed selection step. It proposes two such selection rules: stratifying majority-class points by their pairwise mutual information and then sampling within strata, and optimizing a small set of support points to minimize empirical energy distance to the full majority distribution. On a small breast-cancer dataset, the mutual-information approach is reported to raise balanced accuracy over random undersampling by over 30% for logistic regression, over 20% for random forest, and over 12% for a support-vector classifier. On a large credit-card fraud dataset, the support-points approach matches or slightly exceeds random undersampling for all four classifiers tested. A sympathetic reading of the paper is that information-aware subset selection is a promising middle path between random undersampling and synthetic oversampling.","feed_headline":"Mutual-information sampling beats random undersampling by 30%","feed_subtitle":"Smarter subset selection preserves majority-class structure and lifts balanced accuracy.","key_machinery":"The central object is the empirical energy distance $\\hat{E}(X,Z)$ between the full majority dataset $X$ and the candidate subset $Z$, defined as $2\\cdot\\frac{1}{Nm}\\sum_{i,j}\\|x_i-z_j\\| - \\frac{1}{N^2}\\sum_{i,i'}\\|x_i-x_{i'}\\| - \\frac{1}{m^2}\\sum_{j,j'}\\|z_j-z_{j'}\\|$. Minimizing it with the gradient update pushes the selected points toward the majority distribution while keeping them from collapsing, which is the mechanism the paper claims preserves feature-wise means and standard deviations. The mutual-information branch uses the pairwise matrix $I(X,Y)$ computed by brute force, converted into a dissimilarity matrix and clustered into strata; stratified simple random sampling then draws points from each stratum. The paper validates the support-points branch with feature-wise statistics and Kolmogorov-Smirnov tests compared against the original majority class.","core_discovery":"On its own terms, the paper establishes that both proposed undersampling methods improve or match the balanced accuracy of random undersampling because they keep more of the majority class's distributional structure. The mutual-information branch computes pairwise mutual information among majority points, clusters the resulting dissimilarity matrix into four strata, and applies stratified simple random sampling with Neyman or optimal allocation. The support-points branch clusters the majority class into 50 groups, extracts 5,000 representative points, then refines them by gradient descent on the empirical energy distance, reaching a final energy distance near 0.0102 and mapping the optimized points back to nearest neighbors in the original data. The reported balanced accuracies on the breast-cancer dataset are 0.8485 versus 0.5455 for logistic regression, 0.6970 versus 0.4848 for random forest, and 0.7273 versus 0.6061 for SVC, all against random undersampling. On the credit-card fraud dataset, support points give balanced accuracy at or slightly above random undersampling for logistic regression, random forest, XGBoost, and SVC.","pith_inferences":["A natural next test, not reported in the paper, is to hold out a test set from the original imbalanced distribution before any undersampling; the current 80-20 split after undersampling measures accuracy on a balanced subset and likely overstates real-world performance.","The MI stratification idea could be paired with subquadratic or upper-bound MI estimators to escape the $O(n^2)$ bottleneck, potentially making the method applicable to mid-sized datasets.","The support-points branch's final energy distance and KS-test results suggest the optimized points retain global distributional shape; a testable extension would be to compare support points against other stratified sampling methods on the same energy distance, not only against random undersampling."],"forward_implications":["If the breast-cancer results replicate, imbalanced medical datasets with only a few hundred samples can be undersampled in under a minute while gaining more than 30 percentage points of balanced accuracy for simple linear models.","If the support-points results hold at scale, a majority class of hundreds of thousands of points can be summarized by a few thousand representative points with minimal loss of distributional fidelity.","The MI branch's brute-force pairwise computation is $O(n^2)$, so its practical use is limited to small datasets unless a cheaper mutual-information estimator is substituted.","The support-points branch's $O(T(N'md + m^2d))$ cost makes clustering-based pre-subsampling necessary for very large datasets, as the paper itself does for the 284,807-sample fraud dataset."],"supporting_citations":[{"why":"Introduces support points and the empirical energy distance objective used in the support-points branch.","marker":"Mak & Joseph (2018)"},{"why":"Supplies the mutual-information maximization idea for dataset distillation that the authors adapt to undersampling.","marker":"Shang et al. (2024)"},{"why":"Gives the pairwise mutual-information complexity the paper relies on to justify the O(n^2) cost of the MI branch.","marker":"Ferdosi et al. (2020)"},{"why":"Provides the two empirical assumptions about estimating joint and marginal probabilities used in the MI computation.","marker":"Fernandes & Gloor (2010)"}],"fun_headline_variants":["Mutual info and support points outperform random undersampling","Undersampling with support points lifts balanced accuracy","Smart undersampling: keep structure, lift accuracy","Support points sampling beats random undersampling in accuracy","Information-preserving undersampling improves classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that an 80-20 train-test split performed after undersampling measures how well the classifier generalizes; because the test set is also artificially balanced, the reported balanced accuracy describes performance on a balanced subset rather than on the original imbalanced population.","fun_headline_variants_meta":{"raw":{"variants":["Mutual info and support points outperform random undersampling","Undersampling with support points lifts balanced accuracy","Smart undersampling: keep structure, lift accuracy","Support points sampling beats random undersampling in accuracy","Information-preserving undersampling improves classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1470,"prompt_tokens":837,"completion_tokens":633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":563}},"tokens_in":453,"tokens_out":633,"duration_ms":5222,"temperature":1.0,"reasoning_tokens":563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:08:35.306322+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the breast-cancer experiment with a held-out test set taken from the original 272 samples before any undersampling, then compare logistic regression balanced accuracy under random undersampling versus the mutual-information method; if random undersampling matches or beats the proposed method on that held-out set, the paper's central claim fails.","supporting_citations":[{"cited_title":"and Joseph, V","cited_arxiv_id":null,"evidence_quote":"Introduces support points and the empirical energy distance objective used in the support-points branch."},{"cited_title":"Mim4dd: mutual information maximization for dataset distillation","cited_arxiv_id":null,"evidence_quote":"Supplies the mutual-information maximization idea for dataset distillation that the authors adapt to undersampling."},{"cited_title":"Measuring mutual information between all pairs of variables in subquadratic complexity","cited_arxiv_id":null,"evidence_quote":"Gives the pairwise mutual-information complexity the paper relies on to justify the O(n^2) cost of the MI branch."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the two empirical assumptions about estimating joint and marginal probabilities used in the MI computation."}],"review_version":1}