{"id":"7d128e8b-8733-46a8-8d44-fa383a91ecf1","arxiv_id":"2501.11526","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A meta-classifier trained on nearest neighbor graph features can approximate five reference instance selection methods on held-out datasets in a single pass, with comparable accuracy and large speedups.","lead":"This paper turns instance selection, the task of pruning training data for nearest neighbor classifiers, into a binary classification problem. A meta-classifier is trained on nearest neighbor graph descriptors from other datasets to label each instance as keep or remove, and the authors report accuracy close to five reference pruning methods at lower runtime.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'outperforms almost all cases' claim rests on AUARR, an area metric that rewards extra compression even when accuracy drops; under the reduction-capped metric, MetaIS is better only for ICF and worse for Drop3.","rationale":"The strongest claim in the abstract is that MetaIS gives results comparable to reference methods while cutting computation. The speedup claim has independent support from the single-pass design and from large measured speedups even in cross-language runs, although the lack of controlled benchmarks is a real caveat. The transferability assumption is a limitation but not a flaw for the experiments as scoped, since the paper only evaluates on tabular benchmarks. The most load-bearing weakness is the performance metric used to claim superiority: AUARR credits arbitrary extra reduction, so Table 3's wins may be an artifact. This directly undermines the headline 'outperforms in almost all cases' and changes how the central result should be read. The Table 2 constrained metric is the appropriate comparison for 'comparable,' and it shows a mixed picture. A matched-reduction test would settle whether the claimed advantage is real. The reader's weakest_assumption field concerns cross-domain transferability, which is not the same concern, but the reader's rationale already lists the AUARR issue first, so there is partial agreement.","tokens_in":21318,"tokens_out":8027,"duration_ms":91885,"concrete_test":"Re-run the Section 5.1 comparison using a matched-reduction protocol: for each dataset/fold and each reference method, choose the MetaIS threshold whose achieved reduction rate is closest to the reference method's reduction rate, then compare 1NN F1 at those matched points. Also report the number of datasets where MetaIS is within 1 percentage point of the reference F1. If the number of significant wins drops from 17/15/17/17/8 (Table 3) to roughly parity, the 'outperforms almost all cases' claim should be withdrawn and the abstract should say 'comparable at matched reduction.' This test is cheap because the per-threshold results already exist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claim is not robust. Section 5.1 and Table 3 use the unrestricted AUARR for MetaIS, while each reference IS method is scored as a single trapezoid truncated at its own reduction rate (Figure 4b). Because the x-axis is reduction rate, extending the MetaIS curve to higher reduction adds positive area even when accuracy is falling, as long as accuracy stays positive. The large mean differences in Table 3 for HMN-EI (+0.3896) and ENN (+0.1771) are exactly the low-reduction references: MetaIS can 'win' simply by deleting far more samples. The constrained AUARR_L in Table 2 tells a more moderate story: significant wins only for ICF; significant losses for Drop3; no significant difference for ENN, HMN-EI, and CCIS, with mean differences -0.0028, +0.0003, -0.0093, +0.0538, and -0.0144. Thus the claim that MetaIS 'outperforms the reference IS in almost all cases' is an artifact of the evaluation metric rather than evidence of superior accuracy at matched compression. The abstract's 'comparable results' claim may still survive, but the paper's advertised advantage does not; the practical value of the method must be judged on the matched-reduction comparison, which is much weaker.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MetaIS, a method that recasts instance selection as a per-instance binary classification problem. Each training sample is described by meta-features extracted from the nearest-neighbor graph, and a meta-classifier predicts whether the sample should be kept or removed. The meta-classifier is trained on labels produced by five reference instance selection algorithms (ENN, Drop3, ICF, HMN-EI, CCIS) applied to multiple source datasets, and is evaluated on held-out datasets using a leave-one-dataset-out protocol. The authors report accuracy-reduction tradeoffs on tabular benchmarks, execution-time comparisons, an analysis of meta-classifier choice, and a feature-importance study. The central claims are that MetaIS achieves results comparable to reference methods, that it 'outperforms the reference IS in almost all cases' under one metric, and that its selection cost is dominated by nearest-neighbor graph construction plus a single classifier pass.","tokens_in":21502,"tokens_out":5202,"duration_ms":60867,"significance":"The core idea of amortizing iterative instance selection into a trained meta-classifier is interesting and potentially useful: if the meta-learner transfers across datasets, expensive multi-pass pruning could be replaced by a single-pass prediction. The paper has real strengths: it provides a public implementation (MetaIS library), uses a methodologically appropriate leave-one-dataset-out protocol, reports statistical tests, and introduces both an unrestricted and a reduction-limited area metric. However, the strongest advertised advantage—outperforming reference methods 'in almost all cases'—is an artifact of the unrestricted AUARR metric. The matched-reduction AUARR_L results support only comparability, with mixed individual outcomes. The computational-complexity claim is plausible algorithmically but the empirical speedups are not measured under controlled implementation conditions. With the evaluation framing corrected, the paper would present a useful, though more modest, contribution.","major_comments":[{"comment":"","section":"§4.4, Table 3, §5.1"},{"comment":"","section":"§5.2, Tables 4 and 5"},{"comment":"","section":"§3, first paragraph"}],"minor_comments":[{"comment":"","section":"Throughout"},{"comment":"","section":"Tables 1-3"},{"comment":"","section":"Figure 8 caption"},{"comment":"","section":"Table 4"},{"comment":"","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is genuinely new: instead of using meta-learning to pick an instance selection algorithm for a dataset, the authors train a classifier to label individual instances as keep/remove using nearest-neighbor-graph meta-features. That is a useful reframing, and the dataset-level leave-one-out protocol is the right way to test it. The constrained results (Table 2, AUARR_L) support a modest, credible claim: at matched compression, MetaIS is comparable to ENN, HMN-EI, and CCIS, significantly better than ICF, and significantly worse than Drop3. That is an honest engineering result worth having.\n\nThe soft spots are real but fixable. The abstract's 'outperforms in almost all cases' and the Table 3 AUARR comparison are not supported. The unrestricted area metric gives MetaIS a free threshold sweep while each reference method is a single operating point, and extending the MetaIS curve to higher reduction adds area even when accuracy is falling. The stress-test note is right: the large mean differences for HMN-EI and ENN are mostly an artifact of measuring area beyond the reference reduction rate. The paper should lead with the constrained comparison and treat the speedup as the main practical selling point.\n\nThe runtime speedup numbers are also not controlled benchmarks: reference methods run in Java via RapidMiner, MetaIS runs in Python/numpy, and no attempt is made to align implementations or hardware. The speedups are suggestive, not measured. The NNG transferability assumption (Section 3) is plausible but only tested on tabular Keel/OpenML benchmarks; image, text, and other modalities are untested. The dataset tables are inconsistent (Table 1 lists 17 datasets, Table 2 has 19 rows, and the two large datasets lack Drop3), and the GitHub link is not pinned to a commit. These are all correctable in revision.\n\nWho gets value from this: practitioners pruning kNN training sets on tabular data, and meta-learning researchers interested in per-instance transfer. It does not reshape the field, but it is a sound, reproducible contribution with a clear limitation. I would send it to a serious referee, asking them to focus on whether the revised framing matches the constrained evidence.","headline":"The per-instance meta-classifier idea is real and the constrained comparison is honest; the headline 'outperforms almost all cases' is overstated because it relies on an unrestricted area metric that rewards extra compression even when accuracy drops.","tokens_in":22102,"tokens_out":1414,"would_cite":true,"duration_ms":17053,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Instance selection can be reframed as binary classification over nearest-neighbor graph statistics, so a trained meta-classifier prunes a new dataset in one pass with accuracy comparable to five reference instance-selection methods and…","keywords":["meta-learning","instance selection","data pruning","nearest neighbor graph","meta-features","Balanced Random Forest","data-centric AI","prototype selection"],"falsifier":"Train the meta-classifier on one set of tabular benchmarks and evaluate it on held-out image or text collections with the same reference labels; if the unrestricted AUARR on those collections is no better than always keeping everything, or is worse than a threshold tuned per dataset, the claimed generality of NNG meta-features fails.","tokens_in":21030,"feed_emoji":"⚡","tokens_out":7485,"duration_ms":74363,"temperature":0.7,"pith_summary":"This paper aims to show that the slow, iterative task of instance selection can be replaced by a fast classifier. The idea is to describe every training example by a fixed set of statistics about its nearest neighbors, such as distances and class-label counts at several neighborhood sizes, so that examples from any dataset live in the same meta-feature space. Labels come from running a reference instance-selection algorithm on training datasets: keep or remove. Once a meta-classifier is trained on those labels, pruning a new dataset becomes a nearest-neighbor search plus one prediction per example. The paper reports that this one-pass method matches the accuracy-compression trade-off of the reference methods on 17 tabular datasets while being much cheaper, with speedups that grow with dataset size.","feed_headline":"One-pass meta-classifier prunes datasets up to 219x faster","feed_subtitle":"Trained on nearest-neighbor graph statistics, it matches reference methods while cutting compute to one pass.","key_machinery":"The load-bearing object is the meta-feature space built from the nearest-neighbor graph (NNG). For each vertex (training sample), the method records the average and minimum squared Euclidean distance to neighbors in the same class, the opposite class, and any class, plus the counts of same-class and opposite-class neighbors among the k nearest, computed for k = 3, 5, 9, 15, 23, 33, with -1 inserted for missing opposite-class values. These statistics are normalized by feature count so datasets of different dimensionality become comparable, and the same fixed-size descriptor vector is extracted for every dataset. The meta-classifier, Balanced Random Forest in the recommended setup, maps these descriptors to a removal probability, and a threshold converts that probability into a keep/remove decision; the AUARR metric, area under the accuracy-versus-reduction-rate curve, is introduced to compare the resulting continuum of operating points against the single point returned by a reference algorithm.","core_discovery":"On its own terms, the paper claims that any instance-selection algorithm that works through the nearest-neighbor graph leaves a learnable signature: the local geometry around each training vector, summarized by distance and neighbor-count statistics, determines whether the algorithm keeps or removes that vector. By extracting the same fixed set of these statistics for every dataset and labeling each vector with keep/remove decisions from a reference algorithm, the authors build a single meta-dataset on which a Balanced Random Forest classifier is trained; applying that classifier to a new dataset's nearest-neighbor graph yields a removal probability per instance, and thresholding it reproduces reference-method behavior. In leave-one-dataset-out experiments over 17 tabular datasets and five reference methods (ENN, Drop3, ICF, HMN-EI, CCIS), the resulting meta-instance selection is statistically indistinguishable from ENN, HMN-EI, and CCIS on the compression-limited area-under-curve metric, significantly better than ICF, and significantly worse than Drop3; when the comparison area is not capped by the reference reduction rate, the meta-model wins in all but the Drop3 comparison. The advertised payoff is complexity: selection is a single pass over the dataset, with runtime governed only by nearest-neighbor finding and a constant-cost meta-classifier, yielding measured speedups up to 219x over Drop3.","pith_inferences":["If transferability holds beyond the 17 tabular benchmarks, the method turns any expensive iterative instance-selection algorithm into a one-time training cost plus per-dataset nearest-neighbor construction, making nearest-neighbor search the practical bottleneck.","The continuous removal probability provides a built-in compression-accuracy curve, so a natural extension is to use that probability as a sample-importance score for active learning, noisy-label detection, or budget-constrained training, though the paper does not test those uses.","Because the feature-importance analysis shows each reference method relies on different NNG descriptors, an ensemble of reference-label-specific meta-classifiers could carry complementary information; the paper mentions diversity as a motivation but does not build the ensemble.","A direct test of the core assumption would be applying the same pipeline to non-tabular data such as image patches or text embeddings; the paper's experiments are limited to tabular datasets."],"forward_implications":["A new dataset can be pruned in a single pass: compute nearest neighbors, extract meta-features, and apply the meta-classifier, without iterating through candidate removals.","For large datasets and slow reference algorithms, the measured speedup grows with dataset size, reaching about 219x over Drop3 in the reported experiments.","Users can trade compression against accuracy after the fact by setting a threshold on the removal probability, instead of rerunning the selection algorithm.","The approach is generic across NNG-based instance-selection methods: the same pipeline reproduces or improves the behavior of ENN, HMN-EI, CCIS, and ICF, and in the unrestricted AUARR comparison it significantly beats four of the five reference methods, with Drop3 the only exception.","With nearest-neighbor index structures, the selection phase can run in log-linear or near-linear time, independent of the reference algorithm's iterative cost."],"supporting_citations":[{"why":"Defines the Edited Nearest Neighbor rule, one of the five reference instance-selection algorithms that generate the keep/remove labels for the meta-training set.","marker":"[13]"},{"why":"Defines the Drop3 reduction technique used as a reference method and as a label source for the meta-classifier.","marker":"[3]"},{"why":"Defines the Iterative Case Filtering (ICF) reference method and the local-set concepts that the meta-features approximate.","marker":"[4]"},{"why":"Introduces the Hit Miss Network and the HMN-EI instance-selection variant used as a reference labeler.","marker":"[16]"},{"why":"Defines the Class Conditional Instance Selection (CCIS) algorithm used as a reference labeler.","marker":"[42]"},{"why":"Provides the Balanced Random Forest implementation the paper recommends as the meta-classifier for imbalanced keep/remove labels.","marker":"[39]"},{"why":"Supplies the benchmark datasets and the Java implementations of the five reference instance-selection algorithms used in the experiments.","marker":"[43]"}],"fun_headline_variants":["Meta-classifier learns to prune instances from nearest-neighbor stats","Match classic pruning methods with a fraction of compute","Instance selection as classification: one-pass meta-pruning","Meta-pruning: up to 219x faster dataset compression","Balanced Random Forest identifies keep-or-remove instances"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that statistics of the nearest-neighbor graph are shared across domains, so that a classifier trained on meta-features from some datasets can decide keep/remove on a dataset it has never seen.","fun_headline_variants_meta":{"raw":{"variants":["Meta-classifier learns to prune instances from nearest-neighbor stats","Match classic pruning methods with a fraction of compute","Instance selection as classification: one-pass meta-pruning","Meta-pruning: up to 219x faster dataset compression","Balanced Random Forest identifies keep-or-remove instances"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000878,"raw_usage":{"total_tokens":3846,"prompt_tokens":1044,"completion_tokens":2802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":2723}},"tokens_in":660,"tokens_out":2802,"duration_ms":21132,"temperature":1.0,"reasoning_tokens":2723,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:09:20.573139+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the meta-classifier on one set of tabular benchmarks and evaluate it on held-out image or text collections with the same reference labels; if the unrestricted AUARR on those collections is no better than always keeping everything, or is worse than a threshold tuned per dataset, the claimed generality of NNG meta-features fails.","supporting_citations":[{"cited_title":"Wilson, Assymptotic properties of nearest neighbour rules using edited data., IEEE Trans","cited_arxiv_id":null,"evidence_quote":"Defines the Edited Nearest Neighbor rule, one of the five reference instance-selection algorithms that generate the keep/remove labels for the meta-training set."},{"cited_title":"Wilson, T","cited_arxiv_id":null,"evidence_quote":"Defines the Drop3 reduction technique used as a reference method and as a label source for the meta-classifier."},{"cited_title":"Brighton, C","cited_arxiv_id":null,"evidence_quote":"Defines the Iterative Case Filtering (ICF) reference method and the local-set concepts that the meta-features approximate."},{"cited_title":"Marchiori, Hit miss networks with applications to instance selection, Journal of Machine Learning Research 9 (Jun) (2008) 997–1017","cited_arxiv_id":null,"evidence_quote":"Introduces the Hit Miss Network and the HMN-EI instance-selection variant used as a reference labeler."},{"cited_title":"Marchiori, Class conditional nearest neighbor for large margin in- stance selection, IEEE Transactions on Pattern Analysis and Machine Intelligence 32 (2) (2010) 364–370","cited_arxiv_id":null,"evidence_quote":"Defines the Class Conditional Instance Selection (CCIS) algorithm used as a reference labeler."},{"cited_title":"Triguero, S","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark datasets and the Java implementations of the five reference instance-selection algorithms used in the experiments."}],"review_version":1}