{"id":"3096d109-bd08-4308-bc02-e925e458c353","arxiv_id":"2607.24145","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Greedy recursive elimination almost consistently beats one-shot global feature ranking on supervised metrics across five selectors, at higher computational cost.","lead":"Recursive one-by-one feature elimination usually picks better features than ranking all features once. The study shows this across five selectors and many datasets, at higher compute cost.","discovery_kind":"extension","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The paper’s own rank diagrams do not support “almost consistent” improvement across the multi-metric benchmark; the result appears strong mainly for supervised ACC/AUC, especially under standard ranks.","rationale":"Read charitably, the paper makes a useful controlled comparison: the same importance criterion is used in a one-shot ranking and in recursive elimination, and the standard-rank ACC and AUC diagrams do consistently favor the iterative versions. That evidence plausibly supports a narrower design recommendation for supervised selection on these datasets. The least secure part is therefore not the existence of any effect, but the breadth of the headline claim. The reader’s rationale already notes that unsupervised metrics weaken the broader story, so this concern partially overlaps with that assessment. However, the reader’s designated weakest assumption concerns extrapolation beyond 240 dimensions. I see a more immediate internal issue: even within the evaluated low-dimensional benchmark, several reported metrics and MARS summaries do not show the claimed consistency, and some captions misdescribe their own diagrams. This does not require rejecting the paper. The ACC/AUC evidence remains potentially useful, and the proposed paired analysis could show that the aggregate effect is still favorable under a defensible weighting. But CONDITIONAL remains appropriate: the authors should reconcile MARS and standard-rank results, correct the metric-specific captions, and scope the conclusion to where the evidence actually holds. The separate low-dimensional generalization concern still matters for the curse-of-dimensionality framing, but it is secondary to establishing the claimed consistency on the benchmark itself.","tokens_in":14641,"tokens_out":2512,"duration_ms":64555,"concrete_test":"Using the per-dataset, per-budget CV scores, compute paired differences between each selector and its iterative variant at every matched feature budget; average per dataset, then report win/tie/loss and a Wilcoxon signed-rank test for each metric and rank statistic (positive for ACC/AUC/CLSACC/NMI, negative for AAD), with Holm correction. If iterative variants do not win for a majority of selector–metric cells, narrow the claim to supervised ACC/AUC.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is cross-algorithm and cross-metric, not merely that recursive elimination helps on ACC. For that claim to hold, each base selector should generally improve when made iterative across the reported criteria. The figures are much more mixed—and sometimes point the other way. In Fig. 4, the standard-rank AUC diagram favors all five iterative variants, but the MARS AUC diagram ranks standard XGBoost first and iterative XGBoost last; standard Permutation and ReliefF also beat their iterative counterparts. Fig. 5 is starker: in the standard-rank CLSACC diagram, every standard selector ranks ahead of its iterative counterpart, and MARS is also predominantly favorable to the standard versions. Fig. 7 (AAD, lower better) similarly favors standard ReliefF, RF, Permutation, and XGBoost in the standard-rank diagram, with only LASSO improving iteratively; MARS remains mixed. Nevertheless, the captions for Figs. 4–7 repeatedly state that both rank statistics agree on the superiority of the iterative approach. That is not what the displayed rankings show. Thus the supervised finding may be real, but the broader “multi-metric, almost consistent” conclusion is currently overclaimed and internally inconsistent with several of the paper’s principal summaries.","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper asks an algorithmic-design question about feature selection: does computing importance scores once over the full feature set (global/one-shot) differ from greedily removing the least important feature and re-evaluating importance at each step (iterative/RFE-style)? The authors implement five supervised selectors (Random Forest MDI, XGBoost MDI, ReliefF, LASSO, permutation importance) in both variants, evaluate them on 28 PMLB datasets across nested subset budgets (5%–100% of features) with 5-fold CV, and compare them on ACC, AUC, clustering accuracy (CLSACC), NMI, a PCA-based model-agnostic metric (AAD), ranking similarity, and runtime, using both standard average ranks and the authors' magnitude-aware rank statistic (MARS). The manuscript claims the iterative design improves feature-selection quality 'almost consistently' across metrics, at higher computational cost, and interprets this as evidence that high dimensionality also degrades the importance-estimation step itself.","tokens_in":14987,"tokens_out":3965,"duration_ms":113746,"significance":"The question is genuinely useful and, to my knowledge, not systematically studied: practitioners routinely choose between one-shot global importance ranking and recursive elimination, and paired evidence across five diverse selectors (tree MDI, boosting, ReliefF, LASSO, permutation) on 28 PMLB datasets with 5-fold CV is a real contribution. The supervised result — iterative variants occupying the top rank positions on ACC (Fig. 3) — is, if it holds up under paired testing, actionable guidance. The honest negative/mixed results on CLSACC and AAD are also valuable and would strengthen the paper if reported as such rather than papered over. However, the current version overclaims a cross-metric 'almost consistent' improvement that its own figures refute, and the dimensional regime studied (≤240 features) does not match the curse-of-dimensionality motivation. The contribution is salvageable and worthwhile at a re-scoped claim; it is not currently publishable as stated.","major_comments":[{"comment":"The captions of Figs. 4, 5, and 7 each state that 'Both MARS and Standard rank statistics agree on the superiority of the iterative approach,' but the displayed rankings contradict this. Fig. 4 (AUC), MARS diagram: standard XGB ranks first (9.24) and iterative XGB last (453.53); standard Permutation and ReliefF also outrank their iterative variants. Fig. 5 (CLSACC), standard-rank diagram: every standard selector ranks ahead of its iterative counterpart (Lasso 4.04 < Lasso_iter 4.36; XGB 4.84 < XGB_iter 5.32; RF 5.39 < RF_iter 5.45; ReliefF 5.75 < ReliefF_iter 6.18; Perm 6.25 < Perm_iter 7.43). Fig. 7 (AAD), both diagrams predominantly favor the standard variants of ReliefF, RF, Perm, and XGB. The body text for CLSACC is more careful ('not as consistently superior as for the supervised measure'), but the captions and the Abstract's 'almost consistently' are not supported by the paper's ow","section":"§5.2, Figs. 4–7 captions"},{"comment":"Related to the above: the Abstract and §6 claim the greedy design 'improves the overall feature selection quality almost consistently,' yet by my reading of the displayed rank diagrams the iterative variant is clearly superior only for ACC (Fig. 3, where the top three positions are iterative in both statistics) and for AUC under standard ranks. NMI (Fig. 6) is mixed, and CLSACC/AAD point the other way under standard ranks. Since the paper's contribution is precisely the empirical claim about granularity, the mismatch between claim and evidence is load-bearing. A revision should either (a) restrict the claim to supervised downstream metrics and explain why supervised selectors improving supervised metrics is itself the interesting (and perhaps expected) result, or (b) provide additional analysis supporting the broader claim.","section":"Abstract; §5.2; §6"},{"comment":"The aggregate critical-difference diagrams pool all ten variants (five base selectors x two designs). This shows overall placement but never directly tests the paired comparison the paper is actually about: RF vs RF_iter, XGB vs XGB_iter, etc., across the 28 datasets. Standard practice (Demšar, ref. [8], which the paper cites) would be a Wilcoxon signed-rank test per base-estimator pair per metric. Without paired tests, statements like 'the iterative variants occupy superior rank positions' do not establish that making any given selector iterative significantly helps; a single strong base method can shift pooled ranks. This is fixable with the existing experimental output.","section":"§5.2 (rank analysis methodology)"},{"comment":"The motivating story is the curse of dimensionality obscuring feature-importance estimation, yet the benchmark deliberately caps dimensionality at 240 features (§4.2), with most datasets below 100. The only genuinely high-dimensional example, COIL-20 (1024 features, Fig. 1), is used as an anecdotal t-SNE illustration and is excluded from the quantitative benchmark. The exclusion rationale in §4.2 (sparse high-d data makes granular comparison difficult) is reasonable, but then the conclusion in §6 that high dimensionality degrades feature selection analogously to distance concentration is asserted, not demonstrated — the experiments show an iterative-vs-global gap in low-to-moderate dimensions, which if anything undercuts the dimensionality narrative. Either add experiments in a higher-dimensional regime or reframe the contribution as a granularity study independent of the curse-of-dimens","section":"§4.2; §6"},{"comment":"The dataset count is internally inconsistent: §1 states 'extensive experiments across 38 datasets,' §5 states '28 datasets,' and Table 1 lists exactly 28. If 10 datasets were dropped, the selection criterion must be stated; if 38 is simply wrong, correct it. As written, a reader cannot determine whether there is selection on results.","section":"§1 vs §5 and Table 1"}],"minor_comments":[{"comment":"CLSACC is defined in §4 as 'Clustering Accuracy' but listed in the §5 introductory paragraph as 'Class-Weighted Accuracy.' These are different metrics; unify the terminology.","section":"§4 vs §5"},{"comment":"Eq. (3) and surrounding text: S_0 = {1,...,n} uses n (the number of instances) for the feature index set; it should be d. Also the elimination order is stored in 'a sequence ε' but features are then described as 'appearing late in π' — ε and π are used inconsistently.","section":"§3, Eq. (3)"},{"comment":"Typos: affiliation 'Mahematics'; §3 'might be effected' (affected); Abstract 'on the expense' (at the expense); §5.1 'two show if there is a performance difference' (to show); §5.3 'most of the times, a one-time step' is garbled.","section":"title page; Abstract; §3; §5"},{"comment":"Fig. 1's t-SNE comparison is suggestive but anecdotal: t-SNE projections are stochastic and the visual 'better discrimination' is not quantified. Please label it explicitly as a motivating illustration, and note that COIL-20 does not appear in the benchmark (Table 1).","section":"Fig. 1"},{"comment":"MARS (ref. 24) is a CoRR preprint by the same authors and is one of the two rank statistics used to support the central claim. This should be disclosed in the text where MARS is introduced, and the robustness of conclusions to using only standard ranks (ref. [8]) should be stated explicitly — particularly since for AUC the two statistics disagree.","section":"§5.2; ref. [24]"},{"comment":"The statement in §2 that there is an 'absence of prior studies investigating the granularity of the feature selection process' is strong given that RFE (refs. [9, 12]) and stepwise/wrapper methods (ref. [13]) are established; the contribution is better framed as a systematic cross-method, cross-metric comparison of the two granularities rather than the first study of iterative elimination.","section":"§1"},{"comment":"Hyperparameter choices (50 estimators for RF/XGB but only 20 for the Permutation backbone; LASSO C=0.5 fixed across all datasets) are stated but not justified or sensitivity-checked. Since the Permutation variant uses a weaker backbone than RF, the RF vs Permutation cross-comparisons in the pooled diagrams are not on equal footing; a brief note suffices.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The evaluation pipeline leans heavily on the authors' own recent tooling and metrics: FSEVAL (benchmark suite, ref. 27), FSDEM (ref. 23), AAD (ref. 21), and MARS (ref. 24, a CoRR preprint) are all from the same group, and MARS is used as one of only two rank statistics supporting the central claim while being itself unrefereed. This does not invalidate the ACC/AUC findings, which rest on standard ranks, but the editor may wish to note that the \"both rank statistics agree\" framing amplifies the role of the authors' own statistic, and that the caption inaccuracies in Figs. 4–7 would have been caught by a careful read against the displayed values."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful bit is simple: for the same base criterion, one-at-a-time rescoring often beats one-shot global ranking on ACC/AUC across five selectors. That is not a new algorithm—RFE and wrappers are old and they cite them—but a controlled multi-selector check of design granularity is still worth having, and the paired setup, nested budgets, CV, similarity heatmaps, and runtime plots are done competently.\n\nCredit where due: the supervised CD plots mostly back the iterative side, the ranking-overlap figures make the “different features get chosen” point concrete, and they are honest that cost goes up. The curse-of-dimensionality framing is a reasonable motivation even if it is not a theorem.\n\nSoft spots, in proportion. First, the abstract/conclusion language (“almost consistently,” multi-metric superiority) is stronger than the figures. On CLSACC the standard variants often rank ahead of their iterative twins; AAD is mixed; MARS on AUC even puts standard XGB first and iterative XGB last. Several captions still say both rank statistics agree on iterative superiority. That is an overclaim relative to the displayed ranks, not a fabrication of the whole result—the ACC/AUC story can stand if they narrow the claim. Second, they deliberately capped datasets at ~240 features to avoid sparse ultra-high-d regimes, which undercuts the strongest version of the curse story they sell. Third, minor process noise: abstract says 38 datasets, body 28; heavy reliance on their own secondary stack (FSEVAL, AAD, FSDEM, MARS); no obvious code/data release in the text.\n\nWho it is for: people who ship feature-selection defaults or AutoML pipelines and care whether rescoring is worth the compute. Not a theory paper. I would send it to referees after they tighten claims to match the plots, fix the dataset-count inconsistency, and state the low-d limit up front. Worth engaging if you work on FS practice; skip if you only want a new selector or a high-d theory result.","headline":"Solid paired experiment on selection granularity with a real supervised win, but the multi-metric “almost consistent” claim does not match several of their own rank diagrams.","tokens_in":16052,"tokens_out":521,"would_cite":false,"duration_ms":20247,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Greedy recursive elimination almost always beats one-shot global ranking for the same feature-selection criterion.","keywords":["feature selection","recursive feature elimination","curse of dimensionality","algorithmic design","greedy elimination","feature importance","empirical study"],"falsifier":"Re-run the identical head-to-head comparison on several dense or sparse datasets with thousands of features; if the recursive variants no longer improve accuracy, AUC, clustering quality or angle-difference scores relative to one-shot ranking, the central claim fails.","tokens_in":15945,"feed_emoji":"📉","tokens_out":667,"duration_ms":15987,"temperature":0.7,"pith_summary":"Feature selection usually ranks every feature once on the full set and keeps the top ones. This paper asks whether noisy or redundant features can mask the true importance of better features during that single ranking. The authors re-implement five standard selectors so that the least important feature is dropped and importance is recomputed after every removal, then compare the two designs on the same criteria. Across dozens of datasets and multiple supervised, unsupervised, and model-agnostic metrics, the recursive design produces better feature subsets almost consistently. The price is higher runtime, because the selector must be run many times. The result supports the claim that the curse of dimensionality also degrades the very procedures meant to fight it.","feed_headline":"Recursive feature dropping beats one-shot ranking","feed_subtitle":"Same criteria, better subsets: noisy features also mask importance scores themselves","key_machinery":"Greedy recursive elimination: at each step the current least-important feature is discarded and the base importance estimator is re-run on the remaining features, producing an elimination order that becomes the final ranking.","core_discovery":"When the same feature-importance criterion is applied either once globally or iteratively by removing the least important feature and re-scoring, the iterative design yields higher-quality selected subsets on nearly every evaluation metric examined. The discrepancy is largest among the top-ranked features and shrinks only as larger fractions of the feature set are retained.","pith_inferences":["The same masking effect may also distort unsupervised filters and wrapper methods that were excluded from the study, suggesting a broader design principle.","If removing several least-important features per round still preserves most of the gain, recursive selection could become practical for much higher dimensions.","The result supplies a concrete reason why feature-selection stability metrics often disagree across algorithms: they may be measuring different points on a continuum of granularity rather than purely different criteria."],"forward_implications":["Practitioners can obtain better feature subsets from existing selectors simply by wrapping them in recursive elimination, without inventing new criteria.","One-shot global rankings should be treated as approximate when many noisy or collinear features are present.","The computational overhead of recursion becomes the main practical barrier, so batch or subspace variants become natural next engineering steps.","Any new feature-selection method whose scores depend on the current feature set should be evaluated under both one-shot and recursive protocols."],"fun_headline_variants":["Recursive elimination beats global ranking on feature quality","Greedy drop-one re-scoring yields better subsets than one-shot","Noisy features mask importance: recursive beats single ranking","Same criterion, iterative removal improves nearly every metric","Top features suffer most from one-shot global scoring"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The finding is assumed to carry over from the moderately dimensional datasets used (at most a few hundred features) to the truly high-dimensional regimes where the curse-of-dimensionality argument is usually invoked.","fun_headline_variants_meta":{"raw":{"variants":["Recursive elimination beats global ranking on feature quality","Greedy drop-one re-scoring yields better subsets than one-shot","Noisy features mask importance: recursive beats single ranking","Same criterion, iterative removal improves nearly every metric","Top features suffer most from one-shot global scoring"]},"model":"grok-4.5","effort":"low","cost_usd":0.003832,"raw_usage":{"total_tokens":1207,"prompt_tokens":798,"num_sources_used":0,"completion_tokens":61,"cost_in_usd_ticks":38324000,"prompt_tokens_details":{"text_tokens":798,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":348,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":798,"tokens_out":61,"duration_ms":7345,"temperature":1.0,"reasoning_tokens":348,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T22:34:20.801688+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the identical head-to-head comparison on several dense or sparse datasets with thousands of features; if the recursive variants no longer improve accuracy, AUC, clustering quality or angle-difference scores relative to one-shot ranking, the central claim fails.","supporting_citations":[],"review_version":1}