{"id":"b2c7db8f-079a-449e-8096-17c0b23da729","arxiv_id":"2501.14889","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"EASE combines attention-based subspace scoring with EWC-style incremental updates to evaluate and refine feature spaces more efficiently during iterative feature selection.","lead":"This paper introduces EASE, a learnable evaluator that scores candidate feature spaces during iterative feature selection using attention over feature subspaces and incremental updates instead of full retraining. The authors report modest but consistent accuracy gains across 14 datasets and several feature-selection frameworks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported superiority over baselines is not statistically established: Tables 2/5/6/8 show overlapping error bars and no significance tests, so the central performance claim may be within noise.","rationale":"The reader's weakest assumption focuses on whether the Fisher information in Eq. (10) correctly identifies parameters worth preserving across consecutive feature spaces. That is a real mechanism risk, but it is not the most load-bearing part of the paper's central claim. Even if the EWC-style penalty in Eq. (11) were ineffective, the performance claim could survive: Table 3 shows EASE-IT (the variant without incremental training) is competitive or better on some datasets, such as openml_586 and wine_white, and the time advantage could come from training on small subspaces rather than from the incremental update. The empirical performance claim itself, however, is the foundation of the paper, and the reported evidence has overlapping error bars, no significance tests, no paired comparisons, and no correction for multiple metrics and datasets. These issues were noted in the reader's rationale, but the reader's formal weakest assumption was the EWC similarity premise, so my agreement is partial. Because the appropriate remedy is additional statistical analysis and reproducibility checks rather than outright rejection, the reader's CONDITIONAL verdict remains appropriate, and I recommend UNCHANGED.","tokens_in":22932,"tokens_out":9872,"duration_ms":91901,"concrete_test":"Using the released code and the same 10 seeds, recompute for each dataset the per-seed difference EASE minus the best baseline on the primary metric (Accuracy for classification, MAE or RMSE for regression), then run a paired Wilcoxon signed-rank test across the 14 datasets with Holm correction across metrics, and report 95% bootstrap confidence intervals on the mean difference. If the overall comparison is not significant at p<0.05 or the confidence interval includes zero, the headline 'better downstream performance' claim is not supported by the evidence as presented.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that EASE yields better downstream performance than LR, DT, GBDT, RF, and XGB is supported only by mean-plus-minus-standard-deviation tables (Tables 2, 5, 6, and 8 in Section 5.2 and Appendix C) with no significance tests, paired or otherwise. Appendix C.1 states that all experiments ran 10 times; with n=10, many decisive entries overlap. For example, Table 2 openml_616 R2 is EASE 0.840±0.035 vs XGB 0.837±0.016, wine_red Accuracy is EASE 0.637±0.018 vs LR 0.617±0.042, and Table 6 svmguide3 Accuracy is EASE 0.856±0.014 vs RF 0.861±0.012, where RF is numerically higher. Without paired tests or confidence intervals on the differences, the claim that EASE 'outperforms other baselines in most cases' (Section 5.2.1) is not established. The efficiency half of the strongest claim is also undersupported: Figure 3 and Figure 6 compare cumulative time only against GBDT, RF, and XGB within the RFE framework, and no cumulative-time comparison against LR or DT, nor within FLSR, SDAE, or GRFG, is reported. This is the load-bearing gap: the headline empirical assertion, not the EWC mechanism in Eq. (11), is what most needs support. If a proper statistical test shows most differences are within noise, the central claim fails regardless of whether the incremental update is sound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EASE, a plug-in feature-space evaluator for iterative feature-space optimization. EASE decouples a feature space into feature-sample subspaces by selecting important features and hard samples, scores the subspaces with a weight-sharing multi-head attention network, and updates the network incrementally using elastic weight consolidation (EWC). The authors integrate EASE into RFE, FLSR, SDAE, and GRFG and compare its downstream performance and cumulative training time against LR, DT, GBDT, RF, and XGB as alternative evaluators. The central claims are that EASE produces better feature spaces and reduces cumulative training time across iterative optimization frameworks.","tokens_in":23316,"tokens_out":5532,"duration_ms":57218,"significance":"If the empirical claims held, EASE would be a practically useful plug-in for reducing the cost of iterative feature selection/generation while improving downstream performance. The component design is sensible: the feature-sample subspace generator targets challenging samples and relevant features, and the EWC-based incremental update is a plausible mechanism for exploiting overlap between consecutive feature spaces. The paper also ships code (via an anonymous repository), evaluates with an external RF model in the testing phase, and includes ablations (EASE-PT, EASE-IT, EASE-FC) that isolate the contribution of each component. These are real strengths. However, the headline superiority claim is not statistically established, the feature-importance definition in Eq. (3) is a derivation gap, and the efficiency claim is only partially demonstrated. The contribution is plausible but needs stronger evidence before the central claims can be accepted.","major_comments":[{"comment":"The central performance claim is not statistically supported. All experiments are run only 10 times (Appendix C.1) and no significance tests or confidence intervals on differences are reported. Many decisive entries have overlapping standard deviations, e.g., Table 2 openml_616 R2: EASE 0.840±0.035 vs XGB 0.837±0.016, and Table 6 svmguide3 Accuracy: EASE 0.856±0.014 vs RF 0.861±0.012, where a baseline is numerically higher. Without paired tests (e.g., Wilcoxon signed-rank) or confidence intervals on per-dataset differences, the statement that 'EASE outperforms other baselines in most cases' (Section 5.2.1) is not established. This is load-bearing for the abstract and the paper's main contribution, so the authors should add appropriate statistical evidence or soften the claim.","section":"5.2.1, Tables 2 and 5; Appendix C.1"},{"comment":"The feature importance score is defined as Score(f_i) = M(F(t); Theta) - M(F(t) \\ {f_i}; Theta), where M is the evaluator that produces predictions according to Eqs. (1)-(2). Subtracting raw evaluator outputs is not the same as measuring the impact on model performance; a feature can change predictions without changing loss or accuracy, and vice versa. The text in Section 4.1 says the score assesses 'the impact of removing that feature on the performance of the model,' but Eq. (3) does not implement that. Please either redefine M to output a scalar performance estimate (e.g., expected loss), or present Eq. (3) explicitly as a sensitivity heuristic and validate that it tracks actual performance differences.","section":"4.1, Eq. (3)"},{"comment":"The efficiency claim is only demonstrated for the RFE framework and only against GBDT, RF, and XGB. No cumulative-time comparison against LR or DT is reported, and no cumulative-time comparison is given inside FLSR, SDAE, or GRFG, even though the paper claims that EASE 'can efficiently assess feature space quality' in general and highlights efficiency as a key motivation (Section 1, Section 5.2.2). The reported speed-ups in Figures 3 and 6 are therefore specific to RFE with three baselines. Please add time comparisons in the other frameworks, or explicitly scope the efficiency claim to RFE and the compared baselines.","section":"5.2.2, Figures 3 and 6; Appendix C.3"},{"comment":"The EWC incremental update assumes that the Fisher information computed from the previous iteration's subspaces, Eq. (10), correctly identifies the parameters that remain important for evaluating the current feature space. The manuscript asserts the informational overlap shown in Figure 1b, but it never tests the assumption under low-overlap conditions, e.g., when consecutive feature spaces share few features or when the optimizer makes a large change. If the overlap assumption fails, the EWC penalty may freeze outdated parameters and the reported efficiency gain would disappear. A controlled experiment varying the degree of overlap, or at least a discussion of when the assumption is violated, is needed to support the mechanism behind the claimed speed-up.","section":"4.3, Eqs. (10)-(11)"}],"minor_comments":[{"comment":"Table 1 lists 'spam_base' twice, and the abstract claims 'fourteen real-world datasets' while the introduction and Section 5.1 say 'twelve' (six classification and six regression tasks). Please reconcile the counts and remove the duplicate row.","section":"Table 1 and Abstract"},{"comment":"The paragraph says 'We apply EASE to two iterative feature selection frameworks' but then enumerates four methods: RFE, FLSR, SDAE, and GRFG. Please correct the count or restructure the sentence.","section":"5.1, Baseline Algorithms"},{"comment":"There are small language errors: 'We Additionally test' should be 'We additionally test' in Section 5.2.4, and 'EASE −F Cby' should be 'EASE −F C by' in Appendix C.6.","section":"5.2.4 and C.6"},{"comment":"The y-axis label 'Cumulative Time' has no units; Table 7 reports seconds, so please state the unit in the figures for reproducibility.","section":"Figures 3 and 6"},{"comment":"The sentence 'For classification, EASE can improve by approximately 3% compared to other baselines' is vague; please specify whether this is an average accuracy gain, over which baselines, and for which metric.","section":"5.2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim is underpowered by current evidence; the lack of significance testing is the main risk. The Eq. (3) issue is a genuine derivation gap that should be fixed. The dataset count inconsistency and duplicate row in Table 1 suggest the manuscript was not carefully proofread. I do not see circularity as a major problem because the final evaluation uses an external RF model, and self-citations to FLSR and GRFG are used as test beds rather than as the source of the improvement. The manuscript fits the scope of cs.LG."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: EASE is a real architectural contribution as a plug-in feature-space evaluator — feature-sample subspace decoupling plus a weight-shared multi-head attention evaluator updated with EWC — and the authors ship code and data. But the paper's central empirical claim, that EASE outperforms LR/DT/GBDT/RF/XGB in most cases, is not actually supported by the tables: with 10 runs and no significance tests, most comparisons sit within one standard deviation. That's the thing to know before citing this.\n\nTo its credit, the method is clearly described, the pseudo-code is in the appendix, and the final evaluation uses an external RF, so the gains aren't forced by the method's own equations. The ablation study makes sense, and the sensitivity analysis on h and D is a nice touch.\n\nThe soft spots are in the evidence, not the mechanism. First, Eq. (3) defines feature importance as M(F) − M(F\\{fi}), a difference of evaluator outputs, not a difference in performance/loss. That's not a standard importance score and it reads like a derivational slip; it needs clarification or correction. Second, no statistical tests. Examples: Table 6 svmguide3 has RF at 0.861±0.012 and EASE at 0.856±0.014 — the 'best' is numerically worse. Table 5 wine_white F1 differences are fractions of a std. A paired test or CI on differences would settle it. Third, the efficiency claim is narrower than advertised: Figure 3/6 compare EASE only with GBDT, RF, XGB in RFE, not LR/DT, and not in the FLSR/SDAE/GRFG frameworks. The 'reduces training time' headline is overgeneralized. Fourth, the numbers don't line up: abstract says 14 datasets, intro says twelve, and Table 1 lists 15 rows with spam_base twice; the '6 classification' text is also wrong. Fifth, values of k, q, s, λ aren't given in the main text or Appendix C.1 — only learning rate and epochs. That's a reproducibility gap even with code. The EWC similarity assumption is plausible but untested on low-overlap sequences; I'd rank that as a minor concern next to the missing statistics.\n\nWho's this for? People working on AutoML / feature selection who want a drop-in evaluator with a strong architectural idea. It deserves a serious referee: the method is coherent, code exists, and the flaws are addressable. I'd send it to peer review and ask for paired significance tests, a fix to Eq. (3), and fuller hyperparameter disclosure. If those come back, the paper could be a solid contribution. As it stands, I wouldn't cite it as evidence of a performance gain.","headline":"A coherent plug-in evaluator with a novel architecture, but the headline performance gain isn't statistically supported and Eq. (3) needs fixing.","tokens_in":23825,"tokens_out":3145,"would_cite":false,"duration_ms":26383,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EASE, a plug-in evaluator for iterative feature-space search, claims better selected features and lower cumulative training time than five baseline evaluators on fourteen datasets.","keywords":["feature space evaluation","iterative feature selection","incremental learning","elastic weight consolidation","multi-head attention","feature-sample subspace construction","automated feature optimization"],"falsifier":"Run EASE and its retrain-from-scratch ablation on a dataset where consecutive feature-space candidates are made nearly disjoint, for example by using random restarts or reverse-order feature rankings, and compare cumulative training time and final downstream performance; if the incremental version no longer beats retraining from scratch, the overlap assumption is the load-bearing part. A second check is to compute the Fisher information on the first iteration's subspaces, then apply the same penalty to randomly selected parameters, which would test whether the penalty is specifically preserving task-relevant knowledge.","tokens_in":22763,"feed_emoji":"⚙️","tokens_out":6272,"duration_ms":52550,"temperature":0.7,"pith_summary":"This paper proposes EASE, a plug-in evaluator for iterative feature-space optimization—the loop in which an algorithm repeatedly refines which features to use until a downstream predictor performs well. Its claim is that standard evaluators used inside such loops treat all samples alike, can overfit a chosen downstream model, and are retrained from scratch at every iteration. EASE addresses these problems by splitting the current feature space into feature-sample subspaces that emphasize the most task-relevant features and the hardest-to-evaluate samples, scoring the subspaces with a weight-sharing multi-head attention network, and updating that network incrementally with an elastic-weight-consolidation penalty instead of retraining. On fourteen datasets and inside four iterative feature-selection and feature-generation frameworks, the paper reports that EASE both improves the downstream performance of the resulting feature space and lowers cumulative optimization time relative to five baseline evaluators. If the claim holds, feature-space search—a common bottleneck in applied machine learning—could become faster and more model-agnostic.","feed_headline":"Plug-in evaluator makes feature selection faster and better","feed_subtitle":"A shared-attention scorer with incremental updates cuts retraining cost and lifts downstream accuracy across 14 datasets.","key_machinery":"The load-bearing machinery is the Feature-Sample Subspace Generator together with the Contextual Attention Evaluator: one identifies which features matter and which samples are hardest, and the other encodes each subspace through multi-head attention with shared weights before producing a prediction. The Fisher information computed from the previous iteration's subspaces is what converts overlap between consecutive feature spaces into a training-time saving, via an elastic weight consolidation penalty that restrains changes to parameters judged important; this is what lets the evaluator be updated incrementally instead of retrained from scratch.","core_discovery":"EASE treats the evaluator itself as the thing to be optimized, not just the feature subset. Each iteration, a feature index optimizer ranks features by how much removing them changes the evaluator's loss, and a sample index optimizer samples instances proportionally to their previous prediction error; repeated draws over the selected indices form fixed-length feature subspaces. A contextual attention evaluator with weights shared across subspaces maps each subspace to a prediction, and an elastic weight consolidation term—a Fisher-information-weighted quadratic penalty on parameter movement from the previous iteration—lets the evaluator retain old knowledge while absorbing new information. The paper claims this combination makes the evaluator fairer, more generalizable, and cheaper, and reports that it outperforms linear/logistic regression, decision trees, GBDT, random forests, and XGBoost across the RFE, FLSR, SDAE, and GRFG frameworks and across classification and regression metrics.","pith_inferences":["An untested extension is to apply the same incremental-update design to other iterative loops that repeatedly re-score a moving object, such as neural architecture search or prompt optimization, where consecutive candidates share structure.","A testable prediction is that the efficiency gain degrades as consecutive feature spaces become less overlapping; varying the step size of feature removals between iterations would reveal how much of the gain comes from the overlap assumption.","A cleaner isolating experiment would compare EASE against a same-architecture attention evaluator retrained from scratch with the same subspace construction; the EASE-IT ablation moves in this direction but does not fully separate architecture gains from incremental-update gains."],"forward_implications":["Swapping EASE into an existing iterative feature-selection loop should improve the downstream performance of the selected feature space, with reported classification gains around 3% and consistent regression gains on the test datasets.","Cumulative optimization time should drop substantially relative to tree-based and linear evaluators, with reported savings sometimes exceeding 100 seconds per dataset, because the evaluator no longer needs full retraining at each iteration.","The feature space selected by EASE transfers across downstream predictors, beating the original feature space when evaluated with LR, DT, GBDT, and RF.","EASE operates as a drop-in evaluator across RFE, FLSR, SDAE, and GRFG, so the benefit is not tied to a single search algorithm."],"supporting_citations":[{"why":"Defines Recursive Feature Elimination, the main iterative feature-selection framework into which EASE is plugged for the headline comparisons.","marker":"[Guyon et al., 2002]"},{"why":"Supplies elastic weight consolidation, the incremental-update mechanism whose Fisher-information penalty is used in Eq. (11).","marker":"[Kirkpatrick et al., 2017]"},{"why":"Supplies the multi-head attention backbone that the Contextual Attention Evaluator adapts with weight sharing.","marker":"[Vaswani, 2017]"},{"why":"Provides FLSR, one of the iterative feature-selection frameworks used to show EASE transfers beyond RFE.","marker":"[Zhao et al., 2020]"},{"why":"Provides SDAE, the unsupervised feature-selection framework used in generalization experiments.","marker":"[Hassanieh and Chehade, 2024]"},{"why":"Provides GRFG, the feature-generation framework where EASE is tested as an evaluator.","marker":"[Wang et al., 2022]"},{"why":"Underpins the Fisher-information computation used to estimate parameter importance for the incremental update.","marker":"[Grosse and Martens, 2016]"}],"fun_headline_variants":["EASE: incremental evaluator trims retraining cost","Adaptive feature evaluator learns without full retraining","Feature optimization with an evaluator that gets smarter","No more scratch retraining: EASE updates itself","Incremental evaluator keeps old knowledge, adds new"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All of the reported speed and quality gains rest on the assumption that consecutive feature spaces in the optimization loop share enough information for the Fisher-information penalty to identify parameters worth preserving; if the feature space jumps too much between iterations, the penalty could freeze outdated knowledge and the efficiency gain would vanish.","fun_headline_variants_meta":{"raw":{"variants":["EASE: incremental evaluator trims retraining cost","Adaptive feature evaluator learns without full retraining","Feature optimization with an evaluator that gets smarter","No more scratch retraining: EASE updates itself","Incremental evaluator keeps old knowledge, adds new"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1507,"prompt_tokens":996,"completion_tokens":511,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":434}},"tokens_in":612,"tokens_out":511,"duration_ms":5296,"temperature":1.0,"reasoning_tokens":434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:49:26.037131+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EASE and its retrain-from-scratch ablation on a dataset where consecutive feature-space candidates are made nearly disjoint, for example by using random restarts or reverse-order feature rankings, and compare cumulative training time and final downstream performance; if the incremental version no longer beats retraining from scratch, the overlap assumption is the load-bearing part. A second check is to compute the Fisher information on the first iteration's subspaces, then apply the same penalty to randomly selected parameters, which would test whether the penalty is specifically preserving task-relevant knowledge.","supporting_citations":[{"cited_title":"Gene selection for cancer classification using support vector machines","cited_arxiv_id":null,"evidence_quote":"Defines Recursive Feature Elimination, the main iterative feature-selection framework into which EASE is plugged for the headline comparisons."},{"cited_title":"Simplifying reinforced feature selection via restructured choice strategy of single agent","cited_arxiv_id":null,"evidence_quote":"Provides FLSR, one of the iterative feature-selection frameworks used to show EASE transfers beyond RFE."},{"cited_title":"Selective deep autoencoder for unsupervised feature selection","cited_arxiv_id":null,"evidence_quote":"Provides SDAE, the unsupervised feature-selection framework used in generalization experiments."},{"cited_title":"Group-wise reinforcement feature generation for optimal and explainable representation space reconstruction","cited_arxiv_id":null,"evidence_quote":"Provides GRFG, the feature-generation framework where EASE is tested as an evaluator."}],"review_version":1}