{"id":"131fd94e-224b-42e8-9785-74d968b79176","arxiv_id":"2608.12057","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new label-free evaluation framework ranks feature selection methods by the optimal transport distance between the selected feature subset and the first principal components of the full data.","lead":"This paper argues that common 'unsupervised' evaluations of feature selection secretly use class labels, and offers a label-free alternative that compares selected features with a PCA reference using optimal transport. It gives practitioners a way to score feature selectors when true labels are unavailable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The framework's evaluation of feature-selection quality rests on the unvalidated premise that the top-PCA subspace is the right reference; a synthetic low-variance-signal test would settle whether the metric tracks task-relevant features.","rationale":"The reader identified the load-bearing premise as the PCA reference; my analysis agrees and makes it more precise. The framework defines quality as closeness to PCA, so the central claim 'evaluates the quality of feature selection algorithms' is only meaningful if PCA preservation tracks downstream-relevant information. The paper's own experiments undermine this: two of the four OT variants rank selectors in opposition to supervised performance, so the framework does not even give a stable answer independent of the OT-chosen. This is an internal inconsistency, not a mere disagreement with consensus. The paper is honest about limitations and frames the work as a proof of concept, which supports a conditional rather than a reject verdict. The proposed synthetic test is a minimal falsification check: if the metric ranks variance-preserving noise features above genuinely discriminative low-variance features, then the PCA reference fails in a basic setting and the headline claim would need substantial revision.","tokens_in":10880,"tokens_out":7531,"duration_ms":71203,"concrete_test":"Build a synthetic dataset with d features: two features contain the class separation (e.g., two well-separated Gaussian clusters in a 2D plane), and the remaining d-2 features are independent high-variance Gaussian noise common to both classes. Ensure the top two principal components align with the noise directions (e.g., set noise variance >> signal variance). Run the proposed metric (each OT variant) on the subset selected by a variance-based selector (which will pick the two noise features) and by an oracle selector that picks the two discriminative features. If the noise-feature subset is ranked as closer to PCA than the discriminative subset, the framework does not evaluate task-relevant feature-selection quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the framework provides a truly unsupervised evaluation of feature selection quality. The word 'quality' is given a specific operational meaning in Section 3: a selector is better if its selected-feature representation is closer, in optimal transport distance, to the representation obtained from the first f principal components. This is not a neutral quality measure; it is a definition of quality as preservation of the PCA subspace. For the claim 'evaluates the quality of feature selection algorithms' to hold beyond this tautology, closeness to PCA must be correlated with what practitioners mean by feature selection quality, i.e., retaining information useful for downstream tasks. The paper does not establish this. It offers correlations with supervised metrics, but the results are ambiguous: in Figure 4, OT_EMD2 and OT_SINKHORN2 anti-correlate with ACC/AUC, while OT_GW2 correlates positively. If PCA were a generally valid reference, all four OT instantiations would point in the same direction; instead the framework's output depends on the arbitrary choice of OT metric. Section 5 concedes that PCA 'imposes a structural limitation' but does not test whether the limitation is benign for the evaluated datasets. Because the reference is untested, a feature selector that discards the genuinely informative low-variance directions but preserves high-variance noise can be ranked highly by this framework.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that common 'unsupervised' evaluations of feature selection are not truly unsupervised because they use ground-truth labels in downstream clustering metrics such as CLSACC and NMI. It then proposes a label-free framework: for each feature selector that outputs f features, compute an optimal transport distance between the data restricted to those f features and the data projected onto the first f principal components; convert distances to similarities via Eq. (7), and rank selectors by similarity. Experiments on eight high-dimensional datasets compare four OT variants (EMD, Sinkhorn, Gromov-Wasserstein, Sliced Wasserstein) against supervised (ACC, AUC), pseudo-unsupervised (CLSACC, NMI), and model-agnostic (AAD) metrics, reporting Pearson/Spearman correlations and a ranking plot for Isolet. The paper concludes that the framework is a 'truly unsupervised' evaluation approach and that the results are a proof of concept.","tokens_in":11083,"tokens_out":6361,"duration_ms":54730,"significance":"If the framework's central claim holds, it would fill a genuine gap: a label-free, model-agnostic way to rank feature selectors, which is important in domains where ground-truth labels are unavailable. The method has clear strengths: it uses no labels and no fitted parameters; the OT distance is well-defined; and the paper provides a critical discussion of why standard 'unsupervised' metrics such as CLSACC and NMI are not label-free. The authors are also unusually candid in Section 5 about PCA's 'structural limitation' and the proof-of-concept nature of the experiments. However, the validity of the metric is not yet established, because the PCA-as-reference assumption is untested and the empirical support is mixed: two of the four OT variants invert the supervised ranking, aggregate correlations are modest, and no uncertainty quantification is provided. The paper is a promising first step but is substantively incomplete as a demonstration of 'truly unsupervised evaluation of feature selection.'","major_comments":[{"comment":"The load-bearing premise of the framework is that the top-f PCA representation is a valid reference for feature-selection quality. Section 3 states that 'the objective is to identify feature selection algorithms that select features in a way that they are as close as possible to the representation obtained from PCA,' and Section 5 acknowledges that 'the use of PCA as the reference mechanism imposes a structural limitation.' The manuscript never tests whether closeness to PCA tracks task-relevant information. In particular, a synthetic dataset with a low-variance feature that carries the label signal and high-variance noise features would directly test this: a valid feature-selection quality metric should favor the selector that finds the low-variance feature, while the proposed framework would favor selectors matching PCA's high-variance directions. Without such a test, the central claim that the framework 'evaluates the quality of feature selection algorithms' (Section 3) is not distinguishable from the tautology that it measures agreement with a PCA-based reference.","section":"Section 3 (objective) and Section 5 (Limitations)"},{"comment":"The evidence for predictive validity is mixed and does not support the conclusion that 'the proposed metrics showed consistent correlations with established evaluation metrics.' In Figure 4, OT_EMD2 and OT_SINKHORN2 have Pearson correlations with ACC of -0.02 and -0.03, respectively, and with AUC of 0.07 and 0.06, whereas OT_GW2 and OT_SLICED_SW show correlations of 0.29-0.37 with the supervised block. Because the four instantiations differ only in the OT distance measure, the framework's output is not a single quality measure but a family of measures whose agreement with downstream performance depends on an arbitrary design choice. Moreover, Eq. (7) transforms distances via 1/x; for Pearson correlation this nonlinear transformation changes the coefficient values, so the reported Pearson coefficients are not intrinsic to the OT distance (Spearman, being rank-based, is less affected). The authors should either justify a preferred OT variant a priori, report all results per variant with confidence intervals, or substantially weaken the validity claims.","section":"Section 4.2, Figure 4, and Section 5 (Conclusions)"},{"comment":"The empirical basis is too thin for the strength of the central claim: eight datasets, five feature selectors, and no confidence intervals, significance tests, or per-dataset breakdowns. The paper itself describes the results as 'proof of concept' in Section 5, but the abstract and Section 3 present the framework as the solution to 'truly unsupervised evaluation.' The aggregate correlations in Figure 4 could be driven by a small number of datasets, and specific inversions (e.g., Figure 3 shows OT_SLICED_SW ranking Random and SCFS at the top for Isolet) suggest substantial variability. To support the claim that the framework ranks selectors in a way that is useful for practitioners, the authors should report per-dataset and per-feature-fraction correlations, add bootstrap confidence intervals or permutation tests for the aggregate correlations, and ideally include more datasets and selectors. Without this, the evidence for the framework's validity is anecdotal.","section":"Section 4.1 and Section 4.2 (experimental setup and results)"}],"minor_comments":[{"comment":"The text says 'the eigenvector corresponding to the first principle components'; this should be 'the eigenvector corresponding to the first principal component,' correcting the spelling and the singular/plural mismatch.","section":"Section 2.2 (AAD description)"},{"comment":"Figure 2's caption states 'For all measures, higher is better,' but the text notes that OT_SLICED_SW's resolution degrades; showing rank plots like Figure 3 for all OT variants, not only OT_SLICED_SW, would make the comparison clearer.","section":"Figures 2 and 3"},{"comment":"The heatmaps lack labels for the row and column metric names, which makes the correlation matrix difficult to read; adding axis labels or a legend would improve accessibility.","section":"Figure 4"},{"comment":"The phrase 'per-budget cost matrix normalization' is not defined; please specify the normalization formula or cite the FSEVAL implementation so that the reported OT scores are reproducible.","section":"Section 4.2"},{"comment":"Reference [31] ('Worse than random') is an arXiv preprint by the same group and is central to the Random baseline; if it is under review, this should be stated, and the relevant code should be made available so readers can reproduce the baseline.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript cites a large number of arXiv preprints by the same group (e.g., refs. 31, 33, 34, 35, 36, 38). Some of these are directly load-bearing, such as the Random baseline in ref. 31; if those references are not peer-reviewed, the authors should provide published versions or make the relevant code available. In addition, the claim in Section 5 that 'only one truly unsupervised and model-agnostic metric currently exists' is strong and should be verified against the broader literature on internal cluster validity and feature-selection stability."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core critique is correct and worth stating plainly: if you evaluate feature selection by clustering on selected features and then check against ground-truth labels, that is not unsupervised evaluation. The proposed alternative—score a selector by optimal transport distance between the selected-feature subspace and the top-f PCA representation—is clean, simple, and genuinely label-free. That specific combination is new; the existing AAD metric from the same group only compares the angle of the first PC eigenvector, not a distributional distance. The paper also runs a random baseline, which is good practice and often missing in this literature.\n\nThe framework has no circularity problem: labels are used only as a benchmark after the metric is defined, and the PCA reference is a modeling choice, not an output of the metric. The experiments use eight standard high-dimensional datasets and four OT variants, and the limitations section is candid about computational cost, the PCA dimensionality bound, and the small empirical scope. As a proof of concept, it is a legitimate contribution.\n\nThe soft spots are concentrated where the framework is structurally weakest. The reference is PCA, and the paper defines quality as closeness to PCA. That definition can be defended, but the paper treats it as if it were validated by correlations with supervised metrics. If the informative structure in a dataset lives in a low-variance direction, a selector that discards it can still match PCA and rank highly. A synthetic experiment with a known low-variance signal would settle whether the metric tracks task-relevant features; the paper does not run one. The stress-test note is right about this.\n\nSecond, the choice of OT metric matters a lot. OT_EMD2 and OT_SINKHORN2 invert the supervised ranking, while OT_GW2 and OT_SLICED_SW correlate positively. The conclusion says the metrics showed \"consistent correlations\" with established metrics, which is not what Figure 4 shows. This is not a minor mismatch; it means the ranking a practitioner gets depends on which OT variant they pick, and the paper does not explain why. Minor issues: only eight datasets, no confidence intervals or significance tests, and no released code, which makes the results hard to build on.\n\nThe right audience is researchers working on unsupervised feature selection evaluation and anyone building high-dimensional pipelines without labels. It deserves a serious referee and a revision that adds the synthetic low-variance-signal test, a systematic comparison of the OT variants, error bars, and code release. I would not desk-reject it.","headline":"A clean, label-free feature-selection evaluation idea, honestly framed as a proof of concept; the PCA reference premise and the instability across OT variants are the load-bearing soft spots.","tokens_in":11624,"tokens_out":2284,"would_cite":false,"duration_ms":22848,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a truly unsupervised evaluation of feature selection, ranking selectors by the optimal-transport distance between the selected features and the data's top principal components.","keywords":["unsupervised evaluation","feature selection","optimal transport","principal component analysis","inter-dataset similarity","label-free evaluation","subspace similarity","model-agnostic metric"],"falsifier":"Construct a synthetic dataset in which a class label is determined entirely by a single low-variance feature, while all high-variance features are noise. Run the framework with $f=1$ on three selectors: one that picks the low-variance informative feature, one that picks the highest-variance feature, and one random. If the framework ranks the random or high-variance selector above the informative one while the supervised accuracy ranks them oppositely, the claim that closeness to PCA measures feature-selection quality fails for this case.","tokens_in":10661,"feed_emoji":"📊","tokens_out":14586,"duration_ms":115230,"temperature":0.7,"pith_summary":"The paper claims that the standard way of evaluating unsupervised feature selection—cluster the data on the selected features and compare the clusters with ground-truth labels—is not actually unsupervised, because the labels are used in the scoring. It proposes a truly label-free alternative: score a selection algorithm by how close the data restricted to its selected features is, under an optimal-transport distance, to the data represented by its first principal components. If correct, this gives practitioners a way to rank feature selectors on unlabeled data, without any ground-truth information, which matters for the many real-world datasets that arrive without labels. The paper reports experiments on eight high-dimensional datasets in which some choices of optimal-transport distance produce rankings that track classification-based performance, while others invert it.","feed_headline":"Label-free scoring ranks feature selectors by distance to PCA","feed_subtitle":"No ground-truth labels needed: the score is the optimal-transport cost to the top principal components.","key_machinery":"The central machinery is the pairing of an unsupervised reference representation with an inter-dataset similarity measure. The reference is Principal Component Analysis: the first $f$ principal components of the full data form the target representation that a good feature subset of size $f$ is expected to resemble. The similarity measure is an optimal-transport distance, instantiated as Earth Mover's Distance, entropic-regularized Sinkhorn, Gromov-Wasserstein, or Sliced Wasserstein, all of which quantify the cost of transforming one distribution into another. The framework's scoring step is the reciprocal transform that turns these distances into similarities, and its ranking step sorts feature-selection algorithms by those similarities. This machinery is what lets the evaluation run without any label information.","core_discovery":"The paper's central claim is that feature-selection evaluation can be made completely label-free by comparing the selected feature subspace against a PCA reference. The framework computes, for a fixed number $f$ of features, the optimal-transport distance between the dataset restricted to the $f$ features chosen by a candidate algorithm and the dataset projected onto the first $f$ principal components, then converts the distance into a similarity score (the reciprocal, with infinity meaning identical). The ranking of algorithms follows from these similarities: the more the selected subspace resembles the PCA subspace, the higher it ranks. The paper argues this is “truly unsupervised” because no ground-truth labels enter the scoring at any point, and it uses this framework to show that the widely used clustering-accuracy and NMI evaluations are in fact supervised because they require the true class labels. The paper also finds that the particular optimal-transport measure chosen—Earth Mover's, Sinkhorn, Gromov-Wasserstein, or Sliced Wasserstein—materially changes the resulting rankings.","pith_inferences":["A natural extension would be to correct the similarity scores for chance by comparing them with the distribution of scores for random feature subsets, which would make rankings comparable across datasets and budgets.","If the informative structure for a downstream task lives in low-variance directions, the framework could rank a variance-preserving selector above one that keeps the truly informative features, a failure mode testable on a synthetic dataset with a low-variance class boundary.","The finding that some optimal-transport measures invert the supervised rankings suggests that each distance encodes a distinct notion of “quality”; the choice should be matched to the intended use of the selected features rather than treated as interchangeable.","Because Gromov-Wasserstein can compare distributions of different dimensionality, the framework could compare a selected subset directly against the full-dimensional data, removing the need to fix the number of reference components."],"forward_implications":["The paper's analysis shows that the common “unsupervised” evaluation metrics clustering accuracy and normalized mutual information actually use ground-truth labels and are therefore supervised evaluations run on an unsupervised downstream task.","The proposed framework ranks feature selectors using only the data and a specified number of principal components, so it can be applied to datasets where no labels exist.","The choice of optimal-transport distance substantially changes the rankings; in the reported experiments, Gromov-Wasserstein and Sliced Wasserstein correlate positively with label-based performance, while Earth Mover's and Sinkhorn correlate negatively.","The paper recommends comparing rankings rather than raw optimal-transport scores across different numbers of selected features, because per-budget cost normalization can distort the values.","The framework is modular and could accept other unsupervised reference representations besides PCA, which the paper identifies as a direction for future work."],"supporting_citations":[{"why":"Defines the inter-dataset similarity concept and the only existing model-agnostic label-free evaluation metric that the paper compares against.","marker":"[34]"},{"why":"Supplies the dynamic evaluation philosophy and the fixed-percentage schedule used to sample feature budgets.","marker":"[32]"},{"why":"Introduces the random baseline that the proposed metrics must distinguish from genuine selection algorithms.","marker":"[31]"},{"why":"Defines the Earth Mover's Distance used as one instantiation of the inter-dataset similarity measure.","marker":"[5]"},{"why":"Defines the entropic-regularized Sinkhorn distance used as the GPU-friendly instantiation.","marker":"[10]"},{"why":"Defines the Gromov-Wasserstein distance, which in the experiments produces rankings most aligned with the supervised block.","marker":"[30]"},{"why":"Defines the Sliced Wasserstein distance used as the scalable lower-resolution instantiation.","marker":"[6]"},{"why":"Provides the computational optimal transport background that justifies the use of these distances as inter-dataset similarities.","marker":"[29]"}],"fun_headline_variants":["Truly unsupervised feature selection scoring via PCA distance","Label-free evaluation: feature selectors ranked by PCA fit","Optimal transport to PCA yields unsupervised feature rankings","Feature selection evaluation exposed as partly supervised","PCA and optimal transport give true unsupervised scoring"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scoring assumes that the top principal components of the data are the right reference for feature-selection quality; if the structure that matters for a later task sits in low-variance directions, a selector matching PCA could be ranked above one that keeps the genuinely informative features.","fun_headline_variants_meta":{"raw":{"variants":["Truly unsupervised feature selection scoring via PCA distance","Label-free evaluation: feature selectors ranked by PCA fit","Optimal transport to PCA yields unsupervised feature rankings","Feature selection evaluation exposed as partly supervised","PCA and optimal transport give true unsupervised scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1377,"prompt_tokens":877,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":430}},"tokens_in":493,"tokens_out":500,"duration_ms":5187,"temperature":1.0,"reasoning_tokens":430,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:17:28.039136+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a synthetic dataset in which a class label is determined entirely by a single low-variance feature, while all high-variance features are noise. Run the framework with $f=1$ on three selectors: one that picks the low-variance informative feature, one that picks the highest-variance feature, and one random. If the framework ranks the random or high-variance selector above the informative one while the supervised accuracy ranks them oppositely, the claim that closeness to PCA measures feature-selection quality fails for this case.","supporting_citations":[{"cited_title":"Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation -- Extended Version","cited_arxiv_id":"2501.09591","evidence_quote":"Defines the inter-dataset similarity concept and the only existing model-agnostic label-free evaluation metric that the paper compares against."},{"cited_title":"In: Similarity Search and Applications","cited_arxiv_id":null,"evidence_quote":"Supplies the dynamic evaluation philosophy and the fixed-percentage schedule used to sample feature budgets."},{"cited_title":"Worse than Random: The Importance of a Baseline for Unsupervised Feature Selection","cited_arxiv_id":"2605.22973","evidence_quote":"Introduces the random baseline that the proposed metrics must distinguish from genuine selection algorithms."},{"cited_title":"ACM Trans","cited_arxiv_id":null,"evidence_quote":"Defines the Earth Mover's Distance used as one instantiation of the inter-dataset similarity measure."},{"cited_title":"In: Advances in Neural Information Processing Systems 26","cited_arxiv_id":null,"evidence_quote":"Defines the entropic-regularized Sinkhorn distance used as the GPU-friendly instantiation."},{"cited_title":"In: Proceedings of The 33rd International Conference on Ma- chine Learning","cited_arxiv_id":null,"evidence_quote":"Defines the Gromov-Wasserstein distance, which in the experiments produces rankings most aligned with the supervised block."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Sliced Wasserstein distance used as the scalable lower-resolution instantiation."}],"review_version":1}