Pith. sign in

REVIEW 4 major objections 5 minor 53 references

Optimised Feature Subset Selection via Simulated Annealing

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SA-FDR treats feature selection as global combinatorial optimisation, using the Fisher discriminant ratio as a proxy, and in benchmarks returns sparser logistic models than RFE or Lasso with equal or better AUC

desk verdict A clearly explained wrapper method that pairs simulated annealing with an FDR proxy; the idea is fine, but the evaluation leaks test-set statistics and lacks error bars, so the central accuracy claim isn't yet supported. read the letter →

arxiv 2507.23568 v1 pith:6S4KALKY submitted 2025-07-31 cs.LG cond-mat.stat-mechstat.ML

Fernando Martínez-García , Álvaro Rubio-García , Samuel Fernández-Lorenzo , Juan José García-Ripoll , Diego Porras This is my paper · ORCID
classification cs.LGcond-mat.stat-mechstat.ML
keywords featuresubsetselectionsimulatedannealingFisherdiscriminantratiologisticregressionℓ0-normsparsemodelscombinatorialoptimisationwrappermethods
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that feature subset selection for binary logistic classification can be treated as a combinatorial optimisation problem and solved with simulated annealing, using the Fisher discriminant ratio as a cheap stand-in for model quality during the search. On eight public tabular datasets ranging from a few hundred to about 100,000 samples, the resulting algorithm, SA-FDR, selects smaller feature subsets than recursive feature elimination or Lasso while matching or slightly improving their predictive accuracy, measured by area under the ROC curve. If true, this gives practitioners a wrapper method that finds sparse, interpretable logistic models in settings where greedy methods discard useful features early. The paper also argues the same annealing framework would work with other quality metrics and classifiers, not just logistic regression and the Fisher discriminant ratio.

What carries the argument

The load-bearing object is the Fisher discriminant ratio as a proxy for logistic-regression quality. For a candidate subset the FDR is $\mathrm{FDR} = (\mu_1-\mu_0)^T S_W^{-1}(\mu_1-\mu_0)$, with $S_W=\Sigma_0+\Sigma_1$, and the annealing minimises $-\mathrm{FDR}$ using the Metropolis acceptance rule $P=\min\left(1,e^{\beta[\mathrm{FDR}(s')-\mathrm{FDR}(s)]}\right)$ over $R$ replicas, with inverse temperatures raised until FDR converges. The proxy lets the search skip expensive logistic fits at every proposal; cross-entropy is evaluated only on the final replicas to pick the best subset. Batch matrices computed from different data batches add noise that the authors say helps avoid overfitting and keeps replicas from collapsing to the same local maximum.

What would settle it

Enumerate all feature subsets of size $k$ on a small dataset, compute the FDR and the logistic cross-entropy for each, and check whether subsets with near-maximal FDR also have near-minimal cross-entropy; if the correlation is weak, the annealing search will systematically miss the best logistic subsets on that data.

Watch

Extended reading notes

Core claim

The paper's central claim is that SA-FDR reliably finds the sparsest logistic model with competitive prediction accuracy: in its benchmarks it returns the smallest optimal feature count $k^*$ in most datasets and the best test AUC on several of them, with near-best AUC on the rest. The search is structured so that simulated annealing explores feature subsets of each size $k$ under the objective $-\mathrm{FDR}$, and at the end of annealing the replica whose logistic regression has the lowest cross-entropy is selected, which corrects for mild mismatches between FDR and true logistic quality. The authors attribute the sparsity advantage to the non-greedy exploration of the subset space: combinations that a greedy elimination procedure would discard early can be retained and tested.

Load-bearing premise

The load-bearing premise is that feature subsets scoring high on the Fisher discriminant ratio are also the subsets that give the logistic model low cross-entropy; if that correlation breaks down, the annealing search will miss the best subsets.

Editorial extensions

If this is right

  • On tabular binary-classification problems of the size tested, users can obtain logistic models with fewer features than RFE or Lasso yield, with accuracy that is at least competitive.
  • Because the objective is decoupled from the classifier, the same annealing procedure can be applied to other model-quality metrics and to non-linear classifiers such as neural networks.
  • The runtime, although larger than the baselines, is practical and can be reduced by tuning the number of replicas and sweeps without necessarily losing solution quality.
  • Sparser selected subsets make the resulting logistic models easier to interpret and less prone to overfitting in high-dimensional applications.
  • SA-FDR's advantage over RFE grows when useful features are correlated or redundant in ways that greedy backward elimination cannot recover.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct consequence the paper does not quantify is that the FDR proxy should be most reliable when each class is roughly unimodal and elliptically distributed; on multimodal or skewed features, high-FDR subsets may not coincide with low-cross-entropy subsets, and the final replica-selection step can only correct this within the small set of annealed candidates.
  • The method's dependence on $R$ replicas suggests that richer sampling schemes, such as replica exchange or parallel tempering, could improve the chance of having at least one replica near the true cross-entropy optimum; the paper notes such annealing refinements as future work but does not test them.
  • A testable extension would be to run SA-FDR against an exact $\ell_0$-optimal search on small datasets ($K \lesssim 20$) to measure how often the FDR-optimal subset is also the cross-entropy-optimal subset, quantifying the proxy gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SA-FDR, a simulated-annealing wrapper for ℓ0-norm feature selection in binary logistic models. The objective minimized during annealing is the negative Fisher discriminant ratio (FDR) of selected features, computed on randomly chosen batch matrices, with the final subset chosen among replicas by logistic cross-entropy. A cross-validation routine selects k* as the smallest subset size whose validation AUC is within one standard deviation of the best, and the final model is evaluated on a held-out test set. SA-FDR is compared with RFE and Lasso on nine UCI/public datasets including one with roughly 100,000 samples. The paper reports that SA-FDR finds sparser feature subsets with competitive or better AUC.

Significance. SA-FDR is a plausible combination of known components, and the FDR proxy is a computationally cheap objective that could make simulated annealing practical for wrapper-style feature selection. The paper evaluates on a diverse benchmark, including a large Loan Default dataset, and directly compares with standard RFE and Lasso baselines. The central empirical claim is currently undermined by test-set leakage in the preprocessing pipeline and by the absence of uncertainty quantification. If the evaluation is corrected and the results are reproduced, the method could still be a useful contribution, but the evidence as presented is not sufficient to support the abstract's claim of consistently high predictive accuracy.

major comments (4)
  1. [Section 4, data preprocessing] The sentence 'we also calculate the mean and standard deviation of each feature from the test dataset and use these values to standardise the entire dataset' describes using test-set statistics to transform the data before model fitting. This leaks test information into training and validation, so all AUC values and k* estimates in Table 1 and all curves in Fig. 2 are computed under a protocol in which every algorithm has access to test-set information. This invalidates the quantitative support for the claim of 'high predictive accuracy' and for the reported sparsity-accuracy trade-off. The pipeline should be rerun with a scaler fitted only on the training portion of each fold and then applied without refitting to the validation and test portions.
  2. [Table 1] The paper reports average k* and AUC over 'up to 20' repetitions but gives no standard deviations, confidence intervals, or significance tests. Several reported differences between algorithms are very small (e.g., Cancer AUC 0.9922 versus 0.9933; Card Default AUC 0.7214 versus 0.7220; Student k* 10.05 versus 13.88 with AUC 0.9162 versus 0.9170). Without uncertainty quantification, the claims that SA-FDR 'consistently' selects more compact subsets and 'achieves the best AUC value for some of these datasets' are not quantitatively supported.
  3. [Section 2.2 and Fig. 1] The claim that FDR 'correlates well' with logistic-regression cross-entropy is supported only by one qualitative density plot for SPECTF Heart. Since the annealing optimizes -FDR for essentially all iterations and cross-entropy is used only to select among final replicas, a divergence between FDR and cross-entropy could directly degrade the quality of the selected subsets. The authors should provide a quantitative correlation analysis over sampled subsets (e.g., Spearman or Pearson correlation between FDR and cross-entropy, or between FDR and validation AUC) across multiple datasets and several values of k.
  4. [Section 3.2] The k* heuristic (minimum k within one standard deviation of the maximum mean validation AUC) is acceptable, but the paper does not report the full validation-AUC curves or their standard errors for the additional datasets; only the Loan Default dataset is shown in Fig. 2. Without these curves, it is difficult to assess whether the chosen k* values are stable across folds or whether the one-standard-deviation rule selects very different k* for different algorithms.
minor comments (5)
  1. [Section 4, hyperparameters] The value NS = 0.5 is unusual for a parameter described as the number of sweeps, since Algorithm 1 loops for NS·K swaps. The definition and rationale for this value should be clarified.
  2. [Section 4, Loan Default dataset] The cleaning of the Loan Default dataset is described only as removing missing values and highly correlated features; the exact thresholds and procedure should be specified for reproducibility.
  3. [Section 3.2] The term 'Receiving Operator Characteristic-Area Under the Curve' should be 'Receiver Operating Characteristic-Area Under the Curve'.
  4. [Algorithm 1] The stopping condition 'if the mean of FDR(s) has converged' is not defined; the convergence criterion should be stated concretely.
  5. [Throughout] No statement about code availability or random seeds is provided, which limits the reproducibility of the exact numbers in Table 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the FDR objective, cross-entropy replica selection, and external benchmarks keep the reported AUC independent of the optimization inputs; the test-set standardization leak is a correctness issue, not a circularity.

full rationale

SA-FDR's reported results do not reduce to its inputs by construction. The search objective FDR is defined independently in Section 2.2 from class means and within-class scatter matrices, and it is not fitted to the AUC values reported in Table 1. The final replica choice is made by explicitly minimizing cross-entropy (Section 3.1, Algorithm 1 lines 15-16), an independent criterion, and the optimal feature count k* is selected by validation AUC in Section 3.2. Thus the later test AUC is intended as an independent measurement. The self-citations ([26]-[28]) are used only to motivate simulated annealing generally in the introduction; they are not load-bearing for the benchmark results and do not exclude alternative methods. The closest thing to a circularity concern is the asserted correlation between FDR and logistic cross-entropy, which is supported only by the qualitative density plot in Figure 1; that is a weak-evidence or correctness risk, not a definitional identity. I also explicitly flag a separate, non-circular methodological flaw: Section 4 states 'we also calculate the mean and standard deviation of each feature from the test dataset and use these values to standardise the entire dataset.' This leaks test-set statistics into the training pipeline and undermines the reported AUC and k* values as independent estimates. That is a data-leakage bug that should be fixed, but it is not circularity: no equation in the paper defines AUC or k* in terms of those test statistics, and the leakage does not make the prediction identical to an input. Overall circularity score: 0.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

No new physical or model entities are introduced. The 'batch matrix' randomization is a procedural detail, not a postulated entity. The central claim depends on the proxy assumption and hyperparameter choices listed above.

free parameters (5)
  • Replica count R = 50
    Chosen by hand in Section 4; no sensitivity analysis or evidence that this value is optimal.
  • Sweeps per temperature NS = 0.5
    Chosen by hand; the pseudocode treats NS as a multiplier on K swaps per temperature step, and the paper does not justify 0.5.
  • Inverse temperature step epsilon = 0.7
    Chosen by hand in Section 4; the temperature schedule affects exploration and final solution quality.
  • Temperature steps beta_steps = 100
    Chosen by hand; runs stop earlier if the mean FDR converges, but the convergence threshold is not specified.
  • Maximum subset size kmax = 30 (150 for Loan Default)
    Chosen as 'enough to find k*'; no justification that larger kmax would not improve results.
assumptions (6)
  • domain assumption FDR correlates with logistic-regression cross-entropy for the datasets studied
    Section 2.2 claims 'correlates well with the cross entropy' and shows one density plot (Fig. 1) for the Heart dataset; no quantitative correlation is reported.
  • domain assumption The fixed cooling schedule reaches near-optimal feature subsets
    Section 3.1 invokes the asymptotic guarantee of Ref. [37] but uses a heuristic linear schedule with early stopping; convergence is not certified.
  • ad hoc to paper Using randomly selected batch matrices during optimization adds beneficial noise without harming solution quality
    Section 3.1 introduces this mechanism 'to keep a baseline noise that avoids overfitting' without experimental evidence that it helps.
  • domain assumption The k* selection rule, smallest k with validation AUC within one standard deviation of the maximum, is a good model-complexity choice
    Section 3.2 defines this heuristic; it is plausible but arbitrary and affects all compared algorithms equally.
  • domain assumption Within-class covariance matrices S_W are invertible for all evaluated subsets
    Eq. (8) requires S_W^{-1}; no regularization is described, which is unsafe for collinear or high-dimensional subsets.
  • standard math BFGS converges to the logistic-regression optimum for every fitted model
    Section 2.1 states BFGS is used; for logistic regression this is standard and reliable for well-conditioned problems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimised Feature Subset Selection via Simulated Annealing." pith.science (2026). https://pith.science/paper/6S4KALKY

@misc{pith2026250723568,
  author       = {Pith},
  title        = {Pith review of: Optimised Feature Subset Selection via Simulated Annealing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6S4KALKY}},
  note         = {Machine review of arXiv:2507.23568}
}
abstract

We introduce SA-FDR, a novel algorithm for $\ell_0$-norm feature selection that considers this task as a combinatorial optimisation problem and solves it by using simulated annealing to perform a global search over the space of feature subsets. The optimisation is guided by the Fisher discriminant ratio, which we use as a computationally efficient proxy for model quality in classification tasks. Our experiments, conducted on datasets with up to hundreds of thousands of samples and hundreds of features, demonstrate that SA-FDR consistently selects more compact feature subsets while achieving a high predictive accuracy. This ability to recover informative yet minimal sets of features stems from its capacity to capture inter-feature dependencies often missed by greedy optimisation approaches. As a result, SA-FDR provides a flexible and effective solution for designing interpretable models in high-dimensional settings, particularly when model sparsity, interpretability, and performance are crucial.

Figures

Figures reproduced from arXiv: 2507.23568 by the authors.

Figure 1
Figure 1. Left: Example of the Fisher Linear Discriminant method to find a line that optimally [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Results of the cross-entropy (on train dataset) and AUC scores (on train, validation, [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 51 canonical work pages

  1. [1]

    Hastie T 2009 The elements of statistical learning: data mining, inference, and prediction

    work page 2009

  2. [2]

    2013 Applied predictive modeling vol 26 (Springer)

    Kuhn M, Johnson K et al. 2013 Applied predictive modeling vol 26 (Springer)

    work page 2013

  3. [3]

    Guyon I and Elisseeff A 2003 Journal of machine learning research 3 1157–1182

    work page 2003

  4. [4]

    Guyon I, Gunn S, Nikravesh M and Zadeh L A 2008 Feature extraction: foundations and applications vol 207 (Springer)

    work page 2008

  5. [5]

    Chandrashekar G and Sahin F 2014 Computers & electrical engineering 40 16–28

    work page 2014

  6. [6]

    Guyon I, Weston J, Barnhill S and Vapnik V 2002 Machine learning 46 389–422

    work page 2002

  7. [7]

    Saeys Y, Inza I and Larranaga P 2007 Bioinformatics 23 2507–2517

    work page 2007

  8. [8]

    Forman G 2003 Journal of machine learning research 3 1289–1305

    work page 2003

Show all 53 references
  1. [9]

    Wright J, Yang A Y, Ganesh A, Sastry S S and Ma Y 2008 IEEE transactions on pattern analysis and machine intelligence 31 210–227

    2008

  2. [10]

    Tsay R S 2005 Analysis of financial time series (John wiley & sons)

    2005

  3. [11]

    Luo R C and Kay M G 2002 IEEE Transactions on Systems, Man, and Cybernetics 19 901–931

    2002

  4. [12]

    Yang J, Triendl H, Soltan A A, Prakash M and Clifton D A 2024 BMC Medical Informatics and Decision Making 24 183

    2024

  5. [13]

    Bertsimas D, King A and Mazumder R 2016 The annals of statistics 813–852

    2016

  6. [14]

    Sato T, Takano Y, Miyashiro R and Yoshise A 2016 Computational Optimization and Applications 64 865–880

    2016

  7. [15]

    Bertsimas D, Pauphilet J and Van Parys B 2021 Machine Learning 110 3177–3209

    2021

  8. [16]

    Varma S and Simon R 2006 BMC bioinformatics 7 1–8

    2006

  9. [17]

    Cawley G C and Talbot N L 2010 The Journal of Machine Learning Research 11 2079–2107

    2010

  10. [18]

    Liu H and Motoda H 2007 Computational methods of feature selection (CRC press)

    2007

  11. [19]

    Cai J, Luo J, Wang S and Yang S 2018 Neurocomputing 300 70–79

    2018

  12. [20]

    Moslemi A 2023 Engineering Applications of Artificial Intelligence 126 107136

    2023

  13. [21]

    Edwards S F and Anderson P W 1975 Journal of Physics F: Metal Physics 5 965

    1975

  14. [22]

    Barahona F 1982 Journal of Physics A: Mathematical and General 15 3241

    1982

  15. [23]

    M´ ezard M, Parisi G and Virasoro M A 1987Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications vol 9 (World Scientific Publishing Company)

  16. [24]

    Kirkpatrick S, Gelatt C D and Vecchi M P 1983 Science 220 671–680 publisher: American Association for the Advancement of Science

    1983

  17. [25]

    Wang W, Machta J and Katzgraber H G 2015 Physical Review E 92 013303

    2015

  18. [26]

    Mart ´ ınez-Garc ´ ıa F and Porras D 2025arXiv preprint arXiv:2501.07638

    arXiv

  19. [27]

    Rubio-Garc ´ ıa´A, Garc ´ ıa-Ripoll J J and Porras D 2022arXiv preprint arXiv:2210.00807

    arXiv

  20. [28]

    Rubio-Garc ´ ıa´A, Fern´ andez-Lorenzo S, Garc ´ ıa-Ripoll J J and Porras D 2024Physica A: Statistical Mechanics and its Applications 639 129637

  21. [29]

    Hosmer Jr D W, Lemeshow S and Sturdivant R X 2013 Applied logistic regression (John Wiley & Sons)

    2013

  22. [30]

    Fisher R A 1936 Annals of eugenics 7 179–188

    1936

  23. [31]

    Theodoridis S 2015 Machine learning: a Bayesian and optimization perspective (Academic press)

    2015

  24. [32]

    Nocedal J and Wright S J 1999 Numerical optimization (Springer)

    1999

  25. [33]

    Fletcher R 2000 Practical methods of optimization (John Wiley & Sons) Optimised Feature Subset Selection via Simulated Annealing 12

    2000

  26. [34]

    Parlett B N 1998 The symmetric eigenvalue problem (SIAM)

    1998

  27. [35]

    Horn R A and Johnson C R 2012 Matrix analysis (Cambridge university press)

    2012

  28. [36]

    Robert C P, Casella G and Casella G 1999 Monte Carlo statistical methods vol 2 (Springer)

    1999

  29. [37]

    Mitra D, Romeo F and Sangiovanni-Vincentelli A 1986 Advances in applied probability 18 747–771

    1986

  30. [38]

    2013 An introduction to statistical learning vol 112 (Springer)

    James G, Witten D, Hastie T, Tibshirani R et al. 2013 An introduction to statistical learning vol 112 (Springer)

    2013

  31. [39]

    Fawcett T 2006 Pattern recognition letters 27 861–874

    2006

  32. [40]

    Tibshirani R 1996 Journal of the Royal Statistical Society Series B: Statistical Methodology 58 267–288

    1996

  33. [41]

    Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M and Duchesnay E 2011 Journal of Machine Learning Research 12 2825–2830

    2011

  34. [42]

    Kelly M, Longjohn R and Nottingham K 2023 The UCI Machine Learning Repository

    2023

  35. [43]

    Cios K, Kurgan L and Goodenday L 2001 SPECTF Heart [Dataset] UCI Machine Learning Repository

    2001

  36. [44]

    Wolberg W, Mangasarian O, Street N and Street W 1993 Breast Cancer Wisconsin (Diagnostic) [Dataset] UCI Machine Learning Repository

    1993

  37. [45]

    Zhang K, Fan W and Yuan X 2008 Ozone Level Detection [Dataset] UCI Machine Learning Repository

    2008

  38. [46]

    Hopkins M, Reeber E, Forman G and Suermondt J 1999 Spambase [Dataset] UCI Machine Learning Repository

    1999

  39. [47]

    Realinho V, Vieira Martins M, Machado J and Baptista L 2021 Predict Students’ Dropout and Academic Success [Dataset] UCI Machine Learning Repository

    2021

  40. [48]

    2020 Taiwanese Bankruptcy Prediction [Dataset] UCI Machine Learning Repository

    2020

  41. [49]

    Guyon I 2004 Madelon [Dataset] UCI Machine Learning Repository

    2004

  42. [50]

    Yeh I 2009 Default of Credit Card Clients [Dataset] UCI Machine Learning Repository

    2009

  43. [51]

    Yasser H M 2022 Loan Default [Dataset]

    2022

  44. [52]

    Miasojedow B, Moulines E and Vihola M 2013 Journal of Computational and Graphical Statistics 22 649–664

    2013

  45. [53]

    Barash L Y, Weigel M, Borovsk` y M, Janke W and Shchur L N 2017 Computer Physics Communications 220 341–350

    2017

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.