{"id":"d8d6a09f-9084-409d-b49b-c084f083cba0","arxiv_id":"2601.05151","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"A benchmarking package and a PIONeeR case study conclude that a BH-adjusted p-value filter beats LASSO on stability, predictive AUC, and discovery rate for lung-cancer immunotherapy-resistance biomarkers.","lead":"The authors built ROOFS, a Python package that benchmarks many feature-selection methods on a user's dataset and reports stability, predictive performance, and discovery error. Applied to a lung-cancer immunotherapy trial, it found a simple adjusted-p-value filter performed better than LASSO and similar machine-learning methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semi-synthetic ground truth is built from BH-adjusted p-values on the same data, and p.adjust also uses BH-adjusted p-values; its TPR advantage may be a benchmark artifact.","rationale":"The reader's weakest_assumption identifies the semi-synthetic ground truth as not independent of the winning method, which I agree is the most load-bearing concern. The central claim of optimality hinges on the TPR metric: without the 0.93 TPR, p.adjust's AUC and stability advantages are marginal and statistically indistinguishable given the absence of uncertainty intervals. The construction in Section 2.1.3 directly favors any BH-adjusted p-value filter, and p.adjust is exactly such a filter. My proposed check — using a randomly selected true feature set — would settle whether the TPR advantage is a design artifact or a genuine property. I do not see a more fundamental objection to the package's utility or the general stability/performance observations; the concern is specifically about the 'optimal method' claim. Since the reader already issued a CONDITIONAL verdict, my analysis does not move the verdict, hence UNCHANGED. I agree with the reader's assessment; the only additional nuance is that the varying-subset-size comparison (Section 2.2.5) compounds the circularity, but the core concern is the same.","tokens_in":21011,"tokens_out":5634,"duration_ms":58342,"concrete_test":"Re-run the semi-synthetic benchmark on the post-VIF PIONeeR data with 19 true predictors sampled uniformly at random from the 214 features (or selected by a mechanism independent of univariate p-values, e.g., by clustering or random partitioning). Generate the outcome with the same logistic model, estimating coefficients from the selected features as in Section 2.1.3. Compute TPR/FPR for p.adjust and all other methods. If p.adjust's TPR drops from 0.93 to roughly the 0.7 level seen for t-score and Fisher, the reported discovery advantage is a benchmark artifact. Additionally, report bootstrap confidence intervals for the optimism-corrected AUC values in Table 2 to assess whether the 0.72 vs 0.70 differences are meaningful.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — that p.adjust (union of BH-adjusted t-test and logistic p-values) is the optimal FS method on PIONeeR — rests heavily on its semi-synthetic TPR of 0.93±0.06, far above all other methods (e.g., t-score 0.71, Fisher 0.69, LASSO 0.59). This advantage is suspect because of how the ground truth was constructed. In Section 2.1.3, the 19 'true' predictors were selected from the same PIONeeR data using a 0.01 threshold on BH-adjusted p-values from a Cox model. The winning p.adjust method is also a BH-adjusted p-value filter. Thus the true features are by construction features that any univariate p-value-based filter is likely to recover; the benchmark is circular with respect to the method it crowns. The paper mitigates this only indirectly: Section 2.2.5 acknowledges that fixed-size methods (t-score, Fisher) are capped at 19 features, whereas variable-size p.adjust may select many more (up to 72 in bootstrap), inflating its TPR. But it never addresses the deeper circularity: the ground-truth feature set is itself a p-value-selected set. If the true predictors had been chosen independently of univariate screening — e.g., uniformly at random, or via a different mechanism — p.adjust's TPR would likely drop to the level of other statistical filters, removing its only decisive advantage. Without this TPR edge, p.adjust's AUC (0.72) and stability (0.39) are only marginally better than several other methods (AUC 0.70, stability 0.36–0.38) with no reported confidence intervals, so the 'optimal' designation would not be justified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ROOFS, a Python package that benchmarks multiple feature-selection (FS) methods on a user-provided dataset, reporting optimism-corrected predictive performance, stability, and true/false positive rates from a semi-synthetic outcome simulation. The authors apply ROOFS to the PIONeeR NSCLC immunotherapy dataset (n=435, p=374, reduced to p=214 after VIF filtering), comparing 23 FS methods combined with 11 classifiers. They report that a filter based on the union of Benjamini-Hochberg-adjusted p-values from t-test and logistic regression ('p.adjust') achieves the best optimism-corrected AUC (0.72), stability (S=0.39), and semi-synthetic TPR (0.93±0.06), outperforming LASSO and other embedded/wrapper methods. The paper argues that simple statistical filters can be more stable and comparably predictive than popular ML-based embedded methods in n≈p biomedical data.","tokens_in":21461,"tokens_out":5720,"duration_ms":59860,"significance":"The ROOFS package is a useful open-source contribution: it integrates a broad set of FS methods, automatic reporting, stability assessment, optimism correction, and semi-synthetic discovery evaluation, and the PIONeeR case study addresses a realistic heterogeneous n≈p clinical dataset. If the empirical claims survive scrutiny, the work supports a practical recommendation that simple statistical filters deserve a place in the standard toolkit. However, the central claim that p.adjust is the optimal method depends heavily on a semi-synthetic ground truth that is constructed from the same type of p-value screening as the recommended method; this circularity must be resolved before the empirical conclusions can be accepted.","major_comments":[{"comment":"The semi-synthetic 'true' predictor set S is defined as the 19 features with BH-adjusted p-values <0.01 from a multivariable Cox model fitted to the same PIONeeR data, and the simulated outcome is generated by a linear logistic function of these features. The method later recommended (§3.5) is itself a BH-based p-value filter (union of t-test and logistic regression at 0.05). Consequently, the TPR=0.93 reported in Table 2 is largely a self-consistency check: a p-value filter will, by construction, re-discover features selected by p-value thresholding on the same design matrix. The discussion in §2.2.5 acknowledges the difference in subset size between fixed- and variable-size methods but does not address this circularity. Please re-run the semi-synthetic benchmark with an independent construction of the ground truth (e.g., features selected at random, or from external biological criteria","section":"2.1.3, Table 2"},{"comment":"Even apart from the semi-synthetic issue, the real-outcome evidence for declaring p.adjust 'optimal' is thin: AUC 0.72 vs 0.70 for Fisher score and t-score, and stability 0.39 vs 0.36–0.37. No confidence intervals or pairwise significance tests are provided for these differences, so the superiority could be within bootstrap variability. Since §3.5 explicitly states that the choice was driven by predictive performance, stability, and discovery capability (TPR), and the TPR evidence is affected by the issue above, please supply uncertainty estimates (e.g., bootstrap CIs for AUC and stability, paired tests across classifiers) and adjust the strength of the conclusion accordingly.","section":"3.5, Table 2"},{"comment":"The TPR/FPR comparison is not adjusted for the number of features selected. p.adjust selects on average many more features than the fixed-size methods (up to 72 in bootstrap, vs k=19; see §3.5 and Figure 5C), which mechanically increases TPR and FPR. The paper notes this in §2.2.5 but still treats high TPR as evidence of discovery power. To support a 'best balance' claim, present a size-controlled analysis (e.g., TPR and FPR as functions of the selection threshold, or precision-recall curves) so that methods are compared at comparable signature sizes.","section":"2.2.5, Fig. 4"}],"minor_comments":[{"comment":"The annotation 'Corresponds to better than LASSO' appears incomplete; the table does not clearly show which cells are being compared. Also, the 'Full data' row is formatted differently from the other rows and should be clarified.","section":"Table 2"},{"comment":"The text states that p.adjust FPR is significantly higher than random FS (0.1 vs 0.07), but Table 2 lists two random FS rows: fixed-size (FPR 0.09) and random-size (FPR 0.06). Please specify which random FS baseline is used in the comparison.","section":"3.4, Table 2"},{"comment":"The p.adjust filter is described as 'controlling for user-specified covariates; PD-L1 in the case of PIONeeR data', but the statistical model is not fully specified. State whether PD-L1 was included as a covariate in a univariate logistic regression for each feature, and whether this adjustment was applied consistently in every bootstrap fit.","section":"2.2.1 / Supplementary Table S1"},{"comment":"The phrase 'low to zero additional programming' is unclear: users still need to prepare data and configure the pipeline. Consider rewording to describe the actual user effort required.","section":"Abstract / Section 1"}],"recommendation":"major_revision","confidential_remarks":"The package contribution is solid and likely to be of interest to applied and clinical ML audiences. The main risk is the circular semi-synthetic evaluation behind the headline TPR claim. If the authors can show that the TPR advantage of p.adjust persists under an independent ground-truth construction, the paper's central recommendation would be much stronger. I recommend requiring this re-analysis before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe main thing to know: ROOFS is a real, usable package for benchmarking feature selection methods, and that part is worth attention. The paper's central empirical claim—that a simple p.adjust filter beats LASSO and others on PIONeeR—is undercut by the way the semi-synthetic ground truth was built. The authors selected the 19 \"true\" predictors using BH-adjusted p-values from a Cox model on the same data (Section 2.1.3), then crowned a method that also uses BH-adjusted p-values. So the high TPR (0.93) is partly baked in. That doesn't kill the paper, but it should substantially soften the \"p.adjust is optimal\" conclusion until they redo the experiment with an independent ground truth.\n\nWhat's actually new: ROOFS integrates optimism-corrected performance, Nogueira stability, and semi-synthetic TPR/FPR into one pipeline with a report, on data with mixed types, n≈p. That's a useful tool for applied researchers. The package is public on GitLab with tutorial data, so the software component is verifiable and reproducible. The empirical finding that simple statistical filters can be more stable and comparable in AUC to embedded methods is consistent with prior benchmarks and is a useful sanity check.\n\nWhere it's soft: the circularity above is the main issue, and it's not minor. Also, the AUC differences between p.adjust (0.72) and several other methods (0.70) are small, with no confidence intervals reported. The stability differences are also modest (0.39 vs 0.36–0.38). So even on the real-outcome comparison, the \"optimal\" designation rests on thin margins. The FPR of p.adjust is higher than random selection (0.10 ± 0.03 vs 0.07 ± 0.03), which is an odd result for a method claiming FDR control. The authors attempt an explanation via residual correlations, but it complicates the story. Also, the evaluation priorities (high TPR over low FPR) are stated only after results are shown, which feels post hoc. The paper does explicitly acknowledge that fixed-size methods like t-score are capped at 19 features, which is fair, but it never addresses the deeper circularity that the ground truth itself is a p-value-selected set.\n\nThe paper is clear and honest about many limitations, and the package is likely to be used. It deserves a serious referee, but the empirical claims need substantial revision.\n\nRecommendation: send to peer review, but ask for an independent or randomly chosen true feature set in the semi-synthetic benchmark, uncertainty on AUC, and pre-specified metric weights. For me, the package is the contribution, not the PIONeeR result.","headline":"Genuine software contribution, but the headline empirical claim is undercut by a circular semi-synthetic design.","tokens_in":21984,"tokens_out":3347,"would_cite":true,"duration_ms":32834,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that, on the PIONeeR lung-cancer dataset (435 patients, 214 features after multicollinearity reduction), a statistical filter combining Benjamini-Hochberg-adjusted p-values from a t-test and logistic regression selects the","keywords":["feature selection","biomarker discovery","stability","optimism correction","semi-synthetic benchmark","Benjamini-Hochberg","LASSO","immunotherapy resistance"],"falsifier":"Re-run the semi-synthetic experiment with the 19 true predictors chosen by an independent rule (randomly selected features, features chosen by LASSO on the real outcome, or a nonlinear generative model with interactions) while keeping all other benchmark settings fixed. If p.adjust's true-positive rate no longer exceeds that of Fisher score or t-score, or if a different method wins, the original ranking is an artifact of the ground-truth construction.","tokens_in":20886,"feed_emoji":"📊","tokens_out":4794,"duration_ms":51258,"temperature":0.7,"pith_summary":"The paper introduces ROOFS, a Python package that benchmarks many feature-selection (FS) methods on a user's dataset and reports stability, predictive performance with optimism correction, and true/false discovery rates from a semi-synthetic outcome. Applying it to PIONeeR clinical trial data on resistance to anti-PD-(L)1 immunotherapy, the authors compare 23 FS methods combined with 11 classifiers. They find that a simple filter—the union of Benjamini-Hochberg-adjusted p-values from a t-test and logistic regression at a 0.05 threshold—achieves the best balance of optimism-corrected AUC (0.72), stability (0.39), and true-positive rate (0.93) among all methods, including widely used LASSO (AUC 0.69, stability 0.34). If correct, this means simple statistical filters can be more stable and at least as predictive as popular embedded machine-learning methods in the common biomedical setting where the number of predictors is close to the number of patients, and that benchmarking FS methods before choosing one is worth the computational cost.","feed_headline":"p-value filter beats LASSO in biomarker benchmark","feed_subtitle":"Benchmarking 23 feature-selection methods on PIONeeR trial data finds a simple filter most stable and predictive.","key_machinery":"The central object is the ROOFS benchmarking pipeline, which evaluates FS methods via bootstrap resampling (B = 100): identical preprocessing, FS, and classification are applied to each resample. Predictive performance is estimated with the .632+ optimism-correction method; stability is measured with Nogueira's frequency-based index; and discovery power is assessed on a semi-synthetic dataset where a simulated outcome is generated from 19 pre-selected real features, allowing true-positive and false-positive rates to be computed over 40 noise replicates. The winning method itself is a simple filter: it takes the union of features selected by BH-adjusted p-values (0.05 threshold) from a t-test","core_discovery":"On the post-VIF PIONeeR dataset, the p.adjust filter—the union of features with Benjamini-Hochberg-adjusted p-values below 0.05 from a two-sample t-test and from logistic regression adjusted for PD-L1—was identified as the optimal FS method. It achieved the highest optimism-corrected AUC (0.72) when paired with a gradient boosting classifier, a stability of S = 0.39 by Nogueira's measure, and on the semi-synthetic benchmark a true-positive rate of 0.93 ± 0.06 (with a false-positive rate of 0.10 ± 0.03). This outperformed the LASSO baseline (AUC 0.69, S = 0.34) and all other tested methods, including resampling-based ensembles designed to improve stability (Bolasso, stability selection, Stabl","pith_inferences":["The semi-synthetic ground truth in this benchmark was constructed by selecting 19 'true' predictors using a 0.01 threshold on Benjamini-Hochberg-adjusted p-values from a Cox model fit to the same PIONeeR data—essentially the same family of p-value-based selection logic as the winning p.adjust filter. If this construction predisposes the simulation toward linear, p-value-detectable signals, the TPR","A direct extension would be to rerun the ROOFS benchmark with the synthetic outcome generated from nonlinear or interaction terms (the paper states ROOFS supports polynomial and pairwise-interaction terms), which would test whether p.adjust's high TPR persists when the true model is not linear in the features.","The paper's report of a successful application to the separate SChISM study (nine-variable fragmentomics signature, AUC 0.89) suggests the pipeline is transferable; a systematic cross-dataset comparison could establish which FS method families are consistently competitive, beyond this single PIONeeR example.","The per-feature selection frequencies and robustness measures reported by ROOFS may turn out to be more practically valuable than the single 'optimal' method, because they let clinicians see which biomarkers are selected consistently across many methods and bootstraps; this hypothesis could be tested by externally validating the high-frequency features on independent immunotherapy cohorts."],"forward_implications":["If the central claim holds, applied researchers in n≈p biomedical settings can safely start with simple statistical filters, which are computationally cheap, stable, and predictive, rather than defaulting to LASSO or other embedded methods.","The finding that resampling-based ensembles (Bolasso, stability selection, Stabl, RENT) did not beat plain LASSO in either stability or AUC suggests their added complexity may not pay off in this type of data.","The p.adjust signature of 22 features achieved the same optimism-corrected AUC as the full 214-feature model (0.72 vs 0.71), indicating that a parsimonious signature can match full-data prediction in this setting.","The high true-positive rate (0.93) but also high false-positive rate (0.10) for p.adjust implies a discovery-versus-error trade-off: for hypothesis generation, tolerating false positives may be acceptable, but for strict biomarker panels other methods may be preferable.","Because stability and AUC are only weakly correlated (Spearman r = 0.32), the paper implies that evaluating both is necessary before choosing an FS method, otherwise a stable but weak method (e.g., CIFE) or an unstable but strong one could be mistakenly preferred."],"fun_headline_variants":["Simple p-value filter beats LASSO in biomarker selection","p.adjust filter outperforms LASSO in benchmark","Benchmark crowns p-value filter over LASSO for biomarkers","PIONeeR data: p-value filter wins over LASSO in FS benchmark","Why a simple filter beats LASSO for biomarker discovery"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's simulated 'true' predictors were themselves chosen by Benjamini-Hochberg-adjusted p-values from a Cox model on the same dataset—the same family of procedure as the winning p.adjust filter—so if that ground-truth choice is not independent, the claim that p.adjust is the best discovery method on this data is not established.","fun_headline_variants_meta":{"raw":{"variants":["Simple p-value filter beats LASSO in biomarker selection","p.adjust filter outperforms LASSO in benchmark","Benchmark crowns p-value filter over LASSO for biomarkers","PIONeeR data: p-value filter wins over LASSO in FS benchmark","Why a simple filter beats LASSO for biomarker discovery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1178,"prompt_tokens":881,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":214}},"tokens_in":625,"tokens_out":297,"duration_ms":3513,"temperature":1.0,"reasoning_tokens":214,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T11:45:31.269680+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the semi-synthetic experiment with the 19 true predictors chosen by an independent rule (randomly selected features, features chosen by LASSO on the real outcome, or a nonlinear generative model with interactions) while keeping all other benchmark settings fixed. If p.adjust's true-positive rate no longer exceeds that of Fisher score or t-score, or if a different method wins, the original ranking is an artifact of the ground-truth construction.","supporting_citations":[],"review_version":1}