{"id":"dbec3960-fe84-46e8-85fa-555cfce814c4","arxiv_id":"2508.02625","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AutoML-Med uses Latin Hypercube Sampling and Partial Rank Correlation Coefficient to automatically find preprocessing and model configurations that improve balanced accuracy and sensitivity on imbalanced medical tabular datasets.","lead":"AutoML-Med is an automated machine learning tool that searches over data preprocessing and model choices to predict disease risk in medical tabular data. In tests on multiple sclerosis and diabetes data, it reported higher balanced accuracy and sensitivity than several comparison tools, though the evaluation leaves important open questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never states that the final one-third test split is excluded from pipeline selection and PRCC fine-tuning, so the reported balanced-accuracy/sensitivity gains may be selection-on-test artifacts.","rationale":"A good-faith reading of the paper shows a plausible AutoML framework aimed at medical tabular data, and the reported magnitudes of improvement are not absurd. The MS comparison uses real AutoML baselines, and the BRFSS comparison, while methodologically weaker because the reference models were not optimized for the same objective, still illustrates a sensitivity/balanced-accuracy trade-off. The most load-bearing issue is protocol transparency: §3 never defines a validation split for pipeline selection. The sentence in §3.2 about cross-validation is scoped to hyperparameters of the predictive model, not to the LHS-PRCC search or the grid-search fine-tuning in §3.3. If the final one-third split is used in selection, the reported metrics are optimistically biased and the comparison is invalid. Ten-run averaging does not fix selection-on-test bias. This is an underspecification rather than demonstrated misconduct, and it is addressable by releasing code or by re-running with a locked test split. The reader's weakest assumption identified exactly this issue, so agreement is 'agree.' The verdict should remain CONDITIONAL: the claim is plausible but not yet verifiable, and the requested clarification or code release would settle it. I also considered the PRCC-on-categorical-variables concern (arbitrary integer coding of preprocessing methods makes PRCC rankings order-dependent), but the central empirical claim does not strictly depend on PRCC working as intended, so test-set separation is the more decisive concern.","tokens_in":10121,"tokens_out":7634,"duration_ms":87100,"concrete_test":"Obtain the withheld code (or a minimal reproduction of the BRFSS experiment) and instrument it so that the 1/3 test split is loaded only after phases (a)–(e) finish; then re-run all LHS/PRCC/grid-search pipeline decisions using nested cross-validation on the training portion alone. Compare the nested-CV balanced accuracy and sensitivity against Table 2. If the clean-protocol numbers are within one standard deviation of the reported values (e.g., balanced accuracy 0.7436), the concern is resolved; if they drop materially (e.g., by more than 0.02–0.03), the reported advantage is leakage-driven.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central numerical claim depends on the 1/3 test split being used only for final evaluation, but the paper never explicitly says this. In §3.2, cross-validation is mentioned only for hyperparameter search: 'to reduce the risk of overfitting, a random cross-validation search is performed for the best hyperparameters for the predictive model being trained.' Pipeline selection, PRCC ranking, and grid-search fine-tuning are instead described as acting on 'the target metric(s)' with no validation split defined. §3.3 then selects the final pipeline by 'the highest value on the target metric(s)' and presents it 'along with a complete report of its performance.' If those target metrics are computed on the same one-third split later reported in Tables 1 and 2, the balanced accuracy and sensitivity are optimistically biased by construction, and the comparisons to Auto-sklearn, GAMA, AutoBalance, and Xie et al. [22] are invalid. Averaging over 10 seeds does not remove selection-on-test bias. The conclusion states the code will be shared only after review, so the withheld code cannot currently be inspected to rule this out.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AutoML-Med, an AutoML framework for medical tabular data that searches over five ordered preprocessing stages (imputation, class balancing, feature engineering, scaling, and feature selection) together with a predictive model. The search uses Latin Hypercube Sampling to generate pipeline candidates, trains models on each preprocessed dataset, and then uses the Partial Rank Correlation Coefficient to identify the most influential preprocessing steps for a final grid-search refinement. The experimental sections report results on a proprietary Multiple Sclerosis cohort (Table 1) and the public BRFSS 2014 diabetes dataset (Table 2), claiming higher balanced accuracy and sensitivity than Auto-sklearn, GAMA, AutoBalance, and the published models of Xie et al.","tokens_in":10374,"tokens_out":5649,"duration_ms":57035,"significance":"The problem addressed is real, and the general architecture is plausible: the focus on preprocessing steps, class imbalance, and missing values is appropriate for clinical tabular data. The choice of balanced accuracy and sensitivity as primary metrics is well justified for triage applications. The evaluation on a public dataset is a commendable step toward reproducibility. However, the current manuscript does not yet establish the central performance claim. The description of pipeline selection and fine-tuning never states that the final test split is excluded from selection; the diabetes comparison is against numbers extracted from an earlier paper under different data handling; no significance tests are provided; and the proposed PRCC mechanism is applied to categorical preprocessing choices without methodological justification. The withheld code further prevents independent verification. If the design is corrected and the comparisons tightened, the tool could be a useful contribution, but the evaluation as written is not conclusive.","major_comments":[{"comment":"The most load-bearing ambiguity is which data split is used to compute the target metrics that drive pipeline selection and PRCC fine-tuning. Section 3.2 states only that 'a random cross-validation search is performed for the best hyperparameters for the predictive model being trained'; it does not say that the preprocessed pipelines themselves are evaluated by cross-validation or on the training partition. Section 3.3 then selects 'the pipeline that obtains the highest value on the target metric(s)' and presents it 'along with a complete report of its performance.' If those target metrics are computed on the same one-third split reported in Tables 1 and 2, the reported balanced accuracy and sensitivity are optimistically biased by selection-on-test, and the comparisons to Auto-sklearn, GAMA, AutoBalance, and Xie et al. would be invalid. Averaging over ten seeds does not remove selection-on-test bias. The manuscript must state explicitly that the test split is used only once at the end, and if that was not the case, the experiments must be re-run with a properly held-out test set.","section":"§3.2, §3.3, §4.1, §4.2"},{"comment":"The Type 2 Diabetes comparison is not a controlled benchmark. The authors state that they applied the filtering procedure of Xie et al. [22] but did not remove patients with missing data, and they compare to the published numbers of [22] rather than re-running the same models on the same train/test splits under the same preprocessing. Differences in missing-data handling, feature encoding, class imbalance treatment, and evaluation protocol can fully explain the reported gap. At minimum, the authors should re-run the models of [22] in identical conditions and report the metrics on the same test partitions; otherwise the claim of 'significantly higher sensitivity and balanced accuracy' is not supported.","section":"§4.2, Table 2"},{"comment":"PRCC is applied to preprocessing steps that are categorical choices. The independent variables are the discretized integer indices returned by LHS, whose ordering is arbitrary. A rank-based partial correlation on these indices is not a well-defined sensitivity measure for unordered categorical choices; the result depends on the arbitrary ordering of methods within each step. The manuscript must clarify how the categorical steps are encoded for PRCC (e.g., one-hot dummy variables) and why PRCC is appropriate, or replace it with a categorical sensitivity analysis. This issue is central because PRCC is the mechanism that selects which steps are refined in the final grid search.","section":"§3.3"},{"comment":"No statistical significance tests are reported, and for several metrics the claimed advantage is small. In Table 1, AutoML-Med's F1-macro (0.8745) and MCC (0.7558) are within one standard deviation of the competing tools (e.g., GAMA MCC 0.7683), and the AUC is actually lower than all three baselines. The paper should therefore restrict its claims to the metrics where the difference is meaningful, or provide paired significance tests across the ten seeds.","section":"Tables 1 and 2"},{"comment":"The comparison with Auto-sklearn, GAMA, and AutoBalance does not specify the search budgets, time limits, or hyperparameter configurations given to the competing tools. AutoML results depend strongly on the number of pipeline evaluations and the allowed runtime. Without this information, the balanced-accuracy difference could be an artifact of unequal computational effort. Please report the compute budgets and ideally run the baselines under matched budgets.","section":"§4.1"}],"minor_comments":[{"comment":"The abstract contains a typo: 'a heterogeneous feature types' should read 'heterogeneous feature types'.","section":"Abstract"},{"comment":"Table 1 mixes comma and period decimal separators (e.g., '0,8735' vs '0.9187'); use consistent formatting.","section":"Table 1"},{"comment":"The manuscript claims 'compliance with double-blind review' but lists author names and affiliations in the header; this statement should be removed or the paper should be anonymized.","section":"Title page and Conclusion"},{"comment":"Section 3.2 mentions a set of evaluation metrics but the selection criterion is later referred to as 'the target metric(s)'; clarify how multiple metrics are aggregated when more than one target is selected.","section":"§3.2"},{"comment":"The phrase 'Latin Hypercube Sampling - Partial Rank Correlation Coefficient (LHS-PRCC)' suggests a joint method, but LHS and PRCC are used in distinct phases; the naming could be clarified.","section":"§3.1"},{"comment":"Reference [3] is incomplete ('S. Garcia and altri') and should be replaced with the full author list.","section":"References"},{"comment":"The conclusion states that the code will be shared only after review; this is acceptable for now, but it means the reproducibility claim cannot currently be verified.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a legitimate engineering contribution, but the ambiguity about the evaluation split is the kind of issue that must be resolved before acceptance; also, the comparison to Xie et al. needs to be redone as a matched evaluation. There is no evidence of intentional misconduct, but the current wording makes it impossible to rule out selection-on-test bias."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — AutoML-Med is a reasonable engineering contribution with a genuine new-ish trick: using LHS-PRCC sensitivity analysis to decide which preprocessing steps deserve grid refinement. That's worth knowing about. But the paper's central numbers rest on an unstated validation protocol, and the authors won't share code until after review, so the headline gains over Auto-sklearn, GAMA, and AutoBalance are not yet inspectable.\n\nWhat's actually new: applying LHS-PRCC, a standard sensitivity-analysis tool from computational biology, to AutoML pipeline search is uncommon and sensible. The five-step pipeline is standard, but the PRCC-guided fine-tuning is a real idea. The MS dataset is private, so they've done the right thing by also testing on public BRFSS diabetes data. That shows awareness of reproducibility. The reported improvements are not absurd: balanced accuracy 0.89 vs 0.85 on MS, sensitivity 0.85 vs 0.72–0.75; diabetes balanced accuracy 0.74 vs 0.64–0.67. If those numbers hold under a clean protocol, this is a practically useful tool for clinicians.\n\nSoft spots, in order of severity. First, the paper never explicitly says the 1/3 test split is excluded from pipeline selection, PRCC ranking, and grid refinement. The text in §3.2–3.3 mentions cross-validation only for hyperparameter search, and the final pipeline is selected by the highest value on the target metric(s). If those metrics are computed on the test split, the gains are selection-on-test bias. Averaging over 10 seeds does not fix that. The authors may have done it correctly, but the ambiguity is load-bearing. Second, the BRFSS comparison to Xie et al. is not apples-to-apples: they deliberately keep missing data and impute, while Xie et al. removed missing patients. That alone can shift sensitivity and specificity substantially. They note the difference but do not quantify its effect. Third, there are no significance tests or ablations isolating the PRCC step. Given the reported sensitivity gains, confidence intervals or paired tests would help. Fourth, code is withheld until after review; that is fine for double-blind, but it means the protocol question cannot be checked externally right now.\n\nBottom line: this is a solid candidate for peer review, not a desk reject. The right referee can press for a clear data-separation statement, an ablation of the PRCC step, and a fairness analysis for the diabetes comparison. If those check out, the paper is a decent application contribution. It will not change AutoML theory, but it might actually help in clinical settings. I would not cite it yet, but I would bring it to a reading group and I would send it to review.","headline":"AutoML-Med is a genuinely useful engineering idea—LHS-PRCC for pipeline refinement—but its headline numbers rest on an unstated test-separation protocol and a non-equivalent baseline, so it deserves review rather than acceptance as-is.","tokens_in":10876,"tokens_out":1769,"would_cite":false,"duration_ms":18130,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoML-Med, an automated machine learning tool for medical tabular data, reports higher balanced accuracy and higher sensitivity than competing AutoML tools on two imbalanced clinical datasets.","keywords":["automated machine learning","medical tabular data","class imbalance","missing values","Latin hypercube sampling","partial rank correlation coefficient","sensitivity analysis","preprocessing pipeline"],"falsifier":"Run the tool on the same public diabetes dataset with the test split held in a separate file and the code modified so that PRCC ranking and grid-search fine-tuning can see only training folds, then compare balanced accuracy and sensitivity on the untouched test set with the reported values; if they fall toward the comparison baselines' levels, the claimed advantage comes from test information rather than from the preprocessing search.","tokens_in":9973,"feed_emoji":"🩺","tokens_out":7180,"duration_ms":72510,"temperature":0.7,"pith_summary":"AutoML-Med is a proposed automated machine learning pipeline that targets the data problems common in medical tabular datasets: missing values, class imbalance, many features, and few samples. The paper's central claim is that, by treating preprocessing choices as the main object of search and fine-tuning only the most influential preprocessing steps, AutoML-Med achieves higher balanced accuracy and sensitivity than existing AutoML tools on two clinical prediction tasks. Sensitivity matters because it measures how many at-risk patients are correctly flagged, and the paper argues this trade-off is clinically preferable to high specificity that misses patients. The tool is tested on a proprietary multiple sclerosis progression dataset and on a public type 2 diabetes risk dataset, reporting higher balanced accuracy and sensitivity than previously published benchmarks in both settings.","feed_headline":"AutoML-Med lifts sensitivity and balanced accuracy on clinical data","feed_subtitle":"On two imbalanced medical datasets, the tool finds more at-risk patients than competing AutoML systems.","key_machinery":"The load-bearing mechanism is the LHS-PRCC pair: Latin Hypercube Sampling adapted to a discrete space of preprocessing choices (each of five steps is divided into intervals, a value is sampled per interval, and its integer part selects a method), followed by Partial Rank Correlation Coefficient computed on rank-transformed variables to isolate which preprocessing steps independently drive model performance. Because PRCC removes the influence of other variables by regressing each input on the rest and correlating residuals, it lets the tool focus a final grid search on the most influential preprocessing steps rather than enumerating the full Cartesian product of methods. This converts pipeline search from an exponential enumeration into a stratified sample plus a targeted refinement.","core_discovery":"The core discovery is that in medical AutoML the performance bottleneck often lies in preprocessing configuration rather than model choice, and that a two-stage search can exploit this. AutoML-Med first uses Latin Hypercube Sampling to draw a stratified sample of complete five-step preprocessing pipelines (imputation, class balancing, feature engineering, scaling, feature selection), trains candidate models on each, and records evaluation metrics. It then locks in the best-performing model and applies a Partial Rank Correlation Coefficient (PRCC) sensitivity analysis to identify which preprocessing steps most influence the target metric; a grid search over variations of those steps, with the other steps held fixed, produces the final pipeline. On the multiple sclerosis task this yields balanced accuracy 0.8896 and sensitivity 0.8539, against 0.8466-0.8492 and 0.7158-0.7492 for the comparison AutoML tools; on the type 2 diabetes task it yields balanced accuracy 0.7436 and sensitivity 0.7968, against 0.6398-0.665 and 0.3781-0.4634 for the published models, at the cost of lower specificity in both cases.","pith_inferences":["One implicit consequence is that the LHS-PRCC machinery is not specific to medicine; it could be applied to any tabular AutoML setting with a small, structured pipeline space, such as credit scoring or industrial fault detection, and might help there too.","The reported sensitivity gains on the diabetes task come with specificity around 0.69, meaning roughly 31% of healthy patients would be flagged; a deployment study should weigh the cost of follow-up testing against the value of catching additional cases.","A natural testable extension is to run AutoML-Med with the PRCC fine-tuning stage disabled; if most of the sensitivity gain disappears, the claimed advantage comes from the sensitivity-analysis mechanism rather than from the initial stratified sampling.","The paper averages metrics over ten runs with different seeds; a sharper test would report per-run variability of the selected pipelines, to show whether the same preprocessing configurations are consistently chosen or whether different configurations achieve similar scores."],"forward_implications":["On the two tested datasets, the tool's higher sensitivity means more true positives among at-risk patients, at the price of more false positives; the paper argues this is the clinically preferable trade for risk screening.","If preprocessing configuration is the main lever, then medical AutoML users can expect meaningful gains from automating imputation, balancing, and feature-selection choices even when standard predictive models are used.","The approach transfers across medical domains: the same pipeline design produced higher balanced accuracy on both a proprietary neurological dataset and a public behavioral-risk diabetes dataset, suggesting the preprocessing search generalizes.","Because the tool requires no advanced programming skills, clinicians could run it directly, lowering the barrier to adopting machine learning in hospital workflows."],"supporting_citations":[{"why":"Supports the premise that tree-based ensembles remain strong on medium-sized tabular data, motivating the tool's focus on classical models.","marker":"[1]"},{"why":"Supplies one of the AutoML baselines against which AutoML-Med is compared on the multiple sclerosis task.","marker":"[8]"},{"why":"Supplies another AutoML baseline, including an imbalanced-oriented variant used in the comparison.","marker":"[11]"},{"why":"Benchmarks AutoML tools on clinical data and frames the need for a medical-specific approach.","marker":"[16]"},{"why":"Introduces the LHS-PRCC methodology that AutoML-Med adapts for preprocessing exploration and sensitivity ranking.","marker":"[18]"},{"why":"Defines Latin Hypercube Sampling, the stratified sampling scheme at the core of pipeline exploration.","marker":"[19]"},{"why":"Defines PRCC, the non-parametric sensitivity measure used to rank the influence of preprocessing steps.","marker":"[20]"},{"why":"Supplies the PRCC regression procedure used in the fine-tuning phase.","marker":"[21]"},{"why":"Reports the type 2 diabetes risk prediction models and metrics to which AutoML-Med is compared.","marker":"[22]"}],"fun_headline_variants":["Preprocessing, not models, decides AutoML-Med's clinical wins","AutoML-Med finds at-risk patients by tuning preprocessing","Medical AutoML: Preprocessing search beats model choice","AutoML-Med targets preprocessing to lift sensitivity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the final one-third test split is never used during pipeline selection, PRCC ranking, or grid-search fine-tuning; the paper does not explicitly state this, and any leakage from the test set into the search would make the reported balanced accuracy and sensitivity optimistically biased.","fun_headline_variants_meta":{"raw":{"variants":["Preprocessing, not models, decides AutoML-Med's clinical wins","AutoML-Med finds at-risk patients by tuning preprocessing","Medical AutoML: Preprocessing search beats model choice","AutoML-Med targets preprocessing to lift sensitivity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000354,"raw_usage":{"total_tokens":1930,"prompt_tokens":952,"completion_tokens":978,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":914}},"tokens_in":568,"tokens_out":978,"duration_ms":7101,"temperature":1.0,"reasoning_tokens":914,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:36:57.782708+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the tool on the same public diabetes dataset with the test split held in a separate file and the code modified so that PRCC ranking and grid-search fine-tuning can see only training folds, then compare balanced accuracy and sensitivity on the untouched test set with the reported values; if they fall toward the comparison baselines' levels, the claimed advantage comes from test information rather than from the preprocessing search.","supporting_citations":[{"cited_title":"Benchmarking automl frameworks for disease prediction using medical claims,","cited_arxiv_id":null,"evidence_quote":"Benchmarks AutoML tools on clinical data and frames the need for a medical-specific approach."},{"cited_title":"A method- ology for performing global uncertainty and sensitivity analysis in systems biology,","cited_arxiv_id":null,"evidence_quote":"Introduces the LHS-PRCC methodology that AutoML-Med adapts for preprocessing exploration and sensitivity ranking."},{"cited_title":"Comparison of 3 methods for selecting values of input variables in the analysis of output from a computer code,","cited_arxiv_id":null,"evidence_quote":"Defines Latin Hypercube Sampling, the stratified sampling scheme at the core of pipeline exploration."},{"cited_title":"Non-parametric statistics in sensitivity analysis for model output: A comparison of selected techniques,","cited_arxiv_id":null,"evidence_quote":"Defines PRCC, the non-parametric sensitivity measure used to rank the influence of preprocessing steps."},{"cited_title":"Examining the influence of nondimensionalization on partial rank correlation coefficient results when modeling the epithelial mesenchymal transition,","cited_arxiv_id":null,"evidence_quote":"Supplies the PRCC regression procedure used in the fine-tuning phase."},{"cited_title":"Building risk prediction models for type 2 diabetes using machine learning techniques,","cited_arxiv_id":null,"evidence_quote":"Reports the type 2 diabetes risk prediction models and metrics to which AutoML-Med is compared."}],"review_version":2}