{"id":"8fff79c9-1641-49d0-b371-9706824ba2fa","arxiv_id":"2603.02221","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MedFeat iteratively prompts an LLM to create clinical features guided by SHAP importance and the downstream model's inductive bias, improving tabular prediction AUC/F1 on five EHR tasks.","lead":"MedFeat uses a large language model to propose new patient features for clinical prediction, guiding it with model explanations and the downstream model's strengths and limits. It reports small but consistent AUC/F1 gains across five clinical tasks, though no code or significance tests are currently provided.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's claims of 'statistically significant' and '>10% average improvement' are not supported by the paper's tables: average gains are ~6%, and no significance tests are reported.","rationale":"The reader's weakest assumption focuses on the stability of validation AUC as a selection signal. That is a valid concern about the mechanism by which MedFeat could overfit to noise. However, the more direct and load-bearing problem is that the paper's own reported tables do not support the abstract's quantitative and statistical claims. The '>10% average improvement' appears to be a miscalculation or an overstatement when derived from the listed metrics, and no significance tests are provided to back 'statistically significant.' This is a correctness risk that affects the central contribution itself, not just the internal selection process. The inconsistency between Algorithm 1 and §3.5 in the acceptance condition adds a reproducibility obstacle. I do not believe this warrants outright rejection because the framework is well-motivated and the conditional verdict already requires code release and proper statistical tests; the authors could resolve the concern by providing the exact averaging procedure, significance tests, and aligning the algorithm with the text. Thus, the reader's CONDITIONAL verdict remains appropriate, with no change needed from my stress-test.","tokens_in":18185,"tokens_out":5693,"duration_ms":53897,"concrete_test":"Recompute the average of the percentage improvements across all rows in Tables 2, 3, and 11 (or the exact subset used for the abstract's claim). Then run a paired bootstrap or Wilcoxon signed-rank test on the per-split test AUC/F1 values for each task and model. If the mean improvement is not >10% or the paired test is not significant at α=0.05, the abstract's headline claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that MedFeat 'statistically significantly outperforms state-of-the-art baselines, with an average improvement of more than 10%' (Abstract)—is not supported by the reported evidence. Averaging the percentage improvements listed in Tables 2, 3, and 11 yields roughly 5–6%, not >10%. Many improvements are within one standard deviation of the baseline (e.g., 24h ICU mortality AUC: 0.760±0.008 vs 0.776±0.020; heart failure: 0.685±0.002 vs 0.687±0.009). No significance tests or confidence intervals are reported anywhere; with only three random splits, the standard errors are too large to support 'statistically significant.' Additionally, Algorithm 1 (Appendix A) accepts a candidate if L_{t,k} ≥ L_base − β, whereas §3.5 states the condition as L_{t,k} ≥ L_base + β; this discrepancy changes which features are selected and undermines reproducibility of the claimed gains. The paper's own limitations (Appendix I) acknowledge that SHAP importance ordering may vary across splits, reinforcing the need for proper statistical validation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MedFeat is an iterative, LLM-driven feature engineering framework for clinical tabular prediction. It uses SHAP importance scores to sample small 'islands' of features, conditions the LLM prompt on the downstream model's inductive bias (logistic regression vs. XGBoost), and accepts/rejects proposed transformations using a validation-AUC tolerance. The authors evaluate on five clinical tasks from IORD, MIMIC-IV, and HRS, comparing against raw baselines, classical AFE (OpenFE), and LLM-based methods (CAAFE, FeatLLM, OCTree), in both untuned and HPO-tuned settings. They also report ablations, cross-cohort feature transfer, and temporal-shift robustness. The abstract claims that MedFeat 'statistically significantly outperforms state-of-the-art baselines, with an average improvement of more than 10%'.","tokens_in":18448,"tokens_out":4835,"duration_ms":49881,"significance":"The core idea—using model-aware, explainability-guided feedback to direct LLM feature generation—is timely and relevant to clinical tabular learning. If the empirical claims held, the framework would be a practical advance over both operator-based AFE and existing LLM-based feature engineering, with the additional privacy benefit of not sending raw patient records. The paper includes useful ablations and a thoughtful discussion of model-dependent feature value. However, the current evidence is not sufficient to support the headline statistical claims: no significance tests are reported, the average improvement figure is not reproducible from the tables, and the acceptance rule is specified inconsistently between the text and Algorithm 1. These issues are load-bearing for the paper's main claims and require substantive revision.","major_comments":[{"comment":"The abstract claims 'statistically significantly outperforms' and 'an average improvement of more than 10%'. Neither claim is supported by the reported evidence. No significance tests, confidence intervals, or paired comparisons are provided anywhere; with only three random splits, the standard errors are large. Averaging the AUC percentage improvements across Tables 2, 3, and 11 gives roughly 1.5% (not >10%); the >10% figure appears to arise from relative F1 gains on near-zero F1 baselines (e.g., F1 0.014→0.018 is reported as +28.57%). Many AUC gains are within one standard deviation of the baseline (e.g., Table 2, 24h ICU mortality: 0.760±0.008 vs 0.776±0.020; heart failure: 0.685±0.002 vs 0.687±0.009). The authors should either provide statistically valid evidence (e.g., paired bootstrap or permutation tests across more seeds) or substantially temper the claims.","section":"Abstract; Tables 2, 3, 11"},{"comment":"There is a direct contradiction in the acceptance rule. §3.5 states that a candidate is accepted if L_{t,k*} ≥ L_base + β, whereas Algorithm 1 accepts if L_{t,k*} ≥ L_base − β. These are different thresholds and will select different feature sets. Additionally, §3.5 defines L as 'validation loss' and selects k* = arg min_k L_{t,k}, while Algorithm 1 uses k* = arg max_k L_{t,k}; if L is a loss, higher is worse, and accepting when L_{t,k*} ≥ L_base + β would accept a worse model. This ambiguity makes the reported results non-reproducible and must be fixed.","section":"§3.5 vs Appendix A, Algorithm 1"},{"comment":"The iterative acceptance rule assumes that validation AUC is a stable signal under severe class imbalance, but the paper's own numbers show baseline SDs often larger than the gains and larger than the default tolerance β=0.01. For example, in Table 2 the 24h inpatient mortality AUC gain is 0.054 while the baseline SD is 0.033; in Table 3 several XGBoost AUC gains are 0.00–0.01 with SDs of 0.005–0.038. With only three splits, the validation signal used for feature selection is likely dominated by noise. The paper should report the number of accepted features per split, the validation AUC trajectory, and ideally a nested or repeated-validation estimate of selection stability. Appendix I's acknowledgment that 'importance ordering might vary across splits' further reinforces this concern.","section":"§4.3, Tables 2 and 3"},{"comment":"The HPO results do not support the abstract's unqualified 'statistically significantly outperforms'. In Table 3, XGBoost AUC gains after HPO are 0.83%, 0.00%, 0.56%, 0.14%, and 0.65%—all within one standard deviation. The text appropriately says 'remains competitive', but the abstract and conclusion should be qualified to distinguish the untuned regime, where gains are larger, from the tuned regime, where they largely vanish. This is a central claim of the paper and needs to be represented accurately.","section":"Table 3 and §4.4"}],"minor_comments":[{"comment":"Figure 2 reports 'five different runs' but shows no error bars or significance tests. Figure 3 shows a single temporal trajectory without confidence intervals. For claims of reduced variability and drift robustness, error bars across repeated runs or resampling are needed.","section":"Figures 2 and 3"},{"comment":"Ablation results are reported only as percentage drops relative to the full model, without the underlying absolute AUC/F1 values and without error bars. On very small F1 values (e.g., 0.013–0.018), a 35.9% drop is less meaningful; absolute values and standard deviations should be included.","section":"Table 4"},{"comment":"The symbol L is called 'validation loss' but is used as a higher-is-better metric (AUC). This is confusing and should be renamed (e.g., 'score' or 'metric') to avoid the appearance of accepting worse loss.","section":"§3.5"},{"comment":"AutoFeat is listed in Table 1 and in the methods list, but excluded from experiments due to runtime. This is understandable, but the table should note that it was not evaluated. Also, FeatLLM is marked 'Privacy Preserving: ×' in Table 1, yet Appendix D says few-shot samples were removed to ensure privacy; this inconsistency should be clarified.","section":"Appendix D, Table 1"},{"comment":"Minor typos and inconsistent capitalization: 'Medfeat' vs 'MedFeat', 'statisitics' in the prompt example, and several run-on sentences. These do not affect the technical content.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a strong and interesting framework, but the headline claims overstate the evidence. The authors should be asked to either provide proper statistical validation (more seeds, paired tests, confidence intervals) and correct the Algorithm 1 inconsistency, or rewrite the abstract and conclusion to reflect the actual magnitude and uncertainty of the improvements. The 'statistically significant' and '>10% average improvement' claims are not currently defensible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — MedFeat is a well-constructed framework that makes a nice design point: use SHAP importance and model-awareness to guide LLM feature proposals, with island sampling and memory. The ablations suggest each component matters, and the cross-cohort transfer and temporal shift experiments are a genuine plus. But the paper's central empirical claims are not backed by the reported numbers. The abstract says \"statistically significantly outperforms\" and \">10% average improvement.\" Averaging the improvements in Tables 2, 3, and 11 gives about 5–6%, and there are no significance tests anywhere. Many gains are within one standard deviation of the baseline. With only three random splits, the standard errors are too large to support \"significant.\"\n\nThe framework itself is reasonable. The acceptance rule uses validation AUC with a tolerance β, which is standard model selection, not circular. But there is a real reproducibility bug: Algorithm 1 accepts when L_{t,k} ≥ L_base − β, while Section 3.5 says L_{t,k} ≥ L_base + β. Those are very different. The pseudocode looks like a typo (worse features would be accepted), but it needs to be resolved before anyone can reproduce. Also note the paper's own limitation section acknowledges SHAP ordering may vary across splits, which undercuts the stability premise.\n\nThe strongest part of the paper is the design and the qualitative feature examples; the weakest is the evidence. The authors need paired bootstrapping or repeated cross-validation with confidence intervals, and code with fixed splits. The transfer experiments are suggestive but also based on five runs, not rigorously quantified.\n\nThis paper should not be desk-rejected — the idea is worth refereeing — but it needs major revision. If the empirical claims are fixed, it would be a solid subfield contribution. For now, I'd be cautious about citing the quantitative results. The framework description and algorithm are useful to know.","headline":"A clever and ambitious LLM feature-engineering framework whose abstract overclaims; the design deserves peer review but the empirical evidence as reported does not support the headline numbers.","tokens_in":18957,"tokens_out":1884,"would_cite":false,"duration_ms":19072,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MedFeat claims that large language models can engineer clinical features far more effectively when the generation loop is told which features the model already relies on and what it cannot learn alone.","keywords":["feature engineering","large language models","clinical tabular prediction","SHAP","explainability","model awareness","electronic health records","feature selection"],"falsifier":"Re-run MedFeat on the same splits but with the acceptance rule changed to require the validation AUC gain to exceed the bootstrap standard error of the validation metric. If accepted features no longer improve held-out AUC or F1, the reported gains were validation noise. Also shuffle the SHAP importance rankings before island sampling; if held-out performance still improves, the importance signal is not the active ingredient.","tokens_in":18071,"feed_emoji":"🩺","tokens_out":7412,"duration_ms":71586,"temperature":0.7,"pith_summary":"MedFeat claims that LLM feature engineering for clinical tabular prediction works best as a feedback loop, not a one-shot search: the LLM receives Shapley-based importance scores from the current model, a short profile of the model's inductive bias, and a memory of past successes and failures before proposing new features. The paper argues this model-awareness and explainability guidance makes proposals targeted rather than generic, and shows that across five clinical tasks and two learner families, the loop improves held-out AUC and F1 in the untuned regime, remains competitive after hyperparameter optimization, and produces features that transfer between ICU and general hospital populations and across years. If true, clinicians could obtain interpretable, deployable features with less manual engineering and more robustness to distribution shift. The key mechanism is the iterative accept/reject loop driven by validation AUC with a tolerance threshold, guided by importance-weighted sampling of small feature islands.","feed_headline":"LLM features guided by the model lift clinical prediction AUC","feed_subtitle":"MedFeat feeds the model's own importance scores back to the LLM, so new features stay useful across cohorts and years.","key_machinery":"The load-bearing mechanism is the MedFeat iteration: (1) train a baseline learner on the current feature set; (2) compute SHAP (Shapley additive explanations) importance on validation data; (3) sample K feature islands, which are small subsets drawn proportional to normalized importance, with repeated temporal measurements grouped to handle collinearity; (4) prompt the LLM with the island, a model-awareness paragraph that tells it what the learner can and cannot represent, and a memory bank of accepted and rejected features; (5) execute candidates locally, retrain, and accept the best island only if validation AUC clears the current baseline by tolerance beta = 0.01; (6) update the memory an","core_discovery":"The central claim is that feature engineering should be conditioned on the downstream model's actual behavior. Starting from a baseline model, MedFeat computes Shapley additive explanations (SHAP) on the validation set, ranks features by importance, samples small islands of influential features, and prompts an LLM to write executable transformations that specifically target patterns the learner cannot represent—nonlinear interactions for linear models, temporal and aggregate statistics for tree ensembles. Candidates are evaluated by retraining the model and accepting the best only if validation AUC improves by a tolerance of 0.01, then the importance scores are recomputed and the loop repeat","pith_inferences":["A testable extension is to replace the fixed 0.01 validation-AUC tolerance with a confidence-interval or bootstrap rule; the paper's own tables show split-to-split standard deviations that often exceed reported gains, so a noise-calibrated acceptance rule could separate signal from luck.","If SHAP-guided islands are truly the active ingredient, then feeding the LLM shuffled importance rankings should destroy most of the gain; this null experiment would directly test the explainability-driven design.","The stronger gains for logistic regression than for trees after HPO suggest the framework is most valuable when the downstream model is deliberately kept simple for interpretability or regulatory reasons, which could make simple linear models competitive with heavily tuned ensembles.","The temporal-shift results raise the possibility that the engineered features encode more invariant clinical mechanisms than raw measurements, which could be exploited by using them as priors for models deployed in new sites with different coding systems."],"forward_implications":["In the untuned or default regime, MedFeat reports best or second-best AUC on all tasks for both learner families, with the largest lift on 24-hour inpatient mortality (AUC 0.686 to 0.740 for trees) and consistent F1 gains across tasks.","After 400-trial hyperparameter optimization, AUC gains mostly shrink but F1 gains persist, especially on the most imbalanced tasks, so the practical claim is that feature engineering helps most when tuning budgets are limited.","The acceptance criterion and importance guidance produce features that tend to appear in the top-10 SHAP rankings of the final model, supporting the interpretability goal.","MedFeat features transfer across settings: features generated on the ICU cohort improve AUC and reduce run-to-run variability when applied to a general inpatient cohort, and a model with MedFeat features keeps higher, more stable AUROC under temporal shift than a model retrained annually without new features.","The framework avoids sending patient-level records to the LLM, which is a prerequisite for many clinical privacy constraints."],"fun_headline_variants":["Model-aware LLM features boost clinical prediction by 10%","SHAP-guided LLM engineering lifts clinical AUC across models","MedFeat: LLM features tuned to model blind spots improve predictions","Clinical tabular prediction improved via model-aware LLM feature engineering","Feedback loop: model insights steer LLM features to beat baselines"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The loop accepts features based on a single validation-set AUC comparison with a fixed 0.01 tolerance, assuming that this comparison is a stable, representative signal of held-out generalization despite severe class imbalance and split-to-split variance.","fun_headline_variants_meta":{"raw":{"variants":["Model-aware LLM features boost clinical prediction by 10%","SHAP-guided LLM engineering lifts clinical AUC across models","MedFeat: LLM features tuned to model blind spots improve predictions","Clinical tabular prediction improved via model-aware LLM feature engineering","Feedback loop: model insights steer LLM features to beat baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000555,"raw_usage":{"total_tokens":2463,"prompt_tokens":713,"completion_tokens":1750,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":1676}},"tokens_in":457,"tokens_out":1750,"duration_ms":11634,"temperature":1.0,"reasoning_tokens":1676,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T02:37:52.905467+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run MedFeat on the same splits but with the acceptance rule changed to require the validation AUC gain to exceed the bootstrap standard error of the validation metric. If accepted features no longer improve held-out AUC or F1, the reported gains were validation noise. Also shuffle the SHAP importance rankings before island sampling; if held-out performance still improves, the importance signal is not the active ingredient.","supporting_citations":[],"review_version":1}