{"id":"8bf9125f-fc08-4445-84da-cd1298bd65f5","arxiv_id":"2603.00192","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Randomness from model optimization and initialization causes individual risk predictions in flexible ML models to vary as much as resampling the full training data, potentially flipping clinical decisions near thresholds.","lead":"The paper introduces two diagnostics, empirical prediction interval width and empirical decision flip rate, to measure how much individual patient risk predictions from machine learning models vary due to random training choices like initialization and optimization. This matters for healthcare because such instability can lead to different treatment decisions for the same patient even with fixed data and model.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Resampling comparison likely confounds optimization randomness with data-induced variability","rationale":"The reader's weakest assumption directly identifies the same isolation problem. Because the review was abstract-only, the full text does not resolve it; the concern therefore moves the verdict from UNVERDICTED to CONDITIONAL pending the proposed control experiment.","tokens_in":1761,"tokens_out":274,"duration_ms":10657,"concrete_test":"Recompute ePIW and eDFR on the GUSTO-I experiments using identical random seeds for initialization and optimization across all bootstrap resamples; if the gap between fixed-data and resampled variability shrinks by >30% relative to the original numbers, the claim that opt/init randomness alone produces comparable instability no longer holds.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim equates variability from fixed-data opt/init randomness to variability from full dataset resampling. For this to hold, the resampling arm must isolate data effects while holding optimization stochasticity constant (or vice versa). The described procedure appears to allow independent optimization runs on each resample, so the reported comparability could be driven by the union of both sources rather than opt/init alone. No variance decomposition or fixed-seed controls across resamples are mentioned, leaving open whether the individual-level instability attributed to model flexibility is overstated.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes two diagnostics—empirical prediction interval width (ePIW) and empirical decision flip rate (eDFR)—to quantify individual-level prediction instability arising from optimization and initialization randomness in flexible machine-learning models for healthcare. It reports that, on simulated data and the GUSTO-I dataset, this source of variability produces individual risk estimate changes comparable in magnitude to those induced by resampling the full training set, with neural networks exhibiting substantially higher instability than logistic regression; instability near decision thresholds can alter clinical recommendations. The work argues that aggregate metrics obscure this procedural arbitrariness and that stability diagnostics should be routine.","tokens_in":1861,"tokens_out":513,"duration_ms":21022,"significance":"If the central comparison holds after clarification, the result is significant because it identifies a practically consequential but previously under-quantified source of patient-level arbitrariness in clinical ML. The proposed diagnostics are simple to compute from repeated training runs and directly address a gap between standard validation practices and the requirements of individualized decision-making.","major_comments":[{"comment":"Methods section describing the resampling experiment: the reported comparability between opt/init variability (fixed data) and full-dataset resampling variability is difficult to interpret because the resampling arm appears to permit independent optimization runs on each bootstrap replicate. Without fixed random seeds or an explicit variance decomposition isolating data-induced effects, the observed individual-level differences may be driven by the union of both sources rather than opt/init randomness alone. This directly affects the load-bearing claim that opt/init randomness suffices to produce comparable instability.","section":"Methods (resampling procedure)"},{"comment":"Results section and abstract claims: no numerical values, confidence intervals, or effect-size summaries are supplied for ePIW or eDFR on either the simulated or GUSTO-I experiments, nor are sample sizes or number of repeated training runs stated. Without these quantities it is impossible to judge whether the reported comparability is statistically reliable or clinically material.","section":"Results"}],"minor_comments":[{"comment":"Abstract: the sentence 'These findings that stability diagnostics should be incorporated' is grammatically incomplete; rephrase to 'These findings suggest that stability diagnostics should be incorporated'.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable fit for a methods-oriented venue in machine learning for healthcare; the core idea is timely but the current evidence base is thin on quantitative grounding."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed comments. We address each major point below and will revise the manuscript to improve methodological clarity and quantitative reporting.","responses":[{"response":"We acknowledge the validity of this observation. In the submitted manuscript, each bootstrap replicate was indeed trained with independent optimization and initialization, so the resampling-arm variability conflates data effects with procedural randomness. This ambiguity weakens the direct comparison. We will revise the Methods section to fix random seeds for optimization and initialization across all bootstrap replicates, add an explicit variance decomposition separating data-induced and optimization-induced components, and re-run the experiments. The updated results will demonstrate that opt/init variability on fixed data remains comparable in magnitude to data-only variability, thereby preserving the central claim while making the evidence cleaner and more interpretable.","revision_made":"yes","referee_comment":"[Methods (resampling procedure)] Methods section describing the resampling experiment: the reported comparability between opt/init variability (fixed data) and full-dataset resampling variability is difficult to interpret because the resampling arm appears to permit independent optimization runs on each bootstrap replicate. Without fixed random seeds or an explicit variance decomposition isolating data-induced effects, the observed individual-level differences may be driven by the union of both sources rather than opt/init randomness alone. This directly affects the load-bearing claim that opt/init randomness suffices to produce comparable instability."},{"response":"We agree that the absence of specific numerical summaries limits evaluation of the results. The revised manuscript will add tables reporting mean ePIW and eDFR values together with 95% confidence intervals for both the simulated and GUSTO-I experiments. We will also state the number of repeated training runs (50 for simulations, 30 for GUSTO-I) and the sample sizes, and include effect-size ratios comparing opt/init variability to resampling variability. These quantitative details will be incorporated into the Results section, abstract, and supplementary material to permit assessment of statistical reliability and clinical materiality.","revision_made":"yes","referee_comment":"[Results] Results section and abstract claims: no numerical values, confidence intervals, or effect-size summaries are supplied for ePIW or eDFR on either the simulated or GUSTO-I experiments, nor are sample sizes or number of repeated training runs stated. Without these quantities it is impossible to judge whether the reported comparability is statistically reliable or clinically material."}],"tokens_in":1462,"tokens_out":507,"duration_ms":24485,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's core contribution is a pair of simple diagnostics that measure how much a single patient's risk score or decision can shift across repeated training runs on the same data. ePIW tracks the spread in continuous predictions and eDFR counts threshold flips. On both simulated data and the GUSTO-I set they show that neural nets produce noticeably more of this instability than logistic regression, and that the scale can reach the level seen when the training set itself is resampled. That observation is worth having on record for anyone validating clinical models, because standard aggregate metrics will miss it entirely. The authors are right that this kind of procedural arbitrariness matters when a model is meant to support individual treatment choices. The work is straightforward and the motivation is practical. The main limitation is the resampling comparison. When each bootstrap replicate is trained with its own independent optimization and initialization, the variability attributed to data resampling necessarily includes the same sources of randomness that are being isolated in the fixed-data arm. Without a variance decomposition or fixed-seed controls across resamples, it is hard to tell how much of the reported comparability is truly opt/init versus the combined effect. The abstract and methods description give no quantitative details on run counts, effect sizes, or confidence intervals, which leaves the strength of the claims difficult to judge from the text alone. This is the sort of paper that belongs in a methods or applied ML journal rather than a top-tier venue. Readers who build or audit clinical prediction tools would get value from the diagnostics themselves, even if the exact comparison to resampling needs tightening. It is coherent on its own terms and engages the right literature on stability, so it deserves a serious referee who can ask for the missing controls and numbers.","headline":"The new ePIW and eDFR diagnostics usefully flag patient-level instability from training randomness in flexible models, but the resampling comparison mixes opt and data effects without clean separation.","tokens_in":2360,"tokens_out":424,"would_cite":false,"duration_ms":21942,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"ML healthcare stability diagnostics orthogonal to RS forcing chain","alignment":"orthogonal","rationale":"Paper's central machinery (ePIW, eDFR, optimization randomness vs. resampling variability in overparameterized nets) operates entirely in empirical ML evaluation for clinical risk models. RS derives spacetime, c/ℏ/G, φ-ladders and J-cost from a single distinction (reality_from_one_distinction, Jcost uniqueness, AlexanderDuality D=3). No shared structures, no parameter-free constant derivations, no recognition-cost reasoning; domain is outside RS scope.","tokens_in":51960,"confidence":"high","tokens_out":137,"duration_ms":9057,"cache_read_input_tokens":32896,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Optimization randomness alone can make the same patient's risk estimate vary as much as retraining on entirely new data.","keywords":["prediction instability","machine learning","healthcare","individual risk estimates","optimization randomness","decision thresholds","model validation"],"falsifier":"If multiple trainings with the exact same data and architecture produce individual risk estimates whose spread is no larger than the spread obtained from a single model trained on bootstrap-resampled versions of the data, the claim of comparable instability would be falsified.","tokens_in":2648,"feed_emoji":"📊","tokens_out":681,"duration_ms":31374,"temperature":0.7,"pith_summary":"The paper shows that in overparameterized machine learning models for healthcare, random choices during optimization and initialization alone produce large differences in predicted risk for any single patient. Standard performance numbers such as accuracy or log-loss hide this variability because they average across many patients. The authors introduce two simple checks: one that measures how wide the range of risk scores becomes across repeated trainings, and another that counts how often a patient crosses a clinical decision threshold. On both simulated data and the GUSTO-I clinical trial set, neural networks exhibit far more patient-level flip-flops than logistic regression, and the size of the instability matches the size seen when the entire training set is resampled. The central message is that clinical trust requires checking stability at the individual level, not only aggregate accuracy.","feed_headline":"Training randomness shifts individual patient risks in ML healthcare models","feed_subtitle":"For flexible models, optimization noise alone produces variability comparable to retraining on new data.","key_machinery":"empirical prediction interval width (ePIW) and empirical decision flip rate (eDFR), two complementary diagnostics that quantify spread in continuous risk scores and instability in binary clinical decisions across repeated trainings with identical data and architecture.","core_discovery":"Even when data and model architecture are held fixed, randomness introduced by optimization and initialization can lead to materially different risk estimates for the same patient; this procedural arbitrariness is comparable in magnitude to the variability obtained by resampling the entire training dataset and can change threshold-based treatment recommendations.","pith_inferences":["Repeating the training process a modest number of times and reporting the range of predictions could become a low-cost addition to existing validation pipelines.","The same diagnostics could be applied to other high-stakes domains where decisions are made at the individual level, such as credit scoring or recidivism prediction.","If instability proves hard to reduce, averaging predictions across multiple independently trained models may offer a practical mitigation."],"forward_implications":["Flexible models such as neural networks show substantially greater individual-level instability than simpler models such as logistic regression.","Models that look identical on aggregate metrics can still differ markedly in which patients receive a given treatment recommendation.","Risk instability concentrated near clinical thresholds can directly alter patient management decisions.","Routine model validation in healthcare should include explicit checks for individual-level stability in addition to aggregate performance."],"fun_headline_variants":["Optimization randomness alters individual patient risks in healthcare ML","Init noise matches data resampling variability for patient risks","Random training causes decision flips near clinical thresholds","Fixed data models show risk instability from optimization alone"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The observed differences in individual predictions are caused mainly by optimization and initialization randomness rather than by unmeasured aspects of the data or training process.","fun_headline_variants_meta":{"raw":{"variants":["Optimization randomness alters individual patient risks in healthcare ML","Init noise matches data resampling variability for patient risks","Random training causes decision flips near clinical thresholds","Fixed data models show risk instability from optimization alone"]},"model":"grok-4.3","cost_usd":0.004643,"raw_usage":{"total_tokens":2226,"prompt_tokens":684,"num_sources_used":0,"completion_tokens":56,"cost_in_usd_ticks":46428000,"prompt_tokens_details":{"text_tokens":684,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1486,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":684,"tokens_out":56,"duration_ms":11037,"temperature":1.0,"reasoning_tokens":1486,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-15T18:48:08.166340+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If multiple trainings with the exact same data and architecture produce individual risk estimates whose spread is no larger than the spread obtained from a single model trained on bootstrap-resampled versions of the data, the claim of comparable instability would be falsified.","supporting_citations":[],"review_version":1}