{"id":"2c5184de-b626-4434-bb5d-1c7b12774d50","arxiv_id":"2608.00271","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A reproducible benchmarking framework and a new SHAP extension for competing-risks survival models, with results showing simpler regression models often match deep learning.","lead":"An open-source pipeline benchmarks six competing-risk survival models on five real datasets, testing calibration, discrimination, prediction error, and clinical usefulness. It also adds a SHAP-based tool (CifSHAP(t)) that shows how each risk factor changes each event's predicted probability over time.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark rankings may be artifacts of unequal hyperparameter grid widths and the IBS tuning criterion; the paper acknowledges but does not quantify this, and fold-level CIs overlap, so 'DeSurv outperforms' is not robustly supported.","rationale":"The reader identified the hyperparameter grid width and IBS tuning criterion as the weakest assumption, and I agree. The central claim of a neutral benchmark depends on all methods receiving comparably generous search opportunities and on the tuning metric not systematically favoring some model families. The paper itself flags the dependency on the optimization grid (§3.6) and notes the small IBS spread as partly a consequence of IBS-based tuning (§4.2.2), but it does not quantify how rankings change with grid width for any method other than DeepHit (and only for tuning criterion, not grid). The reported confidence intervals are wide relative to the small mean differences in IBS, and with only 3–5 outer folds, the 'outperform' claims are fragile. This does not invalidate the framework as a reproducible benchmarking tool—the code is public, the nested CV design is sound, and the authors are appropriately cautious in the Discussion. It does, however, mean that the specific model rankings should be interpreted as conditional on the search protocol, exactly as the reader concluded. Therefore the verdict remains CONDITIONAL; no change to the reader's assessment is needed.","tokens_in":39474,"tokens_out":7936,"duration_ms":80487,"concrete_test":"On METABRIC and Framingham, re-run the full pipeline with (i) expanded grids: random search with at least 50 configurations for each tuned method, (ii) alternate tuning criteria: IBS, model-specific loss, and Ctd, and (iii) patient-level bootstrap 95% CIs for IBS and Ctau. If DeepHit moves from last place, or if the DeSurv-versus-best-regression IBS/Ctau difference becomes non-significant (overlapping bootstrap intervals), the headline rankings are artifacts of the search protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core claim of neutrality requires comparably fair hyperparameter search. Table S3 shows orders-of-magnitude differences in grid sizes: FGRP has 30 configurations, RSF has 4, DeSurv has 18, DeepHit has 10. With IBS as the common tuning criterion (§3.6), methods with larger grids or whose objective aligns with IBS have more opportunity to minimize the evaluation metric. The paper states 'any performance claims are bound to the chosen optimisation grid' (§3.6) but provides no sensitivity analysis over grid width—§4.5 only varies the tuning criterion for DeepHit, not the grid. Moreover, Table 4 indicates IBS differences are small relative to fold-level uncertainty (e.g., METABRIC DeSurv 0.1519 [0.1486,0.1561] vs csCPH 0.1550 [0.1487,0.1599]), and 95% CIs are based on only 5 folds and ignore training-set overlap, so 'DeSurv outperformed' in three datasets is not established as a robust effect. Consequently, the headline empirical rankings could be artifacts of the search protocol rather than intrinsic model quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an open-source, extensible benchmarking framework for competing-risks survival models, applies it to six models (csCPH, FGR, FGRP, RSF, DeSurv, DeepHit) across five datasets, and evaluates calibration, discrimination, overall prediction error, and clinical utility under nested cross-validation. It also introduces CifSHAP(t), a time-dependent SHAP extension for cumulative incidence functions. The central empirical claims are that the pipeline provides a neutral comparison, that simpler regression-based methods remain competitive, and that DeSurv outperforms on IBS and Cτ in three of five datasets. The paper includes detailed appendices on hyperparameter grids, imputation, supplementary results, and software/Docker support.","tokens_in":39798,"tokens_out":2655,"duration_ms":28461,"significance":"If the benchmark conclusions are robust, the contribution is valuable: a reproducible, Dockerized pipeline with deterministic training, common nested CV, fold storage, and a common tuning criterion directly addresses the reproducibility gap in competing-risks benchmarking, and CifSHAP(t) is a useful model-agnostic interpretability extension. The paper also ships code and empirical verification of the SHAP reconstruction constraints. However, the headline model rankings are conditional on hand-picked grids and a single tuning criterion, and the statistical support for the claimed outperformance is currently weak; the framework's neutrality claim is therefore not yet fully established.","major_comments":[{"comment":"Hyperparameter search budgets are highly unequal across methods: FGRP has 30 configurations, DeSurv 18, DeepHit 10, and RSF only 4. With IBS as the common tuning criterion, methods with more extensive grids have more opportunity to minimize the evaluation metric. The text acknowledges that performance claims are bound to the chosen grid (§3.6), but no sensitivity analysis over grid width is provided; §4.5 varies only the tuning criterion for DeepHit. This leaves the 'neutral comparison' claim load-bearing on an unquantified fairness assumption. Please add a grid-width sensitivity analysis (e.g., restricted grids for FGRP/DeSurv, or expanded grids for RSF/DeepHit) and report whether the main rankings persist.","section":"§3.6 and Table S3"},{"comment":"The claim that 'DeSurv outperformed based on IBS and Cτ in three out of the five datasets' is not supported by the reported uncertainty. Confidence intervals in Table 4 are computed from only 5 outer folds, ignore training-set overlap, and largely overlap between top and bottom models; e.g., METABRIC IBS for DeSurv is 0.1519 [0.1486,0.1561] versus csCPH 0.1550 [0.1487,0.1599] and FGRP 0.1545 [0.1486,0.1592]. No paired comparisons, hypothesis tests, or effect-size analysis is provided. The paper should either temper the outperformance claim to 'differences were not statistically resolvable at fold-level uncertainty' or add a formal comparison (e.g., paired tests across folds with multiplicity control).","section":"§4.2.2 and Table 4"},{"comment":"The evaluation feedback loop is acknowledged ('potentially driving tunable methods towards similar IBS values') but not mitigated. Because IBS is both the tuning criterion and a headline metric, models whose architecture aligns with smooth CIF estimation may be systematically favored, while methods whose native objectives target discrimination or likelihood are disadvantaged. The sensitivity analysis in §4.5 is limited to one method (DeepHit). To support the neutrality claim, the benchmark should report headline metrics under at least one alternative tuning criterion (e.g., model-specific loss or Cτ) for all tunable methods, or show that the relative rankings are stable when the tuning criterion is changed.","section":"§3.6 and §4.2.2"}],"minor_comments":[{"comment":"Typo: 'preformed' should be 'performed'. Also 'laying above' in §4.2.3 should be 'lying above'.","section":"§3.6"},{"comment":"Supplementary heading 'Missigness' should be 'Missingness'.","section":"S1.1.1"},{"comment":"Please add a column or footnote reporting the number of configurations per method; the uneven grid sizes (30, 4, 18, 10) are relevant to interpreting the benchmark and should be explicit.","section":"Table S3"},{"comment":"The empirical verification of CifSHAP(t) constraints is reported only for METABRIC DeepHit and a single notebook. Since CifSHAP(t) is a proposed general extension, consider verifying constraints across all models and datasets, or state the scope of the verification more precisely in the main text.","section":"§3.5 / S1.5.2"},{"comment":"Pairwise agreement is quantified by Pearson correlation, which is not invariant to monotone transformations and may overstate agreement for skewed risk distributions. Consider also reporting rank-based measures (e.g., Spearman) for the RMTL comparisons.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's framework and code are genuinely valuable, but the empirical ranking claims are not yet robust to the acknowledged grid/tuning-criterion dependencies and fold-level uncertainty. The required additions — grid-width sensitivity, alternative tuning criteria for all methods, and formal uncertainty quantification — are feasible within the manuscript's scope and would convert a potentially artifact-driven benchmark into a defensible one. I do not see a reason for rejection, but the current analysis is not yet suitable for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper to know about, not because it overturns anything but because it plugs a real gap: there is no standard, reproducible benchmark for competing-risks survival models, and this one is careful. The framework ships code, Docker, nested CV, a common tuning criterion, multiple metrics, and a sensible new interpretability tool (CifSHAP(t)), which is a direct time-dependent extension of SurvSHAP(t) to CIFs. The empirical verification that the reconstructed CIF obeys probability constraints is a nice touch. The benchmark itself is honest: they report calibration, discrimination, prediction error, clinical utility, runtime, and they flag that hyperparameter tuning on IBS may compress IBS differences. That is real credit.\n\nThe soft spots are real but not fatal. The headline—'DeSurv outperformed based on IBS and Cτ in three of five datasets'—is not robustly supported by the numbers. The fold-level 95% CIs overlap for METABRIC (e.g. DeSurv IBS 0.1519 [0.1486,0.1561] vs csCPH 0.1550 [0.1487,0.1599]), and they are based on only 5 folds with training-set overlap ignored. So the difference is suggestive, not demonstrated. The second soft spot is the evaluation-feedback loop: tuning on IBS and then reporting IBS can favor tunable models, and the sensitivity analysis only varies the tuning criterion for DeepHit, not the grid width. Table S3 shows very different grid sizes (30 configs for FGRP, 4 for RSF), so rankings may partly reflect search generosity. The authors acknowledge this in §3.6 and §4.2.2, but they don't quantify it.\n\nThe paper does what it claims: it provides a reusable, extensible framework and compares six models across five datasets under a consistent protocol. The conclusions are carefully worded elsewhere—'comparative evidence under this specific benchmark set up'—so the overreach is mostly in the abstract and result phrasing, not in the discussion.\n\nWho benefits: anyone building or selecting competing-risks models, especially in clinical prediction; method developers who want a neutral testbed. It deserves a serious referee. I'd send it out and in revision ask for either a grid-width sensitivity check or a downgrade of the 'outperform' language to 'under this protocol'. I would cite it.","headline":"Careful benchmarking resource for competing-risks models with a sound but unsurprising CifSHAP(t) extension; the empirical rankings are honest but should be read as protocol-specific, not robust.","tokens_in":40214,"tokens_out":2200,"would_cite":true,"duration_ms":23334,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62N01","62N02","62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"A standardized, reproducible benchmarking pipeline can fairly compare competing-risks survival models, and under it simpler regression-based methods remain highly competitive while the deep-learning model DeSurv leads on three of five datas","keywords":["competing risks","survival analysis","benchmarking","cumulative incidence function","model calibration","concordance index","SHAP","interpretability"],"falsifier":"Re-run the benchmark on METABRIC with substantially wider hyperparameter grids for DeepHit (deeper architectures, more learning-rate choices) and for RSF (more tree sizes), and with tuning criteria set to each method's own loss; if DeepHit no longer ranks worst under IBS and Cτ, the paper's model ranking is grid-dependent. A simpler check: compare each tuned model's IBS against the best IBS achievable across a much denser grid—a large gap for DeepHit would flag grid stinginess.","tokens_in":39402,"feed_emoji":"📊","tokens_out":5602,"duration_ms":50397,"temperature":0.7,"pith_summary":"Survival models that ignore competing risks overestimate event risk, yet most published comparisons of newer machine-learning methods are not reproducible. This paper builds an open, extensible benchmarking pipeline that puts six competing-risks models—classical regression, random survival forests, and two deep-learning approaches—through the same nested cross-validation, tuning, and evaluation protocol on five real-world datasets. Its main empirical finding is that simpler regression-based methods remain highly competitive, while the deep-learning model DeSurv outperforms the others on integrated Brier score and the Cτ concordance index in three of the five datasets. The paper also introduces CifSHAP(t), a time-dependent SHAP extension that attributes each covariate's contribution to the cumulative incidence function of each competing cause. A sympathetic reader would care because the framework offers a fair way to compare new methods, and the results caution against assuming deep learning automatically beats classical survival models.","feed_headline":"Simple models rival deep learning in survival benchmark","feed_subtitle":"Reproducible pipeline shows regression methods hold their own; DeSurv leads on three of five datasets.","key_machinery":"The load-bearing mechanism is the benchmark pipeline itself: an outer/inner nested cross-validation with stratified folds, parquet storage for interoperability, and a single pre-specified hyperparameter grid per tunable method, with all tuning done to minimise the same integrated Brier score. On top of this sits a common metric suite (calibration plots, ICI, O/E ratio, tdAUC, Cτ and Ctd, Brier/IBS, and decision curves) and the new CifSHAP(t) extension of SurvSHAP(t), which uses Kernel SHAP to approximate per-covariate contributions to the cause-specific CIF over time. The pipeline is what makes the comparison neutral—the paper's stated goal—by removing inconsistencies in preprocessing, predi","core_discovery":"Under a single evaluation protocol—nested cross-validation with a common time grid, one-hot encoding, and the integrated Brier score as the shared hyperparameter tuning criterion—the paper compares cause-specific Cox, Fine–Gray, penalised Fine–Gray, random survival forests, DeSurv, and DeepHit across five datasets. The central discovery is that simpler regression-based approaches frequently achieved competitive performance, and that DeSurv achieved the best integrated Brier score and Cτ in three of the five datasets, while DeepHit was often the worst. The paper also proposes CifSHAP(t), which decomposes the predicted cumulative incidence function for each cause at each time point into additi","pith_inferences":["The paper's own limitation analysis suggests the rankings are conditional on hyperparameter grid widths; a natural next step is a grid-sensitivity analysis that varies grid generosity per method and checks whether the DeSurv-over-DeepHit ordering persists.","Because CifSHAP(t) attributes contributions to the CIF per cause, it could be used to audit fairness in competing-risk settings—e.g., whether a model systematically shifts the predicted breast-cancer-death CIF differently across racial or age subgroups—something the paper does not explore.","The framework is currently confined to low-dimensional tabular, right-censored data; extending it to high-dimensional genomic or longitudinal/time-varying covariates would test whether the 'simple models are competitive' finding generalises.","The observation that early stopping hurt performance under IBS tuning suggests that common deep-learning training practices may be misaligned with benchmarking protocols; a separate study could isolate early stopping's effect when tuning by the model's own loss."],"forward_implications":["New competing-risks methods can be dropped into the same pipeline and compared against six established baselines without re-implementing different preprocessing or metric code.","Because simpler regression models stayed competitive and often beat deep learning, future work should treat them as the baseline to beat rather than assuming neural methods dominate.","The choice of hyperparameter tuning criterion can change model rankings; reporting the tuning protocol is as important as reporting the metric.","CifSHAP(t) lets clinicians and model developers see how a covariate's contribution to cause-specific risk evolves over time, and whether it pushes the CIF up or down for each competing event.","Model rankings depend on which metric is used (IBS vs Cτ vs Ctd), so deployment should select a metric matched to the intended clinical use."],"fun_headline_variants":["Benchmark shows simple survival models rival deep learning","New framework benchmarks competing-risk survival models","CifSHAP brings interpretability to competing risks","Simple models match deep learning in survival benchmark","Reproducible benchmark: simple survival models compete"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's model rankings rest on the assumption that the pre-specified hyperparameter grids and the common IBS-based tuning criterion treat every method comparably; if some grids are too narrow or some methods' losses are misaligned with IBS, the reported ordering could be an artifact of the search protocol rather than of model quality.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark shows simple survival models rival deep learning","New framework benchmarks competing-risk survival models","CifSHAP brings interpretability to competing risks","Simple models match deep learning in survival benchmark","Reproducible benchmark: simple survival models compete"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1016,"prompt_tokens":650,"completion_tokens":366,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":394,"completion_tokens_details":{"reasoning_tokens":296}},"tokens_in":394,"tokens_out":366,"duration_ms":4123,"temperature":1.0,"reasoning_tokens":296,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T00:50:26.925197+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark on METABRIC with substantially wider hyperparameter grids for DeepHit (deeper architectures, more learning-rate choices) and for RSF (more tree sizes), and with tuning criteria set to each method's own loss; if DeepHit no longer ranks worst under IBS and Cτ, the paper's model ranking is grid-dependent. A simpler check: compare each tuned model's IBS against the best IBS achievable across a much denser grid—a large gap for DeepHit would flag grid stinginess.","supporting_citations":[],"review_version":1}