Pith. sign in

REVIEW 3 major objections 6 minor 20 references

Predictiveness Curve Assessment under Competing Risks for Risk Prediction Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes a cross-validated predictiveness curve for competing-risks predictions, treating the fitted regression as a working model.

desk verdict A genuinely useful extension of the predictiveness curve to competing risks, with solid simulations but an unproven perturbation variance estimator that needs a theorem before the inference claims can be trusted. read the letter →

arxiv 2508.00216 v1 pith:CNOS7XXA submitted 2025-07-31 stat.ME

Wei Tao , Jing Ning , Wen Li , Wenyaw Chan , Xi Luo , Ruosha Li This is my paper · ORCID
classification stat.ME MSC 62N0162G2062P10
keywords predictivenesscurvecompetingriskscumulativeincidencefunctionFine-Graymodelcross-validationperturbationresamplingriskpredictionmisspecification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Competing-risks prediction models output a risk score that is meant to track the cumulative incidence of an event such as liver mortality, but patients can instead die of other causes or receive a transplant. Standard predictiveness curves plot outcome risk against the quantile of a marker, and this paper adapts that tool to the competing-risks setting. The proposed estimator fits a Fine-Gray working model on one cross-validation split, regresses the tau-year event indicator on the resulting risk score using an IPCW-weighted logistic restricted-cubic-spline model, and averages over repeated splits. Perturbation resampling then produces standard errors and Wald confidence intervals for the curve and its inverse. Simulations with sample sizes 200 to 800 show small bias and near-nominal coverage under both correct and misspecified working models.

What carries the argument

The central object is the competing-risks predictiveness curve R(v), the cumulative incidence of the event of interest at the vth quantile of a model-based risk score, together with its inverse $R^{{-1}}$(p), the population fraction with risk below p. The estimation machinery is the combination of (i) repeated two-fold cross-validation to avoid optimism, (ii) a flexible IPCW-weighted binomial regression using restricted cubic splines of the risk score to approximate the CIF locally at time tau, and (iii) perturbation resampling that reweights the Fine-Gray fit, the spline regression, and the quantile computation to obtain standard errors.

What would settle it

Run the proposed method on simulated data with, for example, n = 200, 60% censoring, and a misspecified working model, and compare the perturbation-based 95% interval coverage with empirical coverage from many replicates; if coverage at interior quantiles falls substantially below nominal (say below 85%) while bias is small, the validity assumption fails.

Watch

Extended reading notes

Core claim

The central claim is that the predictiveness curve R(v) = P(T <= tau, epsilon = 1 | xi = Q(v)), where xi is the linear predictor from a Fine-Gray model and Q(v) is its vth quantile, can be estimated reliably even when the model is only a working model. Estimation proceeds by repeated two-fold cross-validation: fit the working model on one split, form risk scores for the other, and fit the flexible binomial model P(T <= tau, epsilon = 1 | xi) = g{theta_0 + theta_B^T e_B(xi)} with inverse-logit link and restricted cubic splines, using inverse probability of censoring weights to handle right-censoring. The vth quantile of the score is plugged in to give R_hat(v), and the whole procedure is repeated with perturbation weights to approximate the sampling distribution. The paper reports that the resulting estimator has minimal bias and confidence coverage close to nominal for n = 200, 400, 800, including when the Fine-Gray model is misspecified, and demonstrates the method on a cirrhosis cohort predicting five-year liver-related mortality.

Load-bearing premise

The inference procedure assumes, without a proof in the paper, that its perturbation resampling algorithm gives a valid approximation to the sampling distribution of the cross-validated estimator and that it captures every source of variability, including uncertainty in the fitted coefficients, the censoring weights, the estimated quantiles, and the random cross-validation splits.

Editorial extensions

If this is right

  • Users can read from the estimated curve the expected cumulative incidence at any score quantile, and from the inverse curve the fraction of the population falling below a clinically chosen risk threshold.
  • The evaluation remains meaningful when the Fine-Gray model is misspecified, because the method treats the model as a working model and only requires the estimated coefficients to settle down to fixed limits.
  • Wald confidence intervals built from perturbation-based standard errors give approximately nominal coverage for samples of a few hundred under independent right censoring, based on the simulation evidence.
  • The method is implemented in an R package, so competing-risks predictiveness curves can be produced for new prediction models without custom code.
  • Comparing two risk scores on the same quantile scale directly shows which one separates high- and low-risk patients more steeply.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: because no asymptotic theorem is given for the perturbation scheme, its validity should be re-checked under heavier censoring or strong covariate effects before clinical use.
  • My inference: the method could be extended to a two-dimensional surface R(v, tau) over multiple prediction horizons, which the discussion names as future work rather than a claim of this paper.
  • My inference: the role of tuning choices (number of knots, number of CV repetitions) is not fully mapped; the simulation pattern suggests a linear fit may be preferable at n = 200, but this is an interpolation by this reader, not a stated conclusion.
  • My inference: if the perturbation scheme is valid, it also provides a ready way to test differences between two predictiveness curves, since perturbed replicates can be paired.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper develops estimation and inference procedures for the predictiveness curve in the presence of competing risks, where the risk score is generated from a working Fine–Gray model. The proposed estimator uses repeated two-fold cross-validation: in each split, a Fine–Gray model is fit on the training part, a risk score is computed on the test part, and the cumulative incidence probability at a given quantile of the score is estimated via an IPCW-weighted flexible binomial regression (restricted cubic splines or a simple logistic term). Variance estimation is carried out by perturbation resampling (Algorithm 1). The methods are evaluated in two simulation settings (one correctly specified, one misspecified) at sample sizes 200, 400, and 800, and applied to a cirrhosis cohort for 5-year liver-related mortality. An R package cmpCurve is provided.

Significance. If the proposed inference is valid, the paper fills a genuine gap: predictiveness curves for competing risks outcomes with model-based risk scores, explicitly allowing the prediction model to be misspecified. The simulation design includes a misspecified working model and the results for the RCS estimator show small bias and mostly near-nominal coverage, which is encouraging. The paper delivers a practical R package and a real-data application, making the method accessible. The central limitation is that the variance estimator's validity is asserted rather than proven, and the simulation summary overstates the performance of the GLM variant. These issues are load-bearing for the paper's main claims about inference, though they are fixable.

major comments (3)
  1. [Section 2.3 and Algorithm 1] The variance estimation procedure is not justified theoretically. The paper states that perturbation resampling 'accounts for all sources of variance comprehensively' and cites Jin et al. (2001) and Ding et al. (2021), but neither reference covers an estimator that is an average over R random two-fold splits of a two-stage procedure with a first-stage Fine–Gray fit, a second-stage IPCW-weighted spline regression, and an estimated quantile. Algorithm 1 does not specify whether the cross-validation splits are fixed across perturbation replications or re-randomized within each e, and no argument is given for why the sample variance of the perturbed estimates has the same asymptotic covariance as the repeated-CV estimator when R is small (R = 5 in the simulations). This is load-bearing for every confidence interval reported in the paper; the simulation coverage provides empirical support but does not replace a proof or a precise citation to a theorem that covers this setting.
  2. [Section 3.2 and Table 1] The text claims that 'bias in estimates across all sample sizes was minimal' and that coverage probabilities are 'close to the nominal level in most scenarios,' but this is contradicted by the GLM results in Setting 2. In Table 1, the GLM estimator at n = 800 shows biases of 0.034 (v = 0.1), –0.026 (v = 0.5), and –0.026 (v = 0.7), with coverage probabilities 0.729, 0.724, and 0.802, respectively. These are far from nominal and do not disappear with increasing sample size. The recommendation that 'a basic parameterization may be adequate for small sample sizes' is also questionable given the coverage of 0.934 at n = 200 (v = 0.1). The simulation summary should either restrict the satisfactory-performance claim to the RCS estimator or acknowledge the GLM's persistent bias and undercoverage.
  3. [Section 3.1] The 'true' predictiveness curve for Setting 2 is not an analytic quantity but is approximated by first estimating the limiting Fine–Gray coefficient from a 'huge dataset' and then averaging over many uncensored datasets. The paper does not report the size of these datasets, the number of repetitions, or the Monte Carlo error of the approximation. Since the simulation bias evaluations in Table 1 and Table 2 for Setting 2 compare estimates against this approximate benchmark, the accuracy of the benchmark directly affects the interpretation of the reported biases. The paper should provide these details or explicitly label the Setting 2 curve as an approximation with a stated precision.
minor comments (6)
  1. [Section 2.1] The notation for the estimated regression coefficient is inconsistent: the text uses bβ in the Introduction and Section 2.2 but eβ for the probability limit; please clarify whether bβ denotes the estimate and eβ the limit, and use the two symbols consistently throughout.
  2. [Section 2.2, Equation (4)] The objective function in equation (4) is written in a way that is difficult to parse; the exponents δi and 1−δi appear to apply to the two terms of the Bernoulli likelihood, but the surrounding text and the placement of braces make this unclear. Please rewrite the equation with explicit brackets.
  3. [Algorithm 1, Step (c)] The weighted quantile computation is not fully specified: it is not stated whether the perturbation weights are used only for the test-set observations, how ties in the risk score are handled, or what quantile definition (e.g., type 7 in R) is used. This matters for reproducibility of the variance estimates.
  4. [Section 2.2] The phrase 'Without loss of generality, we opted for a two-fold repeated cross-validation strategy' is misleading, since the choice of two folds is not without loss of generality; other split proportions (e.g., 3:1 or 4:1) would change the bias-variance tradeoff. Please rephrase.
  5. [Section 4] The application uses 30 cross-validation repetitions, while the simulations use 5. The paper should briefly justify this choice or note that the number of repetitions is a user-specified tuning parameter.
  6. [Tables 1 and 2] The 'True R(v)' and 'TRUE' rows for Setting 2 are not marked as approximate; please add a footnote stating that these values are Monte Carlo approximations and report their precision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the predictiveness curve is a directly estimated population quantity, and the perturbation-resampling gap is an unproved assumption, not a self-referential reduction.

full rationale

The paper's central estimand, R(v)=P(T≤τ,ε=1|ξ=Q(v)), is a population quantity defined through the working-model risk score ξ=Z^T eβ, not through the estimator itself. The estimation procedure is a direct plug-in construction: estimate β on the training fold, form scores on the test fold, fit an IPCW-weighted RCS binomial model for the CIF conditional on the score, and evaluate at the empirical quantile. This is a standard regression/quantile estimation procedure rather than a self-definitional one, because the target is not defined in terms of the fitted spline coefficients and the fitted model is not used to define the estimand. The repeated two-fold cross-validation evaluates the working model out-of-sample, and the simulation 'true' curves are computed analytically (Setting 1) or by large-sample Monte Carlo approximation of the same population quantity (Setting 2), so the simulation comparison is a consistency check, not a fitted-input prediction. The only self-citations (Ding et al. 2021) support two ancillary facts: convergence of the Fine-Gray estimator under misspecification and the general perturbation-resampling technique. Neither fact is the paper's conclusion, and neither is defined in terms of the predictiveness curve. The absence of a theorem showing that perturbation resampling remains valid for the repeated-CV two-stage estimator (Section 2.3, Algorithm 1) is a genuine gap in asymptotic justification, but it is an unproved assumption about the resampling distribution, not a circular reduction: the variance estimator is not an input to the point estimator, and the paper's simulations provide external falsifiable evidence. No derivation step is equivalent to its own inputs by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The ledger is dominated by estimation and tuning choices: beta and theta are fitted to data; Q, [nu0,nu1], CV repetitions, and E are user-chosen. The main domain assumptions are conditional independent censoring, adequacy of spline approximation, and unproved validity of perturbation resampling for the cross-validated estimator. No new physical or conceptual entities are introduced.

free parameters (5)
  • Fine-Gray working model coefficients beta = estimated from training folds (values vary by data)
    Defines the risk score xi = Z^T beta; estimated in DA and used in DB. In simulation beta has known true values; in application values are estimated from data.
  • Binomial RCS coefficients theta = estimated from test-fold data via weighted objective (4)
    Map the risk score to the cumulative incidence curve; these coefficients are fit on DB, so they are data-fitted parameters of the predictiveness curve estimator.
  • Number of RCS knots Q = not reported; paper recommends 3-5
    User-chosen tuning parameter controlling spline flexibility; the paper fixes this in advance without a data-driven selection rule.
  • Quantile trimming range [nu0, nu1] = not specified, described as constants close to 0 and 1
    Estimates the curve only on interior quantiles to avoid boundary instability; the exact values are not reported.
  • Number of CV repetitions and perturbation samples = 5 in simulation, 30 in application; E=400
    User-chosen implementation settings that affect Monte Carlo and cross-validation variability.
assumptions (5)
  • standard math The i.i.d. data and standard regular asymptotics of the Fine-Gray subdistribution hazard estimator hold.
    Background for all estimation; invoked implicitly throughout Sections 2.2-2.3.
  • domain assumption Under model misspecification, the Fine-Gray coefficient estimate converges to a deterministic limit ebeta, and risk scores are bounded uniformly in Z.
    Section 2.1 relies on Ding et al. (2021) to define the target curve using Z^T ebeta.
  • domain assumption Censoring is conditionally independent of event time given covariates, and the censoring distribution G(t|Z) is correctly estimated for the IPCW weights in Eq (4).
    Required for the weighted log-likelihood to identify the CIF given the risk score.
  • domain assumption The flexible binomial model with restricted cubic splines adequately approximates the true conditional CIF P(T <= tau, epsilon=1 | xi) at the chosen horizon.
    Section 2.2 states 3-5 knots are typically sufficient; no formal test of the approximation is given.
  • domain assumption Perturbation resampling is asymptotically valid for the repeated cross-validation two-stage estimator.
    Algorithm 1 assumes the perturbed distribution approximates the sampling distribution; no theorem extends Jin et al. (2001) to this CV setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Predictiveness Curve Assessment under Competing Risks for Risk Prediction Models." pith.science (2026). https://pith.science/paper/CNOS7XXA

@misc{pith2026250800216,
  author       = {Pith},
  title        = {Pith review of: Predictiveness Curve Assessment under Competing Risks for Risk Prediction Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CNOS7XXA}},
  note         = {Machine review of arXiv:2508.00216}
}
read the original abstract

The predictiveness curve is a valuable tool for predictive evaluation, risk stratification, and threshold selection in a target population, given a single biomarker or a prediction model. In the presence of competing risks, regression models are often used to generate predictive risk scores or probabilistic predictions targeting the cumulative incidence function--distinct from the cumulative distribution function used in conventional predictiveness curve analyses. We propose estimation and inference procedures for the predictiveness curve with a competing risks regression model, to display the relationship between the cumulative incidence probability and the quantiles of model-based predictions. The estimation procedure combines cross-validation with a flexible regression model for tau-year event risk given the model-based risk score, with corresponding inference procedures via perturbation resampling. The proposed methods perform satisfactorily in simulation studies and are implemented through an R package. We apply the proposed methods to a cirrhosis study to depict the predictiveness curve with model-based predictions for liver-related mortality.

Figures

Figures reproduced from arXiv: 2508.00216 by the authors.

Figure 1
Figure 1. Illustrative Example of the Predictiveness Curve [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. True Predictiveness Curves under Simulation Settings 1 & 2. [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Rˆ(v) curve and Rˆ−1 (p) curve in data application. slight variations. In [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 16 canonical work pages

  1. [1]

    Kosorok, Ludger Rüschendorf, and Jason P

    Anna Bellach, Michael R. Kosorok, Ludger Rüschendorf, and Jason P. Fine. Weighted NPMLE for the Subdistribution of a Competing Risk . Journal of the American Statistical Association, 114 0 (525): 0 259--270, 2019. ISSN 0162-1459. doi:10.1080/01621459.2017.1401540

    arXiv 2019

  2. [2]

    The binary regression quantile plot: assessing the importance of predictors in binary regression visually

    Efstathia Bura and Joseph L Gastwirth. The binary regression quantile plot: assessing the importance of predictors in binary regression visually. Biometrical Journal: Journal of Mathematical Methods in Biosciences, 43 0 (1): 0 5--21, 2001

    work page 2001

  3. [3]

    Evaluation of competing risks prediction models using polytomous discrimination index

    Maomao Ding, Jing Ning, and Ruosha Li. Evaluation of competing risks prediction models using polytomous discrimination index. Canadian Journal of Statistics, 49 0 (3): 0 731--753, September 2021. ISSN 0319-5724, 1708-945X. doi:10.1002/cjs.11583. URL https://onlinelibrary.wiley.com/doi/10.1002/cjs.11583

    work page doi:10.1002/cjs.11583 2021

  4. [4]

    Copula modeling of receiver operating characteristic and predictiveness curves

    Gabriel Escarela, Carlos Erwin Rodríguez, and Gabriel Núñez‐Antonio. Copula modeling of receiver operating characteristic and predictiveness curves. Statistics in Medicine, 39 0 (28): 0 4252--4266, December 2020. ISSN 0277-6715, 1097-0258. doi:10.1002/sim.8723. URL https://onlinelibrary.wiley.com/doi/10.1002/sim.8723

    work page doi:10.1002/sim.8723 2020

  5. [5]

    Copula modeling for the estimation of measures of marker classification and predictiveness performance with survival outcomes

    Gabriel Escarela, Alejandro Rom \'a n V \'a squez, Graciela Gonz \'a lez-Far \' as, and Jos \'e Ulises M \'a rquez-Urbina. Copula modeling for the estimation of measures of marker classification and predictiveness performance with survival outcomes. Statistical Methods in Medical Research, 32 0 (6): 0 1203--1216, 2023

    work page 2023

  6. [6]

    Fine and Robert J

    Jason P. Fine and Robert J. Gray. A Proportional Hazards Model for the Subdistribution of a Competing Risk . Journal of the American Statistical Association, 94 0 (446): 0 496--509, June 1999. ISSN 0162-1459, 1537-274X. doi:10.1080/01621459.1999.10474144. URL http://www.tandfonline.com/doi/abs/10.1080/01621459.1999.10474144

    arXiv 1999

  7. [7]

    Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis, 2001

    FE Harrell. Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis, 2001

    work page 2001

  8. [8]

    rms: Regression Modeling Strategies, 2023

    Frank E Harrell Jr . rms: Regression Modeling Strategies, 2023. URL https://CRAN.R-project.org/package=rms. R package version 6.7-0

    work page 2023

Show all 20 references
  1. [9]

    Evaluating the Predictiveness of a Continuous Marker

    Ying Huang, Margaret Sullivan Pepe, and Ziding Feng. Evaluating the Predictiveness of a Continuous Marker . Biometrics, 63 0 (4): 0 1181--1188, December 2007. ISSN 0006341X. doi:10.1111/j.1541-0420.2007.00814.x. URL https://onlinelibrary.wiley.com/doi/10.1111/j.1541-0420.2007.00814.x

    2007

  2. [10]

    Direct parametric inference for the cumulative incidence function

    Jong-Hyeon Jeong and Jason Fine. Direct parametric inference for the cumulative incidence function. Journal of the Royal Statistical Society Series C: Applied Statistics, 55 0 (2): 0 187--200, 2006

    2006

  3. [11]

    A simple resampling method by perturbing the minimand

    Zhezhen Jin, Zhiliang Ying, and LJ Wei. A simple resampling method by perturbing the minimand. Biometrika, 88 0 (2): 0 381--390, 2001

    2001

  4. [12]

    Klein and Per Kragh Andersen

    John P. Klein and Per Kragh Andersen. Regression modeling of competing risks data based on pseudovalues of the cumulative incidence function. Biometrics, 61 0 (1): 0 223--229, March 2005. ISSN 0006-341X. doi:10.1111/j.0006-341X.2005.031209.x

    2005 arXiv

  5. [13]

    Estimating Subject - Specific Dependent Competing Risk Profile with Censored Event Time Observations

    Yi Li, Lu Tian, and Lee-Jen Wei. Estimating Subject - Specific Dependent Competing Risk Profile with Censored Event Time Observations . Biometrics, 67 0 (2): 0 427--435, June 2011. ISSN 0006341X. doi:10.1111/j.1541-0420.2010.01456.x. URL https://onlinelibrary.wiley.com/doi/10....

    2011

  6. [14]

    C. S. Moskowitz and M. S. Pepe. Quantifying and comparing the predictive accuracy of continuous prognostic factors for binary outcomes. Biostatistics, 5 0 (1): 0 113--127, January 2004. ISSN 1465-4644, 1468-4357. doi:10.1093/biostatistics/5.1.113. URL https://academic.oup.com/...

    2004 doi

  7. [15]

    M. S. Pepe, Z. Feng, Y. Huang, G. Longton, R. Prentice, I. M. Thompson, and Y. Zheng. Integrating the Predictiveness of a Marker with Its Performance as a Classifier . American Journal of Epidemiology, 167 0 (3): 0 362--368, November 2007. ISSN 0002-9262, 1476-6256. doi:10.109...

    2007 doi

  8. [16]

    Sachs and Xiao‐Hua Zhou

    Michael C. Sachs and Xiao‐Hua Zhou. Partial summary measures of the predictiveness curve. Biometrical Journal, 55 0 (4): 0 589--602, July 2013. ISSN 0323-3847, 1521-4036. doi:10.1002/bimj.201200146. URL https://onlinelibrary.wiley.com/doi/10.1002/bimj.201200146

    2013 doi

  9. [17]

    Predicting cumulative incidence probability by direct binomial regression

    Thomas H Scheike, Mei-Jie Zhang, and Thomas A Gerds. Predicting cumulative incidence probability by direct binomial regression. Biometrika, 95 0 (1): 0 205--220, 2008

    2008

  10. [18]

    [generalized additive models]: comment

    Charles J Stone. [generalized additive models]: comment. Statistical Science, 1 0 (3): 0 312--314, 1986

    1986

  11. [19]

    Discrimination measures for survival outcomes: connection between the auc and the predictiveness curve

    Vivian Viallon and Aur \'e lien Latouche. Discrimination measures for survival outcomes: connection between the auc and the predictiveness curve. Biometrical Journal, 53 0 (2): 0 217--236, 2011

    2011

  12. [20]

    Mortality after transjugular intrahepatic portosystemic shunt in older adult patients with cirrhosis: A validated prediction model

    Francesco Vizzutti, Ciro Celsa, Vincenza Calvaruso, Marco Enea, Salvatore Battaglia, Laura Turco, Marco Senzolo, Silvia Nardelli, Roberto Miraglia, Davide Roccarina, Claudia Campani, Dario Saltini, Cristian Caporali, Federica Indulti, Stefano Gitto, Alberto Zanetto, Gabriele D...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.