{"id":"29b82cb1-89d2-4d7b-ade6-4d4ed06ccf06","arxiv_id":"2505.01145","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Surrogate SHAP fits an XGBoost model to estimated CATEs and uses TreeSHAP to rank predictive biomarkers, with simulations favoring S-learning in RCTs and R/DR-learning in observational settings.","lead":"This paper proposes and benchmarks a practical 'surrogate SHAP' approach: fit an XGBoost model to predicted treatment-effect estimates from any CATE method, then use its SHAP values to rank which baseline biomarkers drive treatment effect heterogeneity. In simulations, the surrogate approach is computationally efficient, and the choice of CATE learner strongly changes biomarker discovery, with S-learning best in RCTs and R/DR-learning best in observational settings.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Surrogate SHAP fidelity to the CATE estimator is asserted, not directly tested for irreducible learners; the benchmark's cross-learner conclusions depend on this untested link.","rationale":"The reader's weakest assumption identified the same load-bearing concern: the surrogate model must preserve the covariate-importance structure of the underlying CATE estimator. My stress-test sharpens this in two ways. First, the existing evidence is limited: Section 4.6 only covers reducible learners (R, DR), and Section 3.2 is a single toy illustration for T-learning with five covariates, not the benchmark setting. Second, the in-sample fitting of both the CATE estimator and the surrogate means that TreeSHAP can capitalize on noise in tau_hat, making rankings potentially sensitive to surrogate overfitting rather than to genuine biomarker signal. This is not an external-consensus objection; it is an internal-validity question about whether the paper's comparative simulations measure what they claim. The paper itself acknowledges that SHAP explains the model, not the truth, but it does not provide the direct fidelity check that would close the gap. I do not regard this as grounds for rejection: the paper is a tutorial with a pragmatic protocol, and the surrogate approach may well be adequate in practice. The conditional verdict already given is appropriate, pending the direct comparison described above. I agree with the reader's assessment and see no reason to change the verdict; the concrete test would strengthen the paper and could be reported in a revision.","tokens_in":35787,"tokens_out":6671,"duration_ms":74720,"concrete_test":"Run the S2 benchmark (n=1000, 100 iterations) with XGBoost base learners and, for each learner, compute direct exact TreeSHAP on tau_hat instead of on the surrogate: for T-learner use phi(f1)-phi(f0); for S-learner use TreeSHAP on the treatment-fixed ensemble; for X-learner in the RCT setting use the constant-weight linear combination of the two component SHAPs; for R- and DR-learners use TreeSHAP on the final pseudo-outcome model (Strategy 2). Compare TOP1, NET3, and MARGIN between direct SHAP and surrogate SHAP for each learner. If any learner's TOP1 changes by more than 0.1, the surrogate does not preserve the learner's importance structure, and the benchmark conclusions cannot be attributed to CATE learner choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that Strategy 3 gives a valid, unifying SHAP-based biomarker ranking for any CATE learner—requires that the XGBoost surrogate M(2) preserves the covariate-importance structure of tau_hat. The paper tests this only indirectly: Section 4.6 compares Strategy 2 against Strategy 3 for R- and DR-learners (reducible learners), and Section 3.2 gives a single T-learner toy with p=5. No direct check is reported for S-, T-, and X-learners in the actual high-dimensional benchmark (p=19–20). In that benchmark, the surrogate is fitted in-sample to the same data used to estimate CATE, and TreeSHAP is computed on the same training data. Because XGBoost can overfit noise in tau_hat, the resulting SHAP rankings could reflect surrogate artifacts rather than the biomarker-discovery ability of the underlying learner. If so, the paper's headline comparative conclusions (e.g., S-learner best in RCTs, T-learner poor, Causal Forest poor) would be uninterpretable as statements about CATE learners. The paper itself flags the related limitation in Section 6—'the model itself can be an arbitrarily bad estimator of CATE'—but it never checks whether M(2) is a faithful estimator of tau_hat. This is the load-bearing gap in the central argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops and benchmarks a 'surrogate SHAP' strategy for identifying predictive biomarkers from CATE models. After reviewing Shapley values, SHAP, and meta-learners, it proposes fitting an XGBoost model to estimated CATEs and deriving TreeSHAP values from that surrogate (Strategy 3), claiming this is a unified, computationally feasible route for any CATE estimator. Simulations under an RCT model (S2) and an observational model (S3) compare T-, S-, X-, R-, DR-learners and Causal Forest using TOP1, NET3, and MARGIN metrics; findings include S-learner best in RCTs, R/DR best in observational data, and poor performance of T-learner and Causal Forest. Additional analyses examine prognostic strength, Strategy 2 versus Strategy 3 for R/DR learners, local SHAP recovery of true effect-modifier functions, and two case studies.","tokens_in":36118,"tokens_out":7599,"duration_ms":70767,"significance":"If the surrogate strategy is valid, it gives practitioners a single protocol for biomarker ranking across heterogeneous CATE learners and avoids the computational burden of KernelSHAP. The paper's strengths include carefully described simulation setups with 100 to 500 iterations and standard errors, reproducible R code, a head-to-head comparison of SHAP against Causal Forest VIP, and falsifiable empirical claims about which learners rank predictive biomarkers well. The central caveat is that the surrogate's fidelity to the underlying CATE learner is asserted rather than directly demonstrated for irreducible learners; the cross-learner conclusions rest on that unverified link.","major_comments":[{"comment":"The central claim of the paper, that Strategy 3 provides a valid unified SHAP-based biomarker ranking for any CATE learner, requires that the XGBoost surrogate M(2) preserve the covariate-importance structure of tau_hat. The manuscript verifies this only for the reducible R- and DR-learners (Section 4.6) and in a single T-learner toy with p=5 (Section 3.2); no direct fidelity check is reported for S-, T-, X-, or Causal Forest learners in the p=19-20 benchmark of Section 4.3. Since M(2) is fitted in-sample and TreeSHAP is computed on the same training data, overfitting of noise in tau_hat could make the resulting rankings surrogate artifacts rather than properties of the underlying learner. I therefore recommend adding a validation experiment that, for each learner, reports the out-of-sample fidelity of M(2) to tau_hat (e.g., held-out R^2 or rank correlation between M(2) and tau_hat), a small-p (p <= 8) comparison of Strategy 3 SHAP rankings with direct strategy-appropriate SHAP rankings from the learner itself, and sensitivity of TOP1, NET3, and MARGIN to XGBoost hyperparameter choices. Without such a check, the headline conclusions about learner choice remain statements about the surrogate pipeline.","section":"Section 2.4 (Strategy 3); Section 4.3, Table 3"},{"comment":"The claim that 'no empirical evidence was found to indicate a statistical difference in performance between the two types of SHAPs' is currently supported only by visual inspection of boxplots; no formal test or equivalence margin is reported. Because this Strategy 2 versus Strategy 3 comparison is the main direct evidence for surrogate validity, and it covers only reducible learners, I ask the authors either to add a formal equivalence analysis (e.g., confidence intervals for the difference in means with a pre-specified margin) or to soften the conclusion to 'no material difference was observed in these simulations.'","section":"Section 4.6, Figure 8"},{"comment":"The instance-level analysis computes correlations between surrogate SHAP values and tau_hat on the same data used to fit both the CATE models and the surrogate. As a result, high values of cor(phi_j, tau_hat) may reflect in-sample overfitting rather than a genuine local biomarker signal, and the P_win metric inherits this problem. Reporting cross-validated or out-of-sample SHAP correlations, or at least the stability of the rankings across refits, would make the local-inference claims more convincing.","section":"Section 4.7, Table 6"}],"minor_comments":[{"comment":"The Discussion states that X-learner fits a single final model allowing direct SHAP, while Section 2.3.2 classifies X-learner as an irreducible meta-learner; these descriptions should be reconciled.","section":"Section 2.3.2 and Section 6"},{"comment":"The sentence 'For small numbers of covariates p, i.e. > 8−15' is self-contradictory; likely '<8-15' or 'up to 8-15' was intended.","section":"Section 3.3"},{"comment":"The text in Section 4.3 refers to Table 3 and Figure 5 for the main metrics, then separately to Figure 6 for the correlation analysis; the cross-references should be tightened so readers do not confuse the two displays.","section":"Section 4.3, Table 3 and Figure 6"},{"comment":"The text refers to 'the five truly prognostic covariates,' but k1(x) contains seven distinct prognostic covariates (x1, x2, and x5-x9, with x2 categorical); the count should be corrected.","section":"Section 4.4"},{"comment":"The computational-cost table is labeled 'Table 1,' duplicating the notation of the main-text Table 1; renumber it (e.g., Table A1).","section":"Appendix"},{"comment":"The sentence 'This closely replicates the ranking from earlier investigations, where VIPs from Causal Forest preferred Age over Pansspos' would be clearer if 'earlier investigations' named the reference explicitly in that sentence.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The surrogate-fidelity gap is the main obstacle, but it is fixable within the scope of the manuscript. The paper is a solid tutorial and benchmarking contribution with clearly described simulations, and I would be comfortable with acceptance after the proposed validation experiments and minor revisions are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a tutorial-plus-benchmark from pharma statisticians (AstraZeneca, Novartis, Lilly, UCL). The main contribution is Strategy 3: fit an XGBoost surrogate to CATE estimates from any learner, then TreeSHAP from that surrogate. That gives a unified, computationally feasible route to SHAP-based biomarker rankings. The benchmark itself is new—six CATE learners, two simulation models (RCT and observational), three metrics (TOP1, NET3, MARGIN), 100–500 iterations with standard errors. The finding that learner choice matters a lot, and that T-learner and Causal Forest can rank worse than random, is practically important.\n\nWhat is genuinely good: the simulations are detailed and reproducible in design (though code is not shipped), the metrics are sensible, and the two case studies illustrate the workflow on realistic data. The comparison of surrogate SHAP against direct SHAP for R/DR-learners (Strategy 2 vs 3) is the right check, and the result—no empirical difference—supports the surrogate for those learners. The computational argument against KernelSHAP is quantified, which helps.\n\nSoft spots, in order of seriousness. First, the load-bearing assumption: the surrogate preserves the covariate-importance structure of the underlying CATE estimator. That is tested for R/DR (reducible) and in one small T-learner toy, but not directly for T/S/X in the actual p≈20 benchmark. The headline cross-learner comparisons (S-learner best in RCT, T-learner poor, CF poor) are all filtered through the surrogate, so if the surrogate distorts rankings differently across learners, those conclusions get muddy. The paper acknowledges the model-can-be-bad limitation but never checks surrogate fidelity. This is a real gap, though not fatal—the rankings are still about the surrogate+learner pipeline, which is what a practitioner would actually use.\n\nSecond, novelty is overstated. The surrogate idea is Virtual Twins (Foster 2011) plus Man et al.'s suggestion to use tree ensembles in the second stage. The paper cites both but calls the approach novel and unifying. What is actually new is the systematic SHAP-based benchmarking, not the surrogate concept.\n\nThird, hyperparameters are described as extensively tuned, but the grids are not fully specified, and no code is released. Given the tutorial framing, that is a fixable omission.\n\nWho is this for? Pharmaceutical biostatisticians and applied researchers who want guidance on ranking predictive biomarkers with SHAP. It deserves a serious referee; the central protocol is sensible and the benchmark is informative. I would accept for review with a request for code, surrogate-fidelity checks, and a toned-down novelty claim.","headline":"A practical, well-benchmarked tutorial on surrogate SHAP for predictive biomarker discovery, with a real but manageable gap: the surrogate's fidelity is asserted for irreducible learners, not directly tested.","tokens_in":36584,"tokens_out":1559,"would_cite":false,"duration_ms":17260,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes and benchmarks a surrogate-model strategy: fit an XGBoost model to the estimated conditional average treatment effect and read SHAP values off that model, giving one protocol for ranking predictive biomarkers that works…","keywords":["treatment effect heterogeneity","conditional average treatment effect","predictive biomarkers","prognostic biomarkers","SHAP values","meta-learners","causal forest","surrogate model"],"falsifier":"A concrete check would be a simulation with a known CATE function and a deliberately misspecified or underpowered CATE estimator, comparing surrogate SHAP rankings with oracle rankings from the true $\\tau(x)$: if the surrogate ranks a known null covariate above a true modifier in a large-n setting more often than chance, the preservation premise fails.","tokens_in":35625,"feed_emoji":"🧬","tokens_out":4707,"duration_ms":41717,"temperature":0.7,"pith_summary":"This paper tries to establish a single, practical protocol for turning any CATE estimator into a SHAP-based ranking of predictive biomarkers. The proposed protocol fits an XGBoost surrogate model to the estimated CATE and derives TreeSHAP values from that surrogate, sidestepping the computational blow-up of model-agnostic SHAP and the ambiguity of where to apply SHAP inside multi-stage meta-learners. Through simulations the paper shows the protocol works across T-, S-, X-, R-, DR-learners and Causal Forest, and that the choice of learner materially changes discovery accuracy. The authors also show that global rankings and local SHAP profiles can be used, with caveats, to recover the shape and direction of true treatment-effect modifiers.","feed_headline":"Surrogate SHAP model ranks predictive biomarkers for any CATE learner","feed_subtitle":"Fitting XGBoost to CATE estimates makes biomarker ranking practical, but learner choice shifts accuracy.","key_machinery":"The central object is the surrogate model $M^{(2)}$: a second-stage XGBoost model regressing the estimated CATE, $\\hat{\\tau}(x)$, on all baseline covariates $x_1,\\ldots,x_p$, from which exact TreeSHAP values are computed. It carries the argument by converting any multi-stage CATE estimator—reducible or irreducible—into a single supervised model whose SHAP values are well-defined and cheap to compute, bypassing the exponential coalition enumeration of model-agnostic KernelSHAP.","core_discovery":"The central claim is that the surrogate approach—fitting an XGBoost model to CATE estimates against all baseline covariates and deriving SHAP values from that model—provides a unified, computationally feasible route to SHAP-based biomarker discovery for any CATE estimator. The paper argues this is valid because the surrogate explains a model that represents estimated CATE as a function of covariates, and it verifies empirically that for reducible learners like R- and DR-learning the surrogate performs no worse than deriving SHAP directly from the final pseudo-outcome regression. Simulation benchmarks then show that the choice of CATE learner strongly affects biomarker discovery: in randomized trials the S-learner performs best, while in observational settings the R-learner dominates; T-learning and Causal Forest perform poorly and can fall below random guessing when prognostic strength is high.","pith_inferences":["The surrogate's validity rests on the surrogate faithfully preserving the CATE estimator's covariate-importance structure; a direct check would be to compare surrogate SHAP rankings against oracle rankings for irreducible learners across varied data-generating processes, which the paper only partially covers.","The protocol could be extended to loss-based or interaction-aware SHAP definitions, which would address the paper's own caveat that SHAP explains the model, not the truth.","Because the paper fixes the surrogate to XGBoost, the protocol's performance may depend on that choice; testing alternative tree ensembles or other surrogates would clarify how much of the result is the surrogate versus the underlying CATE estimator.","The observed failure of T-learning and Causal Forest suggests a diagnostic opportunity: a large gap between surrogate SHAP margins and CATE estimation accuracy could flag learners that confound prognostic with predictive signal."],"forward_implications":["A single SHAP protocol can replace method-specific importance measures across T-, S-, X-, R-, DR-learners and Causal Forest.","Practitioners selecting a CATE learner for biomarker discovery should expect different rankings: S-learner in RCTs, R-learner in observational data.","Model-agnostic KernelSHAP becomes impractical beyond about 8–15 covariates, while TreeSHAP on the surrogate remains feasible; the surrogate avoids this bottleneck.","For R- and DR-learners, deriving SHAP from the surrogate matches the direct pseudo-outcome approach, so the extra modeling step costs no discovery accuracy.","Global SHAP rankings and local SHAP profiles can recover qualitative features of true effect modifiers, but inspecting individual profiles risks false discoveries unless a well-performing learner is used."],"supporting_citations":[{"why":"Supplies the source idea: fitting a model on top of the estimated CATE (Virtual Twins) to identify subgroups, which Strategy 3 generalizes.","marker":"26"},{"why":"Recommends replacing the CART second-stage with any tree ensemble to derive variable importance, grounding the surrogate's choice of XGBoost.","marker":"27"},{"why":"Defines SHAP and the additive attribution framework that the paper uses to score biomarkers.","marker":"21"},{"why":"Provides TreeSHAP, the exact polynomial-time SHAP algorithm for tree ensembles that makes the surrogate strategy computationally feasible.","marker":"56"},{"why":"Defines the simulation scenarios S2 and S3 and the benchmarking setup that the paper adapts to evaluate biomarker discovery.","marker":"6"},{"why":"Documents how prognostic covariates mislead Virtual Twins and Causal Forest, motivating the prognostic-strength simulation.","marker":"39"},{"why":"Introduces the meta-learner taxonomy (T-, S-, X-learners) and known failure modes that the paper's recommendations build on.","marker":"8"},{"why":"TE-VIM benchmark that acknowledges SHAP's computational difficulty; the paper's surrogate is positioned against it.","marker":"25"}],"fun_headline_variants":["Surrogate SHAP unifies biomarker ranking for any CATE learner","Learner choice decides SHAP biomarker ranking in CATE models","XGBoost surrogate makes SHAP biomarker ranking practical for CATE","Surrogate SHAP: learner choice drives biomarker ranking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the surrogate XGBoost model fitted to the estimated CATE preserves the covariate-importance structure of the underlying CATE estimator, so that TreeSHAP values from the surrogate are valid proxies for true predictive biomarker effects; the paper states SHAP explains the model, not the truth, and does not directly verify the premise for irreducible learners.","fun_headline_variants_meta":{"raw":{"variants":["Surrogate SHAP unifies biomarker ranking for any CATE learner","Learner choice decides SHAP biomarker ranking in CATE models","XGBoost surrogate makes SHAP biomarker ranking practical for CATE","Surrogate SHAP: learner choice drives biomarker ranking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000772,"raw_usage":{"total_tokens":3409,"prompt_tokens":930,"completion_tokens":2479,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":2407}},"tokens_in":546,"tokens_out":2479,"duration_ms":18348,"temperature":1.0,"reasoning_tokens":2407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:24:52.273615+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be a simulation with a known CATE function and a deliberately misspecified or underpowered CATE estimator, comparing surrogate SHAP rankings with oracle rankings from the true $\\tau(x)$: if the surrogate ranks a known null covariate above a true modifier in a large-n setting more often than chance, the preservation premise fails.","supporting_citations":[{"cited_title":"Subgroup identification from randomized clinical trial data.Statistics in Medicine2011; 30(24): 2867–2880","cited_arxiv_id":null,"evidence_quote":"Supplies the source idea: fitting a model on top of the estimated CATE (Virtual Twins) to identify subgroups, which Strategy 3 generalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Recommends replacing the CART second-stage with any tree ensemble to derive variable importance, grounding the surrogate's choice of XGBoost."},{"cited_title":"A unified approach to interpreting model predictions..Advances in Neural Information Processing Systems2017; 30","cited_arxiv_id":null,"evidence_quote":"Defines SHAP and the additive attribution framework that the paper uses to score biomarkers."},{"cited_title":"Nature machine intelligence2020; 2(1): 56–67","cited_arxiv_id":null,"evidence_quote":"Provides TreeSHAP, the exact polynomial-time SHAP algorithm for tree ensembles that makes the surrogate strategy computationally feasible."},{"cited_title":"On discovering treatment-effect modifiers using Virtual Twins and Causal Forest ML in the presence of prognostic biomarkers","cited_arxiv_id":null,"evidence_quote":"Documents how prognostic covariates mislead Virtual Twins and Causal Forest, motivating the prognostic-strength simulation."}],"review_version":1}