{"id":"f9583193-7b84-4fb9-b6f6-279a804f696f","arxiv_id":"2412.20638","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A soft-surrogate estimator predicts a sequential policy's long-term value from short-horizon on-policy data and full-horizon historical data, with finite-sample guarantees.","lead":"This paper presents a method for estimating the long-term value of a new treatment policy using only a short period of data from that policy plus longer historical records. It matters because it could help researchers decide sooner whether new drugs or protocols are worth continuing, before multi-year trials finish.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported p-values <10^{-6} are invalid because the t-test treats fitted model predictions as observed returns, so the claim that 10% horizon data suffices to detect policy improvement is unsupported.","rationale":"The reader's verdict is CONDITIONAL, with the most serious issue being the invalid significance testing. My read agrees with that assessment and identifies it as the single most load-bearing concern because it directly undercuts a quantified empirical claim that the authors emphasize (p-values <10^{-6}). The reader's weakest_assumption field names Assumption 2 (soft surrogacy), which is a serious and untestable assumption, but it is explicitly conditional: the theoretical results hold only under Assumption 2, and the authors openly state that the assumption likely fails in the experiments and still report accurate MSE. Thus the Assumption 2 concern is more about external validity and is already disclosed. The hypothesis-testing error, by contrast, is an internal statistical flaw: the reported p-values are not valid evidence, and the paper presents them as evidence of a practical benefit. The concrete bootstrap/ground-truth check would settle whether the significance claim survives. The central consistency theorem and the MSE-based prediction results remain intact under this concern, so the appropriate verdict is unchanged from the reader's CONDITIONAL. I partially agree with the reader because they also flagged the significance testing as most serious, but their stated weakest assumption was Assumption 2 rather than this issue.","tokens_in":26065,"tokens_out":15200,"duration_ms":151742,"concrete_test":"In the HIV and Sepsis simulators, run a bootstrap over the full estimation pipeline: resample D_b and D_e, retrain \\hat f and \\hat h, recompute \\hat V^{π_e} for each resample, and form a confidence interval for V^{π_e} - V^{π_b}. If the bootstrap interval contains 0 while the paper's t-test reports p<10^{-6}, the significance claim fails. Alternatively, using the simulator's ground-truth long-term returns, perform the t-test on the true returns of the on-policy trajectories; if the true-return p-values are not <10^{-6}, the reported significance is an artifact of using predictions as data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical claim that the method 'can accurately predict whether the new policy’s expected returns will outperform the behavioral policy’s with significant p-values < 10^{-6}' rests on a statistically invalid procedure. In Section 12.4.1, the authors feed the 500 per-trajectory estimates \\hat f(τ_{0:h}) (or DR analogues) into scipy t-tests as if they were i.i.d. long-term returns. These estimates are not observations of the long-term outcome: (1) they are conditional expectations, so their variance is strictly smaller than the variance of actual returns, which deflates the standard error and inflates the t-statistic; (2) all 500 predictions share the same fitted regression/density-ratio nuisance functions, inducing correlation across the 'samples' that the t-test assumes away; (3) the point estimates themselves carry estimation error that the test ignores. Consequently, the reported p-values have no valid frequentist interpretation. This is load-bearing because a headline practical benefit of the method is fast identification of superior policies, and the paper's introduction and results section foreground these p-values. The MSE results and the conditional consistency theorem are not invalidated by this issue, but the significance-based conclusion is unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the problem of estimating the long-term value of a sequential policy when the target policy takes novel actions not covered by the behavioral policy's action space, so standard coverage-based off-policy evaluation cannot be applied. The authors propose combining a short-horizon rollout under the target policy with long-horizon historical data collected under a behavioral policy, using a 'soft surrogacy' assumption (Assumption 2) and a coverage condition (Assumption 3). They define unweighted and weighted regression estimators and a doubly robust estimator (Section 5), prove a finite-sample high-probability bound for the doubly robust estimator (Theorem 4) and a product-form bias decomposition (Theorem 2), and report experiments on a toy problem and two clinical simulators (HIV and sepsis) showing that the estimators achieve low MSE after observing only 10% of the full horizon, along with significance tests claiming p-values below 10^-6 for detecting policy improvement.","tokens_in":26269,"tokens_out":7363,"duration_ms":69552,"significance":"If the claims hold, the paper addresses an important and timely problem: predicting long-term outcomes of policies with novel actions using short-horizon data, which is highly relevant in healthcare and other domains. The theoretical contribution is solid: the doubly robust estimator and its finite-sample, exponential-tail bound are useful extensions of the ADML framework of Chernozhukov et al. (2023), and the paper provides detailed proofs and code. However, the empirical significance claims are weakened by the invalid p-value procedure in Appendix 12.4.1, the absence of variability estimates in the HIV experiments, and the gap between the theoretical assumptions and the experimental settings where those assumptions are acknowledged to be violated. The MSE results and the theoretical machinery remain valuable, but the paper as written overstates the strength of the empirical evidence.","major_comments":[{"comment":"The reported p-values less than 1e-6 are not statistically valid. The t-tests compare 500 per-trajectory estimates \\hat f(\\tau_{0:h}) or their doubly robust analogues against observed long-term returns, treating the predictions as i.i.d. observations of the long-term outcome. This is incorrect for three reasons: (i) the predictions are conditional expectations, so their variance is strictly smaller than the variance of actual returns (the term E[Var(G | \\tau)] is missing), which deflates standard errors and inflates the t-statistic; (ii) all 500 predictions are produced from a single fitted regression or density-ratio model, so the predictions are correlated across trajectories, violating the t-test's independence assumption; and (iii) the estimation error in \\hat f and \\hat h is ignored. Consequently, the p-values in Tables 6 and 7, and the claims in the Introduction and Section 7.4 that the estimators 'can reject the null with p-values less than 1e-6', have no valid frequentist interpretation. The MSE results in Table 2 are not invalidated by this issue, but the statistical-significance claim is. The authors should replace this analysis with valid confidence intervals or tests, for example using the asymptotic normality of the cross-fitted DR estimator or a bootstrap that resamples both datasets and refits the nuisance functions, and report those intervals.","section":"Table 2"},{"comment":"The HIV columns in Table 2 report only point estimates of MSE, without any measure of variability across seeds or bootstrap resamples, in contrast to the Sepsis columns which report mean ± std across 5 bootstrap seeds. Given that the on-policy sample size is only 500 and that hyperparameters are selected by cross-validation, the apparent advantage of the proposed estimators over LOPE at h=10 (e.g., 68.15 vs. 69.95 for the soft surrogate estimator) could be within the noise of the experiment. The authors should report the same bootstrap or seed-based variability for the HIV domain as they do for Sepsis, so that the empirical claims are comparable and reproducible.","section":"Section 7.4 and Appendix 9.1"},{"comment":"The theory in Section 6 (Theorems 4–6) establishes consistency and double robustness under Assumption 2, but the authors themselves state in Section 7.4 that Assumption 2 'likely does not hold' in the HIV and sepsis simulators. Appendix 9.1 notes that without Assumption 2 the estimator targets V^{π_e,π_b}, not V^{π_e}, and the experiments compare against the ground-truth V^{π_e}. This creates a load-bearing gap: the empirical demonstrations are not covered by the theoretical results. To support the claim that the method predicts the target policy's long-term value, the authors should either (a) compute V^{π_e,π_b} in the simulators and quantify the bias introduced by the assumption violation, or (b) provide a sensitivity analysis that explicitly varies the degree of violation and shows the estimators remain close to V^{π_e}. Without one of these, the connection between the theoretical guarantees and the main empirical setting is not established.","section":"Section 7.4 and Appendix 9.1"}],"minor_comments":[{"comment":"The displayed bound in Theorem 4 ends with an incomplete fragment 'or 2K log(4K/δ)/N' that should be removed or completed; as written it is unclear whether the last term is of Bernstein type or a product-error bound.","section":"Section 5.2 and Section 6"},{"comment":"The symbol h is used both for the short-horizon length and for the density-ratio estimator (e.g., \\hat h^{(k)}(\\tau) in Equations (7) and (11)). This is confusing, especially in the proof of Theorem 4 where both meanings appear close together; consider using a different symbol such as a(\\tau) for the density ratio.","section":"Table 9 and Section 12.4.2"},{"comment":"In Section 12.4.2 the ground-truth value of the sixth target policy is listed as -0.022 in the text, but Table 9 lists -0.002; the discrepancy should be corrected both places.","section":"Table 8"},{"comment":"The heading 'Weighed soft surrogate estimator' in Table 8 contains a typo; it should be 'Weighted soft surrogate estimator'.","section":"Appendix 12.4.1"},{"comment":"The sentence 'We included the columns for h = 20 and h = 2 in the main text' is confusing because the main text does not include the hypothesis-testing tables; please clarify which results appear in the main text and which are in the appendix.","section":"Appendix 12.4.1"}],"recommendation":"major_revision","confidential_remarks":"The p-value issue in Appendix 12.4.1 is real and affects a headline claim, but it is correctable through a more careful statistical analysis, and the core theory appears sound. The paper is within the journal's scope. I would look favorably on a revised version that provides valid confidence intervals, adds variability estimates for HIV, and explicitly connects the theory to the experimental settings where Assumption 2 is violated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core of this paper is the soft-surrogacy relaxation for sequential RL policies: conditioned on the short-horizon trajectory, the expected future returns under the target policy equal those under the behavioral policy. That moves beyond the standard surrogate-index assumption and beyond Battocchi et al.'s single-intervention setting. The doubly robust estimator with cross-fitting is standard machinery, but the finite-sample exponential-tail bound under covariate shift is a genuine new result. The toy experiment cleanly shows double robustness to misspecified regression or density ratio. The MSE results on the HIV and sepsis simulators at 10% of the horizon are decent and beat the baselines they tried.\n\nThe soft spots are real but not fatal to the method. The biggest one is the significance testing in Section 12.4.1. They feed per-trajectory predictions into t-tests as if those were observed returns. They are conditional expectations, so their variance is smaller than the variance of actual returns, and all 500 predictions share fitted nuisance functions, so the samples are not independent. The reported p-values < 1e-6 do not have the frequentist meaning claimed. This is load-bearing because a headline practical benefit is fast identification of superior policies. The MSE results survive, but the significance conclusion as written is unsupported. Also, the HIV results in Table 2 come without error bars, and code and data are not yet public. These are addressable, but they limit how much a reader can verify.\n\nAssumption 2 is strong and untestable, and the authors admit their simulators likely violate it. That is not a fatal flaw—the paper is honest about it and shows empirical robustness—but it does mean the practical relevance rests on a premise that cannot be checked from data. A careful reader should view the method as useful when soft surrogacy approximately holds, not as a general solution to the novel-action problem.\n\nWho is this for? Researchers working on off-policy evaluation, surrogate indices, or long-term policy evaluation, especially in healthcare. It deserves a serious referee: the theory is real, the empirical comparisons are reasonable, and the flaws are fixable. I would send it to review with a request to correct the hypothesis-testing claims, add uncertainty estimates for HIV, and ideally release the artifacts. Conditional acceptance, not rejection.","headline":"Solid OPE paper with a real theory contribution and a load-bearing statistical flaw in the significance claims; worth reviewing, but the p-values should not be taken at face value.","tokens_in":26823,"tokens_out":1877,"would_cite":true,"duration_ms":20483,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under a relaxed \"soft surrogacy\" assumption, the long-term value of a new sequential policy can be estimated from short-horizon on-policy data combined with long-horizon historical data, with a doubly robust estimator that carries…","keywords":["off-policy policy evaluation","long-term policy value","soft surrogacy","doubly robust estimation","finite-sample guarantees","novel actions","sequential decision making","clinical simulators"],"falsifier":"Run a simulator with known ground truth in which the short-horizon trajectory distributions of the two policies match but their future returns diverge because of a later novel action, and check whether the doubly robust estimator's bias grows with that divergence even when the regression and density-ratio models are well-specified.","tokens_in":25845,"feed_emoji":"⏱️","tokens_out":9090,"duration_ms":77319,"temperature":0.7,"pith_summary":"Off-policy policy evaluation normally breaks when the new policy introduces actions absent from historical data, because importance weights cannot be formed. This paper argues that under a \"soft surrogacy\" condition—conditioned on the short-horizon trajectory, the expected future returns of the new and behavioral policies coincide—the long-term value of the new policy can be estimated from short on-policy data plus full-horizon historical records. It develops a regression-based soft surrogate estimator and a doubly robust variant with finite-sample error bounds, and shows in HIV and sepsis simulators that accurate value estimates (and p<$10^{{-6}}$ significance tests against the behavioral policy) are achievable after observing only 10% of the full horizon.","feed_headline":"Policy value predicted from just 10% of the horizon","feed_subtitle":"A relaxed surrogacy condition lets new treatments be evaluated from short-term data plus historical records.","key_machinery":"The object that carries the argument is the soft surrogacy identity (Equation 1), which equates the expected future returns of the target policy $\\pi_e$ and the behavioral policy $\\pi_b$ given the same short-horizon trajectory $\\tau_{0:h}$. This identity converts the long-term policy value into a linear functional of the historical regression $f(\\tau_{0:h}) := E_{\\pi_b}[G \\mid \\tau_{0:h}]$, averaged over short on-policy trajectories. On top of this, the paper builds a doubly robust estimator with K-fold cross-fitting that adds a density-ratio-weighted residual term; its theoretical engine is the representation of the estimand as a linear functional of a regression under covariate shift, whose Riesz representer is the density ratio, so the bias factorizes as the product of the two nuisance errors and can be controlled by Bernstein bounds.","core_discovery":"The paper's central claim is that evaluating a sequential policy with novel actions becomes tractable once the standard surrogacy assumption is relaxed to a policy-pair \"soft surrogacy\" condition (Equation 1). This identity makes the target policy's value equal to the expectation, over short trajectories drawn from the new policy, of the regression $f(\\tau_{0:h}) = E_{\\pi_b}[G \\mid \\tau_{0:h}]$ learned from behavioral data. The paper proves that a doubly robust, cross-fitted estimator of this quantity has bias equal to the average product of the regression error and the relative density-ratio error across folds, is consistent whenever both nuisance estimates are consistent at any rate, and admits high-probability finite-sample bounds with exponential tails. Empirically, the estimators predict the long-term value of new treatment policies in HIV (H=200, h=10) and sepsis (H=20, h=2) simulators with lower MSE than baselines and significant p-values, even though the soft surrogacy assumption is expected to be violated in these simulators.","pith_inferences":["Because the theory only requires soft surrogacy for the specific pair $(\\pi_b, \\pi_e)$, a practical diagnostic for deploying the method would be to compare $f$'s predictions on overlapping trajectories under the two policies; a systematic gap would warn that the assumption is violated.","The bound's explicit dependence on $|D_e|$ suggests an experiment-design rule: when soft surrogacy holds approximately, collecting more diverse short-horizon trajectories under the new policy may reduce error faster than extending the observation horizon, as long as coverage is maintained.","The fact that estimators perform well in simulators where the assumption is likely violated hints that the assumption is sufficient but not necessary; a sensitivity analysis characterizing the maximum tolerable violation would broaden the method's applicability.","A natural validation strategy before trusting the method on a genuinely novel action is to apply it to past policies (whose full-horizon outcomes are already known) and measure the calibration of its h-step predictions."],"forward_implications":["Clinical trials of novel treatments could be shortened from a full multi-year horizon to a short observation window: in the sepsis simulator, 2 of 20 steps suffice to predict the value of a policy that adds vasopressors, and in HIV, 10 of 200 steps suffice to rank the new policy against the behavioral one.","The doubly robust estimator is consistent at any nuisance convergence rate, and unbiased if either the regression or the density ratio is exactly correct, so it inherits the usual double-robustness guarantees in a covariate-shift sequential setting.","The finite-sample bound has exponential tails, improving on prior finite-sample DML results that did not cover this covariate-shift regime, and it shows the variance terms from true nuisances dominate when the estimates are consistent.","The same machinery transfers, in principle, to education and e-commerce, where short-term activity logs are soft surrogates for end-of-year scores or customer churn, provided the soft surrogacy and coverage assumptions hold."],"supporting_citations":[{"why":"Defines the surrogate index framework that this work relaxes from single interventions to sequential policies.","marker":"Athey et al. (2019)"},{"why":"Introduces the \"dynamic invariance\" assumption for novel treatments, the template for the soft surrogacy condition.","marker":"Battocchi et al. (2021)"},{"why":"Supplies the automatic debiased machine learning framework for linear functionals under covariate shift that the theoretical analysis builds on.","marker":"Chernozhukov et al. (2023)"},{"why":"Provides the prior finite-sample DML theorem that the paper extends to exponential tails in the covariate-shift setting.","marker":"Chernozhukov et al. (2022)"},{"why":"Supplies the density-ratio estimation via classification used to build the weighted regression estimator.","marker":"Sugiyama et al. (2011)"},{"why":"Provides the HIV treatment simulator used in the main experiments.","marker":"Ernst et al. (2006)"},{"why":"Provides the sepsis management simulator used in the main experiments.","marker":"Oberst and Sontag (2019)"},{"why":"Defines the LOPE baseline, the closest prior long-term OPE method, which this paper compares against.","marker":"Saito et al. (2024)"}],"fun_headline_variants":["Soft surrogacy lets new policies be judged from 10% of data","Predicting long-term policy value with short-term data","Doubly robust estimator cuts evaluation horizon to 10%","New treatment policies scored from a fraction of the timeline","Relaxed assumptions make novel-action policy evaluation tractable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that, conditioned on the short-horizon trajectory, the new policy and the historical behavioral policy have identical expected future returns; if that equality fails, the regression target is biased and the error propagates to every estimator.","fun_headline_variants_meta":{"raw":{"variants":["Soft surrogacy lets new policies be judged from 10% of data","Predicting long-term policy value with short-term data","Doubly robust estimator cuts evaluation horizon to 10%","New treatment policies scored from a fraction of the timeline","Relaxed assumptions make novel-action policy evaluation tractable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1338,"prompt_tokens":951,"completion_tokens":387,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":567,"tokens_out":387,"duration_ms":4106,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:15:08.629937+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a simulator with known ground truth in which the short-horizon trajectory distributions of the two policies match but their future returns diverge because of a later novel action, and check whether the doubly robust estimator's bias grows with that divergence even when the regression and density-ratio models are well-specified.","supporting_citations":[{"cited_title":"and Sontag, D","cited_arxiv_id":null,"evidence_quote":"Provides the sepsis management simulator used in the main experiments."},{"cited_title":"W., and Kang, H","cited_arxiv_id":null,"evidence_quote":"Defines the surrogate index framework that this work relaxes from single interventions to sequential policies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the \"dynamic invariance\" assumption for novel treatments, the template for the soft surrogacy condition."},{"cited_title":"K., and Singh, R","cited_arxiv_id":null,"evidence_quote":"Provides the prior finite-sample DML theorem that the paper extends to exponential tails in the covariate-shift setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the density-ratio estimation via classification used to build the weighted regression estimator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the HIV treatment simulator used in the main experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the LOPE baseline, the closest prior long-term OPE method, which this paper compares against."}],"review_version":1}