Pith. sign in

REVIEW 4 cited by

Evaluating the Surrogate Index as a Decision-Making Tool Using 200 A/B Tests at Netflix

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.11922 v2 pith:7U5HWE3Q submitted 2023-11-20 stat.AP stat.ME

Vickie Zhang , Michael Zhao , and Maria Dimakopoulou , Anh Le , Nathan Kallus This is my paper · ORCID
classification stat.APstat.ME
keywords indexsurrogateteststreatmentdirectlyeffectsfindlong-term
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Surrogate index approaches have recently become a popular method of estimating longer-term impact from shorter-term outcomes. In this paper, we leverage 1098 test arms from 200 A/B tests at Netflix to empirically investigate to what degree would decisions made using a surrogate index utilizing 14 days of data would align with those made using direct measurement of day 63 treatment effects. Focusing specifically on linear "auto-surrogate" models that utilize the shorter-term observations of the long-term outcome of interest, we find that the statistical inferences that we would draw from using the surrogate index are ~95% consistent with those from directly measuring the long-term treatment effect. Moreover, when we restrict ourselves to the set of tests that would be "launched" (i.e. positive and statistically significant) based on the 63-day directly measured treatment effects, we find that relying instead on the surrogate index achieves 79% and 65% recall.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards the Efficient Inference by Incorporating Automated Computational Phenotypes under Covariate Shift

    stat.ME 2025-05 conditional novelty 6.0 of 10

    Doubly robust, semiparametrically efficient estimators that incorporate automated computational phenotypes (ACPs) into semi-supervised inference under covariate shift, with explicit efficiency gains driven by ACPs in ...

  2. Predictions as Surrogates: Revisiting Surrogate Outcomes in the Age of AI

    stat.ML 2025-01 accept novelty 6.0 of 10

    Recalibrated prediction-powered inference estimates the optimal imputed loss by cross-fitted machine learning, always improving on label-only inference and matching the best possible variance among PPI estimators when...

  3. Impatient Bandits: Optimizing for the Long-Term Without Delay

    cs.LG 2025-01 conditional novelty 6.0 of 10

    A Thompson sampling bandit with a Bayesian filter over progressively revealed engagement signals improves cold-start recommendation before long-term rewards are observed, with regret bounded by the Value of Progressiv...

  4. Predicting Long Term Sequential Policy Value Using Softer Surrogates

    cs.AI 2024-12 conditional novelty 6.0 of 10

    A soft-surrogate estimator predicts a sequential policy's long-term value from short-horizon on-policy data and full-horizon historical data, with finite-sample guarantees.

Pith tools