Evaluating Decision Rules Across Many Weak Experiments

Aur\'elien Bibaut; Colin Gray; Nathan Kallus; Simon Ejdemyr; Winston Chou

REVIEW 3 major objections 5 minor 22 references

Evaluating Decision Rules Across Many Weak Experiments

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A cross-validation estimator removes the winner's curse when choosing decision rules from weak A/B tests.

desk verdict Solid methodological contribution with a clean cross-validation estimator; the Netflix 33% headline rests on undisclosed details about how the challenger rule was constructed and selected. read the letter →

arxiv 2502.08763 v2 pith:DHDCLWU4 submitted 2025-02-12 stat.ME

Winston Chou , Colin Gray , Nathan Kallus , Aurélien Bibaut , Simon Ejdemyr This is my paper · ORCID

classification stat.ME MSC 62F0762G09

keywords A/Btestingdecisionrulescross-validationwinner'scurseproxymetricscumulativereturnsexperimentsplitting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Firms running thousands of A/B tests need decision rules that map each experiment's results to a launch choice, but the obvious way to score a rule on past data—take the average reward of the arm it would have picked—suffers from the winner's curse: arms picked because of noise look better than they are. This paper proposes scoring rules by their cumulative returns to a north star metric and proves that a cross-validation estimator using experiment splitting is far less biased than that naive plug-in. Under a Poisson model for per-experiment sample size, leave-some-out cross-validation is unbiased up to an explicit scaling factor, so the best rule in a finite candidate set can be selected consistently as the number of experiments grows, even when every experiment is too weak for its arm effects to be measured reliably. A case study on 123 historical tests at Netflix estimates that adopting a new proxy-based rule would have increased cumulative north-star returns by about 33%, and the company adopted the rule.

What carries the argument

The central object is the cross-validation estimator with experiment splitting: random folds per experiment, rule applied to all folds but one, reward of the chosen arm estimated on the held-out fold, then averaged over folds and experiments. The key identity that carries the theory is the Stein-Chen identity for Poisson variables, $\mathbb{E}[X f(X)] = \lambda \, \mathbb{E}[f(X-1)]$, which the paper applies $\ell$ times to show that the expected cross-validation estimate equals, up to the scaling $M_0^{-\ell} \ell!$ times the number of folds, the expected reward of the rule applied to the full experiment. That converts the winner's curse—the bias that arises when arms are chosen partly by noise—from a bias that persists as $N$ grows into a harmless scaling factor, provided the candidate rules are fixed in advance.

What would settle it

Rebuild the challenger proxy's weights using only experiments outside the 123-test evaluation sample, then rerun the cumulative-return comparison on that holdout sample; if the estimated 33% advantage shrinks or disappears, the headline result was an artifact of circular rule selection.

Watch

Extended reading notes

Core claim

The paper's central claim is that the winner's curse that inflates naive estimates of decision-rule returns can be eliminated by separating, within each experiment, the data that chooses the winning arm from the data that scores its reward. The cross-validation estimator does exactly that: it applies the rule to all folds but one, then measures the chosen arm's reward on the withheld fold. Theorem 3.1 shows that when the number of units per experiment is Poisson-distributed, leave-ℓ-out cross-validation estimates the rule's expected reward with zero bias up to a known scaling factor, and Theorem 3.2 gives a regret bound of order $O_p\!\left(\sqrt{\log|\mathcal{D}|/(N M_0)}\right)$ for selecting among a finite set of rules as the number of experiments $N$ grows. The paper also shows analytically that the naive estimator's bias is driven by the covariance in sampling error between proxy and north star metrics, whereas the cross-validation estimator depends only on the covariance of true treatment effects, so the cross-validated ranking of proxy rules is correct in a realistic weak-signal regime.

Load-bearing premise

The load-bearing premise is that the decision rules being compared are fixed before the experiments used to evaluate them are observed; if the challenger proxy in the Netflix case was built from those same 123 experiments, the estimated 33% gain could be inflated by in-sample selection of the rule itself.

Editorial extensions

If this is right

Experimentation programs can compare candidate decision rules—significance thresholds, proxy metrics, guardrail combinations—by their estimated cumulative returns rather than by ad hoc or purely prior-driven arguments.
The naive plug-in estimate of a rule's returns remains biased even with infinitely many experiments whenever each experiment has finite sample size, and the bias grows with the noise-to-signal ratio of the metrics involved.
The cross-validation estimator's negative bias shrinks as the number of folds grows, and under Poisson sample sizes it is exactly unbiased up to a scaling, so increasing the number of experiments rather than the number of units per experiment is the right asymptotic regime for rule selection.
In the Netflix case, cross-validation showed the challenger rules to be comparable to each other and superior to the status quo by roughly 33%, whereas the naive estimator overstated the gap between them; decision-makers adopted the new rule on the strength of that evidence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the challenger proxy's weights were tuned on the evaluation experiments, the 33% estimate is an upper bound on the rule's true improvement; an honest evaluation would need fully external rule construction.
The same experiment-splitting framework generalises naturally to tuning p-value thresholds and guardrail strictness, treating each threshold as a rule candidate and searching over a grid.
When experiments interfere with each other (redundant interventions or carryover), the independence assumption fails; a dependency-aware version of the folds would be needed to retain unbiasedness.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The methodological core is real and worth engaging. The paper adapts experiment splitting to the problem of evaluating decision rules across many weak experiments, and adds two genuinely new theoretical pieces: the Poisson-unbiasedness identity (Theorem 3.1) and the finite-set consistency bound (Theorem 3.2). The Stein-Chen argument is neat, and the simulation study clearly demonstrates that the CV estimator ranks rules correctly while the naive plug-in can prefer a proxy whose correlation with the north star is driven by noise. The theory is internally consistent under the stated assumptions, and the authors are honest that the remaining bias depends on the number of folds and that the Poisson-M_i assumption is a modeling device. Code for the simulations is public, which is a real plus.

The soft spot is the Netflix case study, and the stress-test note lands on it. The 33% estimate is the headline, but the paper never discloses whether the challenger proxy Z1 was fit on the same 123 experiments used for evaluation. Section 5 says Z1 was built using the meta-analysis method from the authors' earlier paper [4], but does not say which experiments fed that fit. If Z1's weights were learned from those 123 tests, then the within-experiment CV does not protect against that leakage — the weights are functions of the full dataset, so the fitted rule is evaluated on data that influenced its construction. The sentence about experimenting with 'a number of other combinations' and omitting them for brevity is also a red flag: if rule selection happened after seeing CV estimates on the same data, that reintroduces a winner's-curse-like bias at the rule level. The p-values of 0.07–0.09 do not account for that search. This is not a fatal flaw in the method, but it means the empirical claim is not fully supported as reported.

The Poisson assumption is slightly contrived, but that is a minor concern because the paper notes it is consistent with enrolling a fixed percentage of incoming units and because the practical recommendation is to assess sensitivity to fold count. The absence of a prespecified sampling plan for the 123 tests is a moderate limitation, not a dealbreaker — industrial case studies are often retrospective.

Who is this for? Practitioners running large experimentation programs and researchers working on meta-analysis or proxy metrics will both get value from the theory and the simulation. The paper deserves a serious referee: the core idea is useful, the proofs check out, and the case study points to a real adoption decision. My recommendation is to send it to peer review, but the authors should be asked to clarify the construction of Z1 and the rule-selection procedure before publication.

Referee Report

3 major / 5 minor

Summary. The paper studies how to evaluate decision rules that map A/B test outcomes to launch decisions, using cumulative returns to a north-star metric across past experiments. It defines an estimand \Gamma(D), shows that the plug-in estimator suffers from winner's-curse bias, and proposes a within-experiment cross-validation estimator that fits the rule on training folds and evaluates the chosen arm on held-out folds. Under a Poisson-M_i assumption, it proves an exact unbiasedness result for a scaled leave-\ell-out version and a consistency rate for selecting among a finite set of rules. The method is illustrated with a closed-form Gaussian proxy-metric model, a simulation study, and a Netflix case study in which a challenger rule is estimated to increase cumulative north-star gains by over 33%, an estimate that the authors say led to adoption of the new rule. The paper closes with a discussion of independence and stationarity assumptions.

Significance. The core proposal is practically motivated and the estimator is simple and plausible; if the fixed-rule premise is met, the within-experiment splitting idea is a useful addition to the experimentation literature. The closed-form analysis in Section 4.2 is a clear demonstration that the naive estimator can prefer noise-driven proxies while the CV estimator ranks them correctly, and the simulation study supports that ranking claim. The availability of replication code for the simulations is a concrete strength, as is the authors' explicit discussion of the independence and stationarity assumptions in Section 6. However, the headline 33% Netflix result is not currently protected by the paper's theory: the manuscript does not disclose whether the challenger proxy Z1 was fit on the same 123 experiments used for evaluation, and footnote 9 indicates that the candidate set was expanded after seeing results. In addition, the proof of the main consistency theorem contains an unjustified MGF identity. These issues make the central applied claim and one of the theoretical guarantees load-bearing and in need of repair.

major comments (3)

[§5.2] The headline estimate of a 33% gain from Rule 3 is not supported by the paper's theory unless the challenger proxy Z1 and the candidate rule set were fixed before looking at the 123 experiments. Section 5 states that Z1 was constructed using the meta-analysis method of [4] from historical experiments, but it never discloses whether those historical experiments include the same 123 tests used in Section 5.2. Since the within-experiment cross-validation splits units but not experiments, any weights of Z1 fit on the 123 experiments would enter every training fold, so the CV estimate would remain optimistic. Moreover, footnote 9 says the authors 'also experimented with a number of other combinations' of rules and omitted them; selecting Rule 3 after computing its CV estimate on the same data reintroduces a winner's-curse-like selection bias at the rule level, and the reported p-values of 0.07-0.09 do not account for this search. Theorems 3.1 and 3.2 cover a fixed decision rule, so an additional independence or sample-splitting argument is needed to support the 33% claim.
[§3] The proof of Theorem 3.2 uses the identity E[exp(s X_i(D))] = exp(M0(E[exp((s/M0)\hat R^CV_{ip})] - 1)) to obtain a sub-Gaussian bound. This identity would hold if, conditional on M_i, the leave-one-out fold estimates were independent summands of a compound-Poisson sum, but they are not independent: each \hat R^CV_{ip} is evaluated on fold p using a decision rule fit on all other folds, so the summands share training data for every p. A simple example with i.i.d. units and the rule 'select the arm with the larger sample mean' shows dependence among the fold estimates. The proof also treats Γ(D) as a constant in the MGF factorization although Γ(D) was defined as a random average; if Γ(D) is meant to be E[R_i D(O_i)], this must be stated explicitly. The consistency rate may be recoverable with a bounded-difference or U-statistic argument, but the argument as written does not establish it.
[§3] The exact unbiasedness result is stated for a scaled leave-ℓ-out estimator under the assumption M_i ~ Poisson(M0), and the scaling factor M0^{-ℓ}ℓ! depends on the unknown mean M0. For the fixed-M_i regime that describes the Netflix case study, the estimator has the nonzero bias in Eq. (4), and no theoretical bound is given for this bias as a function of the number of folds or the signal-to-noise ratio; the evidence for the 'much less biased' claim in that regime is limited to the simulation in Section 4.3. A sensitivity analysis for non-Poisson M_i (e.g., fixed M_i or overdispersed counts with the same mean) would clarify how load-bearing the Poisson assumption is for the paper's central claim.

minor comments (5)

[§2] The summation in Eq. (4) uses K where the experiment has K_i arms; this is inconsistent with Eq. (3) and with the notation established in Section 2.
[§5.1] The phrase '123 A/B past tests' should be '123 past A/B tests'; more substantively, the sampling plan for selecting these tests is not described (e.g., random sample versus convenience sample), which limits the generalizability of the case-study estimate.
[§5.2] The p-values reported as 'between 0.07 and 0.09' are not accompanied by a description of the null hypothesis, the test statistic, or how the multiple folds are aggregated; a brief statement would improve reproducibility.
[§3] The paper says it uses weighted averages in practice but omits the weights; defining the weights and explaining how P_i are chosen in the Netflix application would help readers apply the method.
[§4.3] The simulation results are reported without Monte Carlo standard errors; since the simulation study is the main evidence for the fixed-M regime, reporting standard errors would strengthen the comparison between the naive and CV estimators.

Circularity Check

2 steps flagged · score 6.0 of 10

The CV estimator itself is self-contained, but the Netflix 33% headline is an in-sample, data-selected estimate: the challenger rule Z1 is constructed by the authors' own meta-analysis fitting and the reported rule was chosen after searching over combinations on the same 123 tests, so it is not protected by the fixed-D unbiasedness theorem.

fitted input called prediction [Section 5, Real-World Application, first paragraph; Section 5.2, Results]
"In that paper, we developed methods for fitting linear structural models of treatment effects using data from past experiments. We deployed these methods to construct an improved proxy metric as a linear combination of component metrics, which include the previous status quo metric. We then used techniques in this paper to demonstrate the favorability of the resulting metric relative to the previous decision metric."

Theorems 3.1 and 3.2 establish unbiasedness and consistency only for rules in a fixed set D, with Theorem 3.2 explicitly starting 'Fix |D| < infinity'. The CV algorithm splits units only within each experiment; it does not split experiments. If the challenger proxy Z1 is a linear combination whose weights were fitted by the authors' [4] procedure 'using data from past experiments', and the '123 A/B past tests' of Section 5.2 are drawn from those same experiments, then the weights are functions of the evaluation data and appear in every training fold. The reported 33% gain is then an in-sample evaluation of a data-fitted rule, not a counterfactual prediction protected by the paper's fixed-rule theorems.
fitted input called prediction [Section 5.1, footnote 9; Section 5.2, Results]
"We also experimented with a number of other combinations (such as different tie-breakers and requiring statistical significance in both Z0 and Z1), which we omit for brevity."

Theorems 3.1-3.2 control selection error only when the set D is fixed before seeing the evaluation data. Here the reported Rule 3 was chosen after the authors 'experimented with a number of other combinations' on the same data, so the 33% estimate is the maximum of an implicit, data-dependent family of rules. This reintroduces, at the rule-selection level, the winner's-curse bias the paper aims to eliminate, and the reported p-values of 0.07-0.09 condition on the selected rule without accounting for the search. The result is a fitted selection reported as a predicted gain.

full rationale

The central methodological contribution of the paper is not circular: the cross-validation estimator is derived from first principles, Theorem 3.1 uses a valid Stein-Chen identity argument for a fixed decision rule, Theorem 3.2 gives a finite-set consistency bound, and the simulation study evaluates the estimator against external benchmarks. The circularity is concentrated in the case-study claim that drives the headline. Section 5 constructs the challenger proxy Z1 using the authors' own prior meta-analysis method [4] and does not state that Z1's weights were fit on data disjoint from the 123 experiments used to evaluate it. Additionally, the paper admits to searching over other rule combinations on the same data before reporting Rule 3. Both facts mean the 33% estimate is not covered by the fixed-D unbiasedness and consistency theorems: the rule is not ex ante fixed, and experiment splitting cannot remove dependence on globally fitted weights. The p-values reported for the chosen rule do not account for this selection. The case-study finding is therefore partially circular, while the estimator itself remains independently supported.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The method's theoretical core rests on a small set of standard tools plus a deliberately chosen Poisson assumption. The case study introduces a tuned proxy metric whose fitting details are not disclosed, which is the main source of free parameters and potential circularity.

free parameters (2)

Number of folds P = P=10 in simulation; increasing folds in case study
The analyst chooses P; it trades a small negative bias (rule sees (P-1)/P of data) against variance. The paper recommends sensitivity analysis rather than a data-driven choice.
Weights of challenger proxy Z1 = Not disclosed
Z1 is a linear combination of component metrics constructed using the authors' prior meta-analysis method [4]. If these weights were fit on the same 123 experiments, they are free parameters on which the reported 33% gain depends; the paper does not disclose the fitting procedure or data split.

assumptions (5)

standard math Stein-Chen identity for Poisson random variables
Used in the proof of Theorem 3.1 to show exact unbiasedness under Poisson M_i.
ad hoc to paper M_i ~ Poisson(M0)
The Poisson sample-size assumption is introduced specifically so the leave-l-out CV estimator is exactly unbiased for E[R_i D(O_i)]; it is a modeling device rather than a standard experimental assumption.
domain assumption Independence of treatment effects across experiments
Stated in Section 6: a test's outcome cannot depend on other tests' launch decisions. This underlies the aggregation of independent experiments.
domain assumption Stationarity: rules that perform well on past returns will perform well in the future
Acknowledged in Section 6 as an implicit assumption, monitored by re-running the analysis periodically.
domain assumption Candidate decision rules are fixed in advance and do not use the evaluation data
The theory assumes a finite set D of pre-specified rules. The case study may violate this if the challenger proxy Z1 was fit on the same experiments used for evaluation.

invented entities (1)

Netflix challenger proxy metric Z1
purpose: A linear combination of component metrics used as the basis for decision rules 2 and 3 in the case study
Z1 is constructed from the authors' prior meta-analysis work [4] and evaluated on the same 123 historical tests. No external or forward-looking validation is provided, so there is no independent evidence for its superiority.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating Decision Rules Across Many Weak Experiments." pith.science (2026). https://pith.science/paper/DHDCLWU4

@misc{pith2026250208763,
  author       = {Pith},
  title        = {Pith review of: Evaluating Decision Rules Across Many Weak Experiments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DHDCLWU4}},
  note         = {Machine review of arXiv:2502.08763}
}

abstract

Technology firms conduct randomized controlled experiments ("A/B tests") to learn which actions to take to improve business outcomes. In firms with mature experimentation platforms, experimentation programs can consist of many thousands of tests. To effectively scale experimentation, firms rely on decision rules: standard operating procedures for mapping the results of an experiment to a choice of treatment arm to launch to the general user population. Despite the critical role of decision rules in translating experimentation into business decisions, rigorous guidance on how to evaluate and choose decision rules is scarce. This paper proposes to evaluate decision rules based on their cumulative returns to business north star metrics. Although intuitive and easy to explain to decision-makers, this quantity can be difficult to estimate, especially when experiments have weak signal-to-noise ratios. We develop a cross-validation estimator that is much less biased than the naive plug-in estimator under conditions realistic to digital experimentation. We demonstrate the efficacy of our approach via a case study of 123 historical A/B tests at Netflix, where we used it to show that a new decision rule would have increased cumulative returns to the north star metric by an estimated $33\%$, directly leading to the adoption of the new rule.

Figures

Figures reproduced from arXiv: 2502.08763 by the authors.

**Figure 2.** Bias of Naive and CV Estimators vs. Noise-to-Signal [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Asymptotics as 𝑀 → ∞. The light-opacity dashed lines denote the truth, while the solid lines denote estimators. While the bias of both estimators goes to zero as the number of units per treatment arm 𝑀 → ∞, the naive estimator is much more biased at small values of 𝑀, which can lead to the wrong choice of proxy. we estimated the cumulative returns to a handful of candidate launch rules:9 (1) Launch the arm with the … view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Estimated Cumulative Returns to Decision Rules [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 20 canonical work pages

[4]

Aurélien Bibaut, Winston Chou, Simon Ejdemyr, and Nathan Kallus. 2024. Learn- ing the Covariance of Treatment Effects Across Many Weak Experiments. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 153–162

work page 2024
[1]

Susan Athey, Raj Chetty, Guido W Imbens, and Hyunseung Kang. 2019. The Surrogate Index: Combining Short-term Proxies to Estimate Long-term Treatment Effects More Rapidly and Precisely. Technical Report. National Bureau of Economic Research

work page 2019
[2]

Eduardo M Azevedo, Alex Deng, Jose Luis Montiel Olea, Justin Rao, and E Glen Weyl. 2018. The A/B Testing Problem. In Proceedings of the 2018 ACM Conference on Economics and Computation . 461–462

work page 2018
[3]

Eduardo M Azevedo, David Mao, José Luis Montiel Olea, and Amilcar Velez. 2023. The A/B Testing Problem with Gaussian Priors. Journal of Economic Theory 210 (2023), 105646

work page 2023
[5]

Iavor Bojinov, David Holtz, Ramesh Johari, Sven Schmit, and Martin Tingley. 2025. Want Your Company To Get Better at Experimentation? https://hbr.org/2025/ 01/want-your-company-to-get-better-at-experimentation. Harvard Business Review, Accessed January 2025

work page 2025
[6]

Louis HY Chen. 1975. Poisson approximation for dependent trials. The Annals of Probability 3, 3 (1975), 534–545

work page 1975
[7]

Dominic Coey and Tom Cunningham. 2019. Improving Treatment Effect Estima- tors through Experiment Splitting. In The World Wide Web Conference. 285–295

work page 2019
[8]

Tom Cunningham and Josh Kim. 2020. Interpreting Experiments with Multiple Outcomes

work page 2020

Show all 22 references

[9]

Alex Deng, Pavel Dmitriev, Somit Gupta, Ron Kohavi, Paul Raff, and Lukas Ver- meer. 2017. A/B Testing at Scale: Accelerating Software Innovation. InProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1395–1397

2017
[10]

Alex Deng and Xiaolin Shi. 2016. Data-driven metric development for online controlled experiments: Seven lessons learned. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . 77– 86

2016
[11]

Simon Ejdemyr, Martin Tingley, Yian Shang, and Travis Brooks. 2024. Estimating the Returns from an Experimentation Program. In ACIC Conference

2024
[12]

Olivier Jeunen and Aleksei Ustimenko. 2024. Learning Metrics that Maximise Power for Accelerated A/B-Tests. InProceedings of the 30th ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining . 5183–5193

2024
[13]

Eugene Kharitonov, Alexey Drutsa, and Pavel Serdyukov. 2017. Learning sensitive combinations of A/B test metrics. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining . 651–659

2017
[14]

Ron Kohavi and Nanyu Chen. 2024. False Positives in A/B Tests. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 5240–5250

2024
[15]

Ron Kohavi, Alex Deng, Roger Longbotham, and Ya Xu. 2014. Seven rules of thumb for web site experimenters. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining . 1857–1866

2014
[16]

Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi, and Nathaniel T Stevens. 2024. Statistical challenges in online controlled experiments: A review of a/b testing methodology. The American Statistician 78, 2 (2024), 135– 149

2024
[17]

Minyong R Lee and Milan Shen. 2018. Winner’s Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 491–499

2018
[18]

Lee Richardson, Alessandro Zito, Dylan Greaves, and Jacopo Soriano. 2023. Pareto Optimal Proxy Metrics. arXiv preprint arXiv:2307.01000 (2023)

2023 arXiv
[19]

Timothy Sudijono, Simon Ejdemyr, Apoorva Lal, and Martin Tingley. 2024. Opti- mizing Returns from Experimentation Programs. arXiv preprint arXiv:2412.05508 (2024)

2024 arXiv
[20]

Diane Tang, Ashish Agarwal, Deirdre O’Brien, and Mike Meyer. 2010. Overlap- ping experiment infrastructure: More, better, faster experimentation. In Proceed- ings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. 17–26

2010
[21]

Stefan Thomke. 2020. Building a Culture of Experimentation. Harvard Business Review 98, 2 (2020), 40–47

2020
[22]

Nilesh Tripuraneni, Lee Richardson, Alexander D’Amour, Jacopo Soriano, and Steve Yadlowsky. 2024. Choosing a Proxy Metric from Past Experiments. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5803–5812. KDD ’25, August 3–7, 2025, Toron...

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [4]

Aurélien Bibaut, Winston Chou, Simon Ejdemyr, and Nathan Kallus. 2024. Learn- ing the Covariance of Treatment Effects Across Many Weak Experiments. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 153–162

work page 2024

[2] [1]

Susan Athey, Raj Chetty, Guido W Imbens, and Hyunseung Kang. 2019. The Surrogate Index: Combining Short-term Proxies to Estimate Long-term Treatment Effects More Rapidly and Precisely. Technical Report. National Bureau of Economic Research

work page 2019

[3] [2]

Eduardo M Azevedo, Alex Deng, Jose Luis Montiel Olea, Justin Rao, and E Glen Weyl. 2018. The A/B Testing Problem. In Proceedings of the 2018 ACM Conference on Economics and Computation . 461–462

work page 2018

[4] [3]

Eduardo M Azevedo, David Mao, José Luis Montiel Olea, and Amilcar Velez. 2023. The A/B Testing Problem with Gaussian Priors. Journal of Economic Theory 210 (2023), 105646

work page 2023

[5] [5]

Iavor Bojinov, David Holtz, Ramesh Johari, Sven Schmit, and Martin Tingley. 2025. Want Your Company To Get Better at Experimentation? https://hbr.org/2025/ 01/want-your-company-to-get-better-at-experimentation. Harvard Business Review, Accessed January 2025

work page 2025

[6] [6]

Louis HY Chen. 1975. Poisson approximation for dependent trials. The Annals of Probability 3, 3 (1975), 534–545

work page 1975

[7] [7]

Dominic Coey and Tom Cunningham. 2019. Improving Treatment Effect Estima- tors through Experiment Splitting. In The World Wide Web Conference. 285–295

work page 2019

[8] [8]

Tom Cunningham and Josh Kim. 2020. Interpreting Experiments with Multiple Outcomes

work page 2020