Pith. sign in

REVIEW 3 major objections 7 minor 81 references

Bayesian Inference Procedures for A/B Testing: An Overview

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Bayes factor stopping is the pivotal A/B testing configuration: it bounds false positives at every stopping time and is near-optimal for a broad class of costs.

desk verdict A useful industry-oriented synthesis of Bayesian A/B testing configurations, with a real but fixable gap: the anytime-valid FPR guarantee is exact only under known variance, and all simulations assume it. read the letter →

arxiv 2608.12949 v1 pith:AKTU4Q4M submitted 2026-08-13 stat.ME math.STstat.TH

M{aa}rten Schultzberg , Mattias Fr{aa}nberg This is my paper · ORCID
classification stat.MEmath.STstat.TH MSC 62F1562L1062C1062L15
keywords BayesianA/BtestingBayesfactoroptionalstoppingempiricalfalsediscoveryratesequentialanytime-validinferenceexpectedloss
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Bayesian A/B testing is routinely discussed as a single method, but the paper argues it is a family of (prior, stopping rule) configurations with different statistical guarantees, and it organizes them into three tiers. The central result is that Bayes factor stopping under any proper prior both keeps the false positive rate below alpha at every stopping time and is near-optimal for a broad class of cost functions used in A/B testing, so the frequentist-versus-Bayesian choice is largely one of parameterization. The paper also claims that calibrated shrinkage and false discovery rate control require an empirical Bayes prior estimated from a representative historical corpus, and that certain corpus defects make that calibration impossible no matter how much data is collected. A sympathetic reader should care because these distinctions determine which risks a real experimentation program actually controls: error rates, estimation accuracy, and regret are different risks, and the appropriate configuration follows from the risk being managed.

What carries the argument

The load-bearing object is the Bayes factor sequence $\mathrm{BF}_{10,n}$ computed under a proper prior, which is a nonnegative martingale under the null hypothesis; Ville's inequality converts the stopping threshold $1/\alpha$ into an unconditional anytime-valid bound $P(\sup_n \mathrm{BF}_{10,n} \ge 1/\alpha \mid H_0) \le \alpha$, so the rule can be checked after every observation without inflating the false positive rate. The same likelihood-ratio structure makes the Bayes factor an optimal decision statistic for cost functions with affine terminal payoffs, and the empirical Bayes mixture $\Lambda_n = (1-p) + p \cdot \mathrm{BF}_{10,n}$ adds a $(1-p)$ floor that yields per-metric false discovery rate control through a posterior-odds argument while inheriting the martingale property.

What would settle it

Simulate continuous monitoring of the two-group normal test with the variance estimated at each look, stop by $\mathrm{BF}_{10} > 1/\alpha$ under the null, and count how often the treatment is declared superior; if the empirical false positive rate exceeds $\alpha$ by more than Monte Carlo error at finite sample sizes, the anytime-valid guarantee the paper relies on does not hold exactly in the estimated-variance deployment.

Watch

Extended reading notes

Core claim

The paper's central claim is that the most common Bayesian A/B testing configurations form a three-tier hierarchy, and that the middle tier - Bayes factor stopping with any proper prior - is the pivotal one: it simultaneously controls the false positive rate at any stopping time through Ville's inequality and is near-optimal for sequential launch decisions whose terminal payoffs are affine in the posterior. As a result, the debate between decision-theoretic and frequentist formulations of A/B testing is largely a debate about parameterization, not about substance. The paper further claims that Tier 3, which adds per-metric false discovery rate calibration and calibrated shrinkage, is reachable only with an empirical Bayes mixture prior estimated from a representative corpus of historical experiments, and that winner-selected corpora, pooled programs, and heterogeneous metrics prevent calibration regardless of corpus size. Simulations show that flat-prior posterior-probability stopping reproduces naive peeking, that an oracle empirical Bayes prior achieves the lowest estimation error, and that expected-loss stopping minimizes regret only when shipping a null-effect variant is costless.

Load-bearing premise

The exact anytime-valid false positive bound and the Bayes factor closed form assume the per-observation variance is known; in real deployments variance is estimated, so the guarantees hold only asymptotically under regularity conditions.

Editorial extensions

If this is right

  • A platform that defaults to flat-prior posterior-probability stopping is running the same procedure as naive peeking, with false positive rate around 0.30 under continuous monitoring, so Bayesian branding alone confers no error-rate protection.
  • Bayes factor stopping with any proper prior offers both decision-theoretic near-optimality for affine terminal payoffs and unconditional false positive rate control at every stopping time, so for that cost class the Bayesian/frequentist choice is a parameterization choice.
  • Tier 3 benefits, per-metric false discovery rate control and calibrated shrinkage, require an empirical Bayes prior estimated from a representative corpus of roughly 200 historical experiments; below that, a program should stay at Tier 2 with explicit multiple-testing corrections.
  • Expected-loss stopping wins on expected regret only when shipping a null-effect variant is essentially free; once the shipping cost exceeds about 2% of a typical true effect, evidence-requiring rules such as Bayes factor thresholding or group-sequential designs dominate.
  • Relative to group-sequential testing, Bayes factor stopping needs no maximum sample size commitment and yields lower mean squared error from shrinkage, but lower power; the choice between them should follow which operational risk the program controls.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Bayes factor thresholding is near-optimal for any affine terminal cost, then business costs could be mapped directly into Bayes factor thresholds, letting platforms set stopping rules from customer-impact utilities rather than from a nominal alpha.
  • The paper's equivalence of the mSPRT mixing distribution and a Bayesian prior suggests that frequentist always-valid procedures could close the mean squared error gap by applying shrinkage estimators at stopping, a practice the paper notes is allowed but does not itself test.
  • The corpus failure modes imply that any organization-level empirical Bayes prior should be validated for exchangeability across programs and metric types before deployment; otherwise no amount of historical data fixes the inflated false positive rate that winner selection or pooling introduces.
  • A direct extension would measure the regret crossover point in a real experimentation program, since the simulation's crossover at shipping cost about 0.003 depends on the assumed effect-size distribution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper provides a systematic overview of Bayesian A/B testing configurations, organizing them into a three-tier hierarchy: Tier 1 (posterior coherence without error-rate control), Tier 2 (Bayes factor stopping with anytime-valid false positive rate control via Ville's inequality), and Tier 3 (empirical Bayes priors for FDR calibration and calibrated shrinkage). The central claims are that Bayes factor stopping is near-optimal for a broad class of decision-theoretic cost functions and, because it also controls the FPR, the choice between decision-theoretic and frequentist formulations is largely one of parameterization. The paper supports these claims with closed-form derivations, an extensive simulation study (Settings A-D), and comparisons against group-sequential tests and always-valid confidence sequences. The main body of the paper is clear and the appendices provide substantial formal detail, though the treatment of estimated variance is limited to an asymptotic statement and several of the headline guarantees are exact only under known variance.

Significance. If the central claims hold, the paper is a valuable reference for practitioners and researchers: it clarifies a frequently conflated set of methods, gives a useful decision matrix (Table 1), provides exact anytime-valid FPR control for Bayes factor stopping (Eq. 5), and offers a well-designed simulation comparison (Tables 2-3, Figures 2-9) showing the trade-offs among error rates, estimation accuracy, and regret. The paper also gives credit where it is due by making the equivalence between mSPRT and Bayes factor stopping explicit, and by documenting the corpus-quality failure modes for empirical Bayes priors. The appendices contain several rigorous derivations, including the Bellman argument for BF threshold rules (Appendix E), the prior-averaged FNR bound (Appendix F), and the proof that BF stopping implies credible-interval exclusion (Appendix G). However, the load-bearing claim that the same rule controls FPR in realistic deployments is only established under known variance; the extension to estimated variance is asserted asymptotically and not examined in finite samples.

major comments (3)
  1. [Section 2, Eq. (5), and Section 6] The central anytime-valid FPR guarantee is stated and simulated only under known variance. The text in Section 2 says σ² is treated as known and that the key results 'extend asymptotically to the estimated-variance case under standard regularity conditions,' but no finite-sample analysis is provided. In practice, the analyst plugs in an estimate σ̂²_n, and the exact e-process/martingale property of the BF sequence can be lost; the conditional expectation of the likelihood-ratio increment evaluated at an estimated variance is not generally 1 under the true variance. Since all simulations in Appendix D set σ=1 known (Settings A and B) and the headline FPR numbers in Tables 2 and 3 are for the idealized rule, the claim that 'the same rule also controls the false positive rate' in deployed A/B testing is not demonstrated. Please provide finite-sample simulation evidence with estimated variance, or an exact anytime-valid modification (e.g., variance estimation on a separate fraction of data), or explicitly scope the FPR guarantee to the known-variance setting throughout the abstract and Section 3.
  2. [Appendix H, Sections 2.3 and 3.2.2] The assertion that 'Bayes factor stopping is near-optimal for a broad class of cost functions' is not as well quantified as the abstract suggests. Appendix H proves exact BF-threshold optimality only under the two-point model, and for the continuous normal prior the proof covers the symmetric-cost case exactly, while for asymmetric costs the single BF threshold is described as a 'constrained approximation' with a gap that is not quantified. The numerical example for Wan et al. is informative but is a single parameter configuration. Please either provide a quantitative bound on the suboptimality gap for asymmetric costs under the normal prior, or temper the near-optimality claim in the abstract and Section 1 to reflect the conditions under which it has been established.
  3. [Section 3.3.1 and Appendix B] The Tier 3 FDR guarantee is supported mainly by a posterior-odds argument and an appeal to Storey (2003), but the paper does not provide a full frequentist FDR proof for the sequential stopping rule. Equation (6) is a fixed-sample Storey approximation, and the statement 'the (1−p̂) floor ensures that a rejected metric has posterior null probability at most (1−p̂)α' is a conditional Bayesian statement rather than an unconditional frequentist FDR bound at the random stopping time. The simulation evidence in Setting C is helpful, but the theoretical claim of 'per-metric FDR calibration' would be stronger with a precise statement of the conditions under which the Bayesian and frequentist FDR coincide at stopping times, or with a counterexample showing where they can diverge under prior misspecification.
minor comments (7)
  1. [Section 3.2.1] Typo: 'Non max need be committed to in advance' should read 'No n_max need be committed to in advance.'
  2. [Acknowledgements] The heading 'Acknowlegements' is misspelled.
  3. [Figure 1] The caption omits the value of δ or the context for the boundary; please clarify that the figure is for the null hypothesis with known σ=1 and V=0.1.
  4. [References] In the reference list, 'Deng, A., et al. (2023)' uses an incomplete author list; please list all authors or use the journal's abbreviation style consistently.
  5. [Table 2] The row label 'Flat Expected loss' is a bit terse; consider writing 'Flat prior with expected-loss stopping (ε=0.02)' to match the column header 'Stopping rule'.
  6. [Section 5.2] The statement that the mSPRT 'is exactly the Bayes factor under the same prior' is correct for the normal model used, but it would be helpful to note explicitly that the equivalence holds for the specific mixture martingale used by Johari et al. (2022) and not for all mSPRT variants.
  7. [Section 3.3.3] The sentence 'Non max need be committed to in advance' appears to be a typographical artifact; the intended phrase is likely 'No maximum sample size needs to be committed to in advance.'

Circularity Check

0 steps flagged · score 0.0 of 10

No load-bearing circularity: central equivalences and guarantees are derived from standard external results (Ville, Arrow et al., Johari et al.) and benchmarked against external frequentist procedures.

full rationale

The paper's central claims—BF stopping controls FPR via the martingale/Ville inequality (Eq. 5), the mSPRT is algebraically the Bayes factor under the same prior, flat-prior posterior thresholds equal z-test rejection regions, and affine-cost sequential problems have BF-threshold optimal policies (Appendix H)—are derived from the paper's own equations or from external cited results, not from the paper's outputs. The three-tier hierarchy is a taxonomy, and the statement that BF stopping achieves Tier 2 is backed by a proof sketch (Section 3.2.1) plus the standard Ville inequality, so it is not self-definitional in a damaging sense. The EB/FDR claims are supported by posterior-odds derivations and Storey's external FDR result. Simulations compare against external frequentist baselines (O'Brien–Fleming GST and Waudby-Smith–Ramdas confidence sequences) on a common DGP, so the empirical rankings are externally anchored. The two self-citations (Schultzberg et al. 2026; Schultzberg 2026) appear in the reference list but are not used to derive the load-bearing claims, so they do not constitute circularity. The known-variance assumption in Section 2 is an explicit modeling condition with an asymptotic extension cited to Johari et al. (2022) and van der Vaart (1998); it is a correctness/robustness limitation, not a circular step, because no target result is assumed into the derivation. No step was found where a prediction reduces by construction to its input.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on standard martingale and decision theory and on four domain assumptions: normal model with known variance, likelihood principle, corpus exchangeability, and affine costs. The only fitted quantities in the paper's own demonstrations are the empirical Bayes parameters (p̂, V̂) and the fixed threshold ε.

free parameters (4)
  • p̂ (EB non-null rate) = oracle 0.30; winner-selected ≈1.0; pooled ≈0.51
    Fitted with MLE/EM to a corpus of past experiments; the (1−p̂) floor controls FDR calibration and multiplicity (Section 3.3).
  • V̂ (EB prior scale) = oracle 0.20; pooled ≈0.37
    Fitted with the mixture; controls amount of shrinkage and BF power (Section 3.3).
  • ε (expected-loss tolerance) = 0.02
    Hand-selected threshold of the Stucchio (2015) rule, used in Setting A; determines when the expected-loss rule stops, but not a fidelity parameter.
  • V (BF Gaussian prior scale) = δ_MDE/σ = 0.2 in Settings B and D
    Chosen by the analyst to match the minimum detectable effect; affects power and shrinkage but not the Ville FPR bound.
assumptions (5)
  • domain assumption The two-group normal model with known variance σ² is a valid approximation; results extend asymptotically to estimated variance (Section 2).
    All exact derivations of the BF closed form and Ville bound assume known σ²; the paper hand-waves the extension.
  • standard math Bayes factor sequence is a nonnegative martingale under H0 for any proper prior (Ville's inequality).
    This is the mathematical foundation of Tier 2 FPR control (Section 3.2.1, Appendix B).
  • domain assumption The Likelihood Principle justifies posterior coherence regardless of stopping rule (Section 3.1).
    This philosophical premise underpins Tier 1's claim that posterior updates are valid.
  • domain assumption Experiments and metrics are exchangeable draws from a stable corpus distribution for EB calibration (Section 3.3.3, Discussion).
    Tier 3's calibration and FDR control depend on the corpus representing future experiments; the paper's failure-mode analysis acknowledges this.
  • domain assumption Terminal payoffs of the relevant cost functions are affine in the posterior (Appendix H).
    The near-optimality of BF threshold rules for decision-theoretic costs requires this affine structure; the paper excludes expected-loss stopping as a counterexample.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bayesian Inference Procedures for A/B Testing: An Overview." pith.science (2026). https://pith.science/paper/AKTU4Q4M

@misc{pith2026260812949,
  author       = {Pith},
  title        = {Pith review of: Bayesian Inference Procedures for A/B Testing: An Overview},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AKTU4Q4M}},
  note         = {Machine review of arXiv:2608.12949}
}
read the original abstract

Bayesian inference for A/B testing is a family of prior and stopping-rule configurations with fundamentally different statistical properties, but it is often discussed as a single method, and no systematic overview exists. This paper organizes common configurations into a three-tier hierarchy: 1) posterior coherence with no error control, 2) false positive rates bounded under continuous monitoring via Bayes factor stopping, and 3) false discovery rate control and calibrated shrinkage via empirical Bayes. Many commercial platforms operate at the lowest tier by default. We show that Bayes factor stopping is near-optimal for a broad class of cost functions, including most proposed in the A/B testing literature; because the same rule also controls the false positive rate, the choice between a decision-theoretic and a frequentist formulation is largely one of parameterization. Furthermore, the empirical Bayes prior is the only path to the third tier, but winner-selected corpora, pooled programs, and heterogeneous metrics can each prevent calibration regardless of corpus size. Simulations against group-sequential and always-valid frequentist baselines show that flat-prior posterior stopping exactly reproduces naive peeking, that a well-calibrated empirical Bayes prior achieves the lowest estimation error, and that expected-loss stopping minimizes regret only when shipping a null-effect variant is nearly free. Error rates, estimation accuracy, and regret are all different risks, and the appropriate method follows from the risks an experimentation program needs to control, not the other way around.

Figures

Figures reproduced from arXiv: 2608.12949 by the authors.

Figure 1
Figure 1. Stopping boundaries on |Zn| as a function of n for the normal￾conjugate model (σ = 1, V = 0.1, α = 0.05). The blue region is the BF continuation zone. The orange band is where the credible interval excludes zero but the BF has not yet crossed 1/α. the BF exceeds 1/α (Grünwald et al., 2024), so the interval and the stopping decision always agree. The credible interval and the confidence sequence answer different ques… view at source ↗
Figure 2
Figure 2. Cumulative FPR under continuous monitoring ( [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Reliability diagrams for three prior specifications under BF stopping. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Estimation variance of EB prior parameters [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Setting C, experiment-level FPR vs. corpus size [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 9
Figure 9. Figure 9: Setting D: expected regret per decision as a function of shipping cost [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

81 extracted references · 70 canonical work pages

  1. [1]

    Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57(1), 289--300

    1995

  2. [2]

    Berger, J. O. (1985). Statistical Decision Theory and Bayesian Analysis (2nd ed.). Springer

    work page 1985

  3. [3]

    O., & Berliner, L

    Berger, J. O., & Berliner, L. M. (1986). Robust Bayes and empirical Bayes analysis with -contaminated priors. Annals of Statistics, 14(2), 461--486

    work page 1986

  4. [4]

    O., & Berry, D

    Berger, J. O., & Berry, D. A. (1988). Statistical analysis and the illusion of objectivity. American Scientist, 76(2), 159--165

    work page 1988

  5. [5]

    Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29(4), 1165--1188

    2001

  6. [6]

    O., & Wolpert, R

    Berger, J. O., & Wolpert, R. L. (1988). The Likelihood Principle (2nd ed.). IMS Lecture Notes--Monograph Series

    work page 1988

  7. [7]

    Box, G. E. P., & Tiao, G. C. (1973). Bayesian Inference in Statistical Analysis. Addison-Wesley

    work page 1973

  8. [8]

    S., & Yu, B

    Bloniarz, A., Liu, H., Zhang, C.-H., Sekhon, J. S., & Yu, B. (2016). Lasso adjustments of treatment effect estimates in randomized experiments. Proceedings of the National Academy of Sciences, 113(27), 7383--7390

    work page 2016

Show all 81 references
  1. [9]

    H., Martin, J., & Goswami, B

    Chennu, S., Maher, A., Pangerl, C., Prabanantham, S., Bae, J. H., Martin, J., & Goswami, B. (2023). Rapid and scalable Bayesian AB testing. arXiv preprint, arXiv:2307.14628

    2023 arXiv

  2. [10]

    Campbell, H., & Gustafson, P. (2024). Defining a credible interval is not always possible with point-null priors. Bayesian Analysis, 19(2), 627--644

    2024

  3. [11]

    Cochran, W. G. (1957). Analysis of covariance: its nature and uses. Biometrics, 13(3), 261--281

    1957

  4. [12]

    de Heide, R., & Grünwald, P. D. (2021). Why optional stopping can be a problem for Bayesians. Psychonomic Bulletin & Review, 28, 795--812

    2021

  5. [13]

    P., Laird, N

    Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39(1), 1--38

    1977

  6. [14]

    Deng, A. (2015). Objective Bayesian two sample hypothesis testing for online controlled experiments. In Proceedings of the 24th International Conference on World Wide Web Companion (pp. 923--928). ACM

    2015

  7. [15]

    Deng, A., Li, Y., Lu, J., & Ramamurthy, V. (2018). On the statistical treatment of residuals in empirical Bayes estimation for A/B testing. In Proceedings of the 27th International Conference on World Wide Web Companion (pp. 1681--1685). ACM

    2018

  8. [16]

    Deng, A., Lu, J., & Chen, S. (2016). Continuous monitoring of A/B tests without pain: optional stopping in Bayesian testing. arXiv preprint, arXiv:1602.05549

    2016 arXiv

  9. [17]

    Deng, A., et al. (2023). From augmentation to decomposition: a new look at CUPED in 2023. arXiv preprint, arXiv:2312.02935

    2023 arXiv

  10. [18]

    Dienes, Z. (2014). Using Bayes to get the most out of non-significant results. Frontiers in Psychology, 5, 781

    2014

  11. [19]

    Dimmery, D., Bakshy, E., & Sekhon, J. (2019). Shrinkage estimators in online experiments. In Proceedings of KDD 2019 (pp. 2914--2922). ACM

    2019

  12. [20]

    M., & Berman, R

    Feit, E. M., & Berman, R. (2019). Test & Roll: Profit-maximizing A/B tests. Marketing Science, 38(6), 1038--1058

    2019

  13. [21]

    Frasco, M. (2018). The power of Bayesian A/B testing. Convoy Engineering Blog. https://medium.com/convoy-tech/the-power-of-bayesian-a-b-testing-f859d2219d5

    2018

  14. [22]

    Efron, B. (2008). Microarrays, empirical Bayes and the two-groups model. Statistical Science, 23(1), 1--22

    2008

  15. [23]

    Efron, B., & Hastie, T. (2016). Computer Age Statistical Inference: Algorithms, Evidence, and Data Science. Cambridge University Press

    2016

  16. [24]

    Egidi, L., Pauli, F., & Torelli, N. (2022). Avoiding prior-data conflict in regression models via mixture priors. Canadian Journal of Statistics, 50(2), 491--510

    2022

  17. [25]

    Eppo. (2024). Analysis methods. Eppo Documentation. https://docs.geteppo.com/statistics/confidence-intervals/analysis-methods/

    2024

  18. [26]

    GrowthBook. (2024). GrowthBook statistics. GrowthBook Documentation. https://docs.growthbook.io/statistics/overview

    2024

  19. [27]

    Goldberg, D., & Johndrow, J. (2017). A decision-theoretic approach to A/B testing. arXiv preprint, arXiv:1710.03410

    2017 arXiv

  20. [28]

    H., Kadane, J

    Garthwaite, P. H., Kadane, J. B., & O'Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470), 680--701

    2005

  21. [29]

    Gelman, A., & Carlin, J. (2014). Beyond power calculations: assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641--651

    2014

  22. [30]

    B., Stern, H

    Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian Data Analysis (3rd ed.). CRC Press

    2013

  23. [31]

    Gelman, A., Hill, J., & Yajima, M. (2012). Why we (usually) don't have to worry about multiple comparisons. Journal of Research on Educational Effectiveness, 5(2), 189--211

    2012

  24. [32]

    Georgiev, G. Z. (2017). Issues with current Bayesian approaches to A/B testing in conversion rate optimization. Analytics Toolkit Whitepaper

    2017

  25. [33]

    F., Raj K.N., A., & Wagenmakers, E.-J

    Gronau, Q. F., Raj K.N., A., & Wagenmakers, E.-J. (2021). Informed Bayesian inference for the A/B test. Journal of Statistical Software, 100(17). https://doi.org/10.18637/jss.v100.i17

    2021 doi

  26. [34]

    Grünwald, P., de Heide, R., & Koolen, W. (2024). Safe testing. Journal of the Royal Statistical Society: Series B, 86(5), 1091--1128

    2024

  27. [35]

    Hagar, L., & Stevens, N. T. (2025). Design of Bayesian A/B tests controlling false discovery rates and power. Journal of Business & Economic Statistics. arXiv:2312.10814

    2025

  28. [36]

    Hartog, W., & Lei, L. (2023). Multiple A/B testing with always-valid e-values. CODE@MIT 2023

    2023

  29. [37]

    R., Ramdas, A., McAuliffe, J., & Sekhon, J

    Howard, S. R., Ramdas, A., McAuliffe, J., & Sekhon, J. (2021). Time-uniform, nonparametric, nonasymptotic confidence sequences. Annals of Statistics, 49(2), 1055--1080

    2021

  30. [38]

    Howard, J. V. (1998). The 2 2 table: a discussion from a Bayesian viewpoint. Statistical Science, 13(4), 351--367

    1998

  31. [39]

    James, W., & Stein, C. (1961). Estimation with quadratic loss. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability (Vol. 1, pp. 361--379)

    1961

  32. [40]

    Jennison, C., & Turnbull, B. W. (2000). Group Sequential Methods with Applications to Clinical Trials. Chapman & Hall/CRC

    2000

  33. [41]

    Lan, K. K. G., & DeMets, D. L. (1983). Discrete sequential boundaries for clinical trials. Biometrika, 70(3), 659--663

    1983

  34. [42]

    Johari, R., Koomen, P., Pekelis, L., & Walsh, D. J. (2022). Always valid inference: continuous monitoring of A/B tests. Operations Research, 70(3), 1806--1821

    2022

  35. [43]

    Kamalbasha, S., & Perktold, J. (2021). Bayesian A/B testing at Amazon. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 3131--3139

    2021

  36. [44]

    E., & Raftery, A

    Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773--795

    1995

  37. [45]

    E., & Vaidyanathan, S

    Kass, R. E., & Vaidyanathan, S. K. (1992). Approximate Bayes factors and orthogonal parameters, with application to testing equality of two binomial proportions. Journal of the Royal Statistical Society: Series B, 54(1), 129--144

    1992

  38. [46]

    Kessler, R. (2024). Overcoming the winner's curse: leveraging Bayesian inference to improve estimates of the impact of features launched via A/B tests. CODE@MIT 2024

    2024

  39. [47]

    Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press

    2020

  40. [48]

    Larsen, N., Stallrich, J., Sengupta, S., Deng, A., Kohavi, R., & Stevens, N. T. (2023). Statistical challenges in online controlled experiments: a review of A/B testing methodology. The American Statistician, 78(2), 135--149

    2023

  41. [49]

    Lu, Y. (2023). Bayesian sequential experimentation at Amazon. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 5609--5610

    2023

  42. [50]

    Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: reexamining Freedman's critique. Annals of Applied Statistics, 7(1), 295--318

    2013

  43. [51]

    C., & Fleming, T

    O'Brien, P. C., & Fleming, T. R. (1979). A multiple testing procedure for clinical trials. Biometrics, 35(3), 549--556

    1979

  44. [52]

    Pawel, S., & Held, L. (2025). Bayes factor group sequential designs. arXiv preprint, arXiv:2601.02851

    2025 arXiv

  45. [53]

    Pocock, S. J. (1977). Group sequential methods in the design and analysis of clinical trials. Biometrika, 64(2), 191--199

    1977

  46. [54]

    Ramdas, A., Grünwald, P., Vovk, V., & Shafer, G. (2023). Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4), 576--601

    2023

  47. [55]

    Robbins, H. (1970). Statistical methods related to the law of the iterated logarithm. Annals of Mathematical Statistics, 41(5), 1397--1409

    1970

  48. [56]

    Robinson, D. (2015). Is Bayesian A/B testing immune to peeking? Not exactly. http://varianceexplained.org/r/bayesian-ab-testing/

    2015

  49. [57]

    N., Morey, R

    Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E.-J. (2014). Optional stopping: No problem for Bayesians. Psychonomic Bulletin & Review, 21(2), 301--308

    2014

  50. [58]

    N., Speckman, P

    Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225--237

    2009

  51. [59]

    Shiryaev, A. N. (1978). Optimal Stopping Rules. Springer

    1978

  52. [60]

    D., & Wagenmakers, E.-J

    Sch \"o nbrodt, F. D., & Wagenmakers, E.-J. (2018). Bayes factor design analysis: planning for compelling evidence. Psychonomic Bulletin & Review, 25(1), 128--142

    2018

  53. [61]

    Schultzberg, M., Ankargren, S., & Fr nberg, M. (2026). Risk-aware product decisions in A/B tests with multiple metrics. Journal of Statistical Planning and Inference, 245, 106393

    2026

  54. [62]

    Schultzberg, M. (2026). Nobody puts Bonferroni in a corner. arXiv preprint, arXiv:2604.09256

    2026 arXiv

  55. [63]

    G., & Berger, J

    Scott, J. G., & Berger, J. O. (2010). Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem. Annals of Statistics, 38(5), 2587--2619

    2010

  56. [64]

    M., Gronau, Q

    Stefan, A. M., Gronau, Q. F., Sch \"o nbrodt, F. D., & Wagenmakers, E.-J. (2019). A tutorial on Bayes Factor Design Analysis using an informed prior. Behavior Research Methods, 51(3), 1042--1058

    2019

  57. [65]

    Statsig. (2024). Bayesian experiments. Statsig Documentation. https://docs.statsig.com/experiments/advanced-setup/bayesian

    2024

  58. [66]

    Stephens, M. (2017). False discovery rates: a new deal. Biostatistics, 18(2), 275--294

    2017

  59. [67]

    Storey, J. D. (2002). A direct approach to false discovery rates. Journal of the Royal Statistical Society: Series B, 64(3), 479--498

    2002

  60. [68]

    Storey, J. D. (2003). The positive false discovery rate: a Bayesian interpretation and the q -value. Annals of Statistics, 31(6), 2013--2035

    2003

  61. [69]

    Stucchio, C. (2015). Bayesian A/B testing at VWO. VWO Technical Whitepaper

    2015

  62. [70]

    van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press

    1998

  63. [71]

    Ville, J. (1939). \'E tude critique de la notion de collectif . Gauthier-Villars

    1939

  64. [72]

    Wager, S., Du, W., Taylor, J., & Tibshirani, R. J. (2016). High-dimensional regression adjustments in randomized experiments. Proceedings of the National Academy of Sciences, 113(45), 12673--12678

    2016

  65. [73]

    Wang, R., & Ramdas, A. (2022). False discovery rate control with e-values. Journal of the Royal Statistical Society: Series B, 84(3), 822--852

    2022

  66. [74]

    Wang, H., Dandapanthula, S., & Ramdas, A. (2025). Anytime-valid FDR control with the stopped e-BH procedure. arXiv preprint, arXiv:2502.08539

    2025 arXiv

  67. [75]

    Waudby-Smith, I., & Ramdas, A. (2024). Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society: Series B, 86(1), 1--27

    2024

  68. [76]

    Wan, R., Liu, Y., McQueen, J., Hains, D., & Song, R. (2023). Experimentation platforms meet reinforcement learning: Bayesian sequential decision-making for continuous monitoring. In Proceedings of KDD 2023. ACM

    2023

  69. [77]

    Zellner, A. (1986). On assessing prior distributions and Bayesian regression analysis with g -prior distributions. In P. Goel & A. Zellner (Eds.), Bayesian Inference and Decision Techniques (pp. 233--243). North-Holland

    1986

  70. [78]

    J., Blackwell, D., & Girshick, M

    Arrow, K. J., Blackwell, D., & Girshick, M. A. (1949). Bayes and minimax solutions of sequential decision problems. Econometrica, 17(3/4), 213--244

    1949

  71. [79]

    Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14(5), 779--804

    2007

  72. [80]

    F., Dablander, F., & Etz, A

    Wagenmakers, E.-J., Gronau, Q. F., Dablander, F., & Etz, A. (2020). The support interval. Erkenntnis, 85, 589--601

    2020

  73. [81]

    Wald, A. (1948). Sequential Analysis. Wiley

    1948

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.