{"id":"b6177be7-bae5-43cb-8b60-38fe3be67be3","arxiv_id":"2608.12949","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"Bayesian A/B testing is a family of configurations, and the right one depends on whether the program needs error-rate control, accurate estimates, or low regret.","lead":"A team from Spotify's experimentation platform reviews Bayesian A/B testing methods and sorts them into three tiers based on what error guarantees they provide. It argues that the choice of method should depend on which risk the program wants to control, not on whether the method is called Bayesian.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Known-variance gap: Tier 2's anytime-valid FPR bound (Eq. 5) and all simulations (Appendix D) assume known σ², but deployments plug in an estimate; the deployed rule's finite-sample FPR is unexamined, leaving the central 'same rule controls FPR' claim unverified in practice.","rationale":"The paper is a well-structured overview and is unusually candid about its own limitations: it flags the asymptotic extension for estimated variance (Section 2), the FNR bound's blind spot for non-crossing paths (Section 3.2.2 and Appendix F), the fixed-n scope of the Hagar–Stevens FDR formula (Appendix B), the unquantified near-optimality gap for asymmetric costs under continuous priors (Appendix H: 'we expect the gap to be small for moderate asymmetry, though we have not quantified it'), and the three corpus failure modes that break Tier 3 guarantees (Section 3.3.3 and Setting C). I checked the central pieces I could verify analytically: the Ville bound application in Eq. (5) is standard; the flat-prior/naive-peeking equivalence in Section 5.1 is algebraically correct; Appendix G's claim that BF crossing 1/α implies credible-interval exclusion is sound (the inequality 2 log(1/α) > z²_{α/2} holds, and the limiting argument near α→1 is valid); and the per-metric Bayesian FDR bound α(1−p̂) from the mixture e-process is consistent with the simulated per-metric rates in Setting C (the larger experiment-level FDR of ≈0.12–0.13 comes from the ship-if-any rule over K=10 metrics, which the paper states explicitly). The single most load-bearing gap is the one the reader identified: the exact anytime-valid FPR guarantee—the bridge that lets the paper claim the decision-theoretic and frequentist formulations are largely one of parameterization—is proven only for known σ², while the deployed rule must estimate σ², and the paper's own simulations never exercise that deployed rule. I do not contest the asymptotic claim, but 'asymptotically under standard regularity conditions' is not the exact guarantee advertised, and the paper provides no simulation evidence that the finite-sample FPR of the plug-in rule stays near the known-variance rate at the sample sizes where A/B tests actually run. Independent support I credit: the closed-form BF (Eq. 2) is correct, the Appendix H proof for the two-point model is valid, and the simulation designs are detailed enough to be reproduced; the concern is an unexamined extrapolation from the idealized rule to the deployed statistic, not internal inconsistency. The reader's conditions (Monte Carlo error bars; code release or a more precise asymptotic statement) were reasonable; my concrete test sharpens them into a direct simulation of the plug-in-variance rule. If that test shows negligible FPR inflation, the paper's claims stand for practical purposes; if it shows material inflation, the Tier 2 recommendation needs a finite-sample qualifier or an exact variance-robust construction, and the practical content of the central claim is weakened. I also considered the unquantified asymmetric-cost gap in Appendix H as an alternative concern; it is secondary because the paper restricts its near-optimality claim to affine-terminal-payoff costs, discloses the gap, and the symmetric-cost case is argued rigorously.","tokens_in":23885,"tokens_out":24797,"duration_ms":225638,"concrete_test":"Rerun Setting A under H0 (δ=0, continuous monitoring, n_max=5,000, 10,000 replicates) with the known variance in Eq. (2) replaced by the pooled per-group sample variance σ̂²_n recomputed at each look, and measure the empirical directional FPR at the 1/α=20 threshold for per-arm sample sizes n ∈ {50, 100, 200, 500, 1,000} plus the full n_max run. If the empirical FPR stays within Monte Carlo error of the known-σ² rate (0.016–0.019 in Table 2) at every n, the concern is resolved; if it rises toward or above α=0.05 at the sample sizes typical of A/B tests, the Tier 2 guarantee is only asymptotic, and the paper must add a finite-sample correction (e.g., a variance-mixture or safe-t e-process) or qualify the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 states: 'Throughout the paper σ² is treated as known. In practice it is estimated, and the key results (Ville's inequality, the BF closed form) extend asymptotically to the estimated-variance case under standard regularity conditions.' The Tier 2 guarantee that the paper's practical recommendations are built on is Eq. (5): P(sup_n BF10,n ≥ 1/α | H0) ≤ α. This is exact because, with known σ², {BF10,n} is a nonnegative martingale under H0. Every simulation in the paper uses known variance—Appendix D sets 'σ=1 known' for Setting A and 'same normal model as Setting A' for Setting B—so the headline FPR numbers in Tables 2 and 3 are for the idealized rule, not the rule an analyst runs, which must plug in an estimated σ̂²_n. Replacing a nuisance parameter by a data-dependent estimate generally destroys the exact e-process property (the empirical e-value problem): the conditional expectation of the likelihood-ratio increment evaluated at σ̂²_n is not 1 under the true variance, so only asymptotic validity follows. The paper gives no finite-sample evidence on how much the FPR deviates from α when σ̂² is noisy—precisely the regime of n ≈ 100–1,000 per arm where most A/B tests operate—and this is the load-bearing clause of the central claim: 'because the same rule also controls the false positive rate, the choice between a decision-theoretic and a frequentist formulation is largely one of parameterization.' If the plug-in rule's FPR exceeds α materially, practitioners cannot treat the Tier 2 rule as an error-rate-controlled procedure, and the decision-theoretic/frequentist equivalence holds only for an idealized statistic. The paper is careful to quantify Tier 3 degradation under misspecified corpora (Setting C) but brings no analogous scrutiny to the variance assumption, which is violated in every deployment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper provides a systematic overview of Bayesian A/B testing configurations, organizing them into a three-tier hierarchy: Tier 1 (posterior coherence without error-rate control), Tier 2 (Bayes factor stopping with anytime-valid false positive rate control via Ville's inequality), and Tier 3 (empirical Bayes priors for FDR calibration and calibrated shrinkage). The central claims are that Bayes factor stopping is near-optimal for a broad class of decision-theoretic cost functions and, because it also controls the FPR, the choice between decision-theoretic and frequentist formulations is largely one of parameterization. The paper supports these claims with closed-form derivations, an extensive simulation study (Settings A-D), and comparisons against group-sequential tests and always-valid confidence sequences. The main body of the paper is clear and the appendices provide substantial formal detail, though the treatment of estimated variance is limited to an asymptotic statement and several of the headline guarantees are exact only under known variance.","tokens_in":24266,"tokens_out":4587,"duration_ms":50021,"significance":"If the central claims hold, the paper is a valuable reference for practitioners and researchers: it clarifies a frequently conflated set of methods, gives a useful decision matrix (Table 1), provides exact anytime-valid FPR control for Bayes factor stopping (Eq. 5), and offers a well-designed simulation comparison (Tables 2-3, Figures 2-9) showing the trade-offs among error rates, estimation accuracy, and regret. The paper also gives credit where it is due by making the equivalence between mSPRT and Bayes factor stopping explicit, and by documenting the corpus-quality failure modes for empirical Bayes priors. The appendices contain several rigorous derivations, including the Bellman argument for BF threshold rules (Appendix E), the prior-averaged FNR bound (Appendix F), and the proof that BF stopping implies credible-interval exclusion (Appendix G). However, the load-bearing claim that the same rule controls FPR in realistic deployments is only established under known variance; the extension to estimated variance is asserted asymptotically and not examined in finite samples.","major_comments":[{"comment":"The central anytime-valid FPR guarantee is stated and simulated only under known variance. The text in Section 2 says σ² is treated as known and that the key results 'extend asymptotically to the estimated-variance case under standard regularity conditions,' but no finite-sample analysis is provided. In practice, the analyst plugs in an estimate σ̂²_n, and the exact e-process/martingale property of the BF sequence can be lost; the conditional expectation of the likelihood-ratio increment evaluated at an estimated variance is not generally 1 under the true variance. Since all simulations in Appendix D set σ=1 known (Settings A and B) and the headline FPR numbers in Tables 2 and 3 are for the idealized rule, the claim that 'the same rule also controls the false positive rate' in deployed A/B testing is not demonstrated. Please provide finite-sample simulation evidence with estimated variance, or an exact anytime-valid modification (e.g., variance estimation on a separate fraction of data), or explicitly scope the FPR guarantee to the known-variance setting throughout the abstract and Section 3.","section":"Section 2, Eq. (5), and Section 6"},{"comment":"The assertion that 'Bayes factor stopping is near-optimal for a broad class of cost functions' is not as well quantified as the abstract suggests. Appendix H proves exact BF-threshold optimality only under the two-point model, and for the continuous normal prior the proof covers the symmetric-cost case exactly, while for asymmetric costs the single BF threshold is described as a 'constrained approximation' with a gap that is not quantified. The numerical example for Wan et al. is informative but is a single parameter configuration. Please either provide a quantitative bound on the suboptimality gap for asymmetric costs under the normal prior, or temper the near-optimality claim in the abstract and Section 1 to reflect the conditions under which it has been established.","section":"Appendix H, Sections 2.3 and 3.2.2"},{"comment":"The Tier 3 FDR guarantee is supported mainly by a posterior-odds argument and an appeal to Storey (2003), but the paper does not provide a full frequentist FDR proof for the sequential stopping rule. Equation (6) is a fixed-sample Storey approximation, and the statement 'the (1−p̂) floor ensures that a rejected metric has posterior null probability at most (1−p̂)α' is a conditional Bayesian statement rather than an unconditional frequentist FDR bound at the random stopping time. The simulation evidence in Setting C is helpful, but the theoretical claim of 'per-metric FDR calibration' would be stronger with a precise statement of the conditions under which the Bayesian and frequentist FDR coincide at stopping times, or with a counterexample showing where they can diverge under prior misspecification.","section":"Section 3.3.1 and Appendix B"}],"minor_comments":[{"comment":"Typo: 'Non max need be committed to in advance' should read 'No n_max need be committed to in advance.'","section":"Section 3.2.1"},{"comment":"The heading 'Acknowlegements' is misspelled.","section":"Acknowledgements"},{"comment":"The caption omits the value of δ or the context for the boundary; please clarify that the figure is for the null hypothesis with known σ=1 and V=0.1.","section":"Figure 1"},{"comment":"In the reference list, 'Deng, A., et al. (2023)' uses an incomplete author list; please list all authors or use the journal's abbreviation style consistently.","section":"References"},{"comment":"The row label 'Flat Expected loss' is a bit terse; consider writing 'Flat prior with expected-loss stopping (ε=0.02)' to match the column header 'Stopping rule'.","section":"Table 2"},{"comment":"The statement that the mSPRT 'is exactly the Bayes factor under the same prior' is correct for the normal model used, but it would be helpful to note explicitly that the equivalence holds for the specific mixture martingale used by Johari et al. (2022) and not for all mSPRT variants.","section":"Section 5.2"},{"comment":"The sentence 'Non max need be committed to in advance' appears to be a typographical artifact; the intended phrase is likely 'No maximum sample size needs to be committed to in advance.'","section":"Section 3.3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of stat.ME and provides a useful synthesis. The main concern is that the FPR guarantee is exact only under known variance, while the practical recommendations and simulations use the known-variance rule; this is a load-bearing issue for the central claim. The near-optimality and FDR claims are also somewhat stronger than the formal results in the appendices. If the authors can add finite-sample estimated-variance simulations (or clearly scope the claims), quantify the near-optimality gap, and tighten the FDR statement, the paper would be a solid contribution. The self-citations appear related but not circular, and I do not see a novelty or scope problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one if you care about how Bayesian methods are actually used in A/B testing platforms. The paper's contribution is organizational but valuable: it sorts configurations into three tiers—posterior coherence, Bayes-factor stopping with bounded FPR, and empirical Bayes for FDR and shrinkage—and gives a decision matrix (Table 1) that practitioners could actually use. The equivalence results (mSPRT=BF, Ville's inequality, Storey FDR approximation) are not new, but the paper frames them in a way that exposes a real conflation in the industry: posterior probability thresholds are not error control. The simulation study is careful, and the failure modes for empirical Bayes corpora (winner-selected, pooled programs, heterogeneous metrics) are honestly spelled out. That is more than most overviews do.\n\nWhere I'd push back: the known-variance assumption. The paper states up front that σ² is treated as known and that results extend asymptotically when estimated, but every simulation in Table 2 and Table 3 uses known σ². The anytime-valid FPR bound is exact for the martingale with the true variance; plugging in an estimated σ̂² generally breaks the e-process property in finite samples. For an audience running tests at n≈100–1,000 per arm, that is exactly the regime where variance estimation error is not negligible. The paper gives no finite-sample evidence on how much the FPR deviates from α with estimated variance, and the central claim that the BF rule \"controls the false positive rate\" in real deployments is therefore not actually verified. This is not fatal—the asymptotic argument is standard and the paper is transparent—but it deserves a simulation or at least a sharper statement of when the plug-in rule is approximately anytime-valid.\n\nAlso, the simulations would benefit from Monte Carlo standard errors; some FPR differences (0.011 vs 0.013) are within noise without error bars. And I'd like to see code, or at least enough detail to reproduce the GST boundary computation.\n\nWho is this for? Any statistician or engineer designing an experimentation platform, and anyone writing about Bayesian A/B testing who wants a clear taxonomy instead of the usual \"Bayesian can do everything\" pitch. It deserves a serious referee—the synthesis is solid, the tables are largely reproducible in spirit, and the known-variance gap is addressable with modest revisions. I'd accept it into review.\n\nSincerely.","headline":"A useful industry-oriented synthesis of Bayesian A/B testing configurations, with a real but fixable gap: the anytime-valid FPR guarantee is exact only under known variance, and all simulations assume it.","tokens_in":24844,"tokens_out":2762,"would_cite":true,"duration_ms":27474,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","62L10","62C10","62L15"],"pacs":[],"model":"deepseek-v4-flash","headline":"Bayes factor stopping is the pivotal A/B testing configuration: it bounds false positives at every stopping time and is near-optimal for a broad class of costs.","keywords":["Bayesian A/B testing","Bayes factor","optional stopping","empirical Bayes","false discovery rate","sequential testing","anytime-valid inference","expected loss"],"falsifier":"Simulate continuous monitoring of the two-group normal test with the variance estimated at each look, stop by $\\mathrm{BF}_{10} > 1/\\alpha$ under the null, and count how often the treatment is declared superior; if the empirical false positive rate exceeds $\\alpha$ by more than Monte Carlo error at finite sample sizes, the anytime-valid guarantee the paper relies on does not hold exactly in the estimated-variance deployment.","tokens_in":23691,"feed_emoji":"📊","tokens_out":6724,"duration_ms":57510,"temperature":0.7,"pith_summary":"Bayesian A/B testing is routinely discussed as a single method, but the paper argues it is a family of (prior, stopping rule) configurations with different statistical guarantees, and it organizes them into three tiers. The central result is that Bayes factor stopping under any proper prior both keeps the false positive rate below alpha at every stopping time and is near-optimal for a broad class of cost functions used in A/B testing, so the frequentist-versus-Bayesian choice is largely one of parameterization. The paper also claims that calibrated shrinkage and false discovery rate control require an empirical Bayes prior estimated from a representative historical corpus, and that certain corpus defects make that calibration impossible no matter how much data is collected. A sympathetic reader should care because these distinctions determine which risks a real experimentation program actually controls: error rates, estimation accuracy, and regret are different risks, and the appropriate configuration follows from the risk being managed.","feed_headline":"Bayes-factor stopping keeps A/B false positives capped while you peek","feed_subtitle":"A three-tier hierarchy separates posterior coherence from error-controlled stopping and calibrated shrinkage.","key_machinery":"The load-bearing object is the Bayes factor sequence $\\mathrm{BF}_{10,n}$ computed under a proper prior, which is a nonnegative martingale under the null hypothesis; Ville's inequality converts the stopping threshold $1/\\alpha$ into an unconditional anytime-valid bound $P(\\sup_n \\mathrm{BF}_{10,n} \\ge 1/\\alpha \\mid H_0) \\le \\alpha$, so the rule can be checked after every observation without inflating the false positive rate. The same likelihood-ratio structure makes the Bayes factor an optimal decision statistic for cost functions with affine terminal payoffs, and the empirical Bayes mixture $\\Lambda_n = (1-p) + p \\cdot \\mathrm{BF}_{10,n}$ adds a $(1-p)$ floor that yields per-metric false discovery rate control through a posterior-odds argument while inheriting the martingale property.","core_discovery":"The paper's central claim is that the most common Bayesian A/B testing configurations form a three-tier hierarchy, and that the middle tier - Bayes factor stopping with any proper prior - is the pivotal one: it simultaneously controls the false positive rate at any stopping time through Ville's inequality and is near-optimal for sequential launch decisions whose terminal payoffs are affine in the posterior. As a result, the debate between decision-theoretic and frequentist formulations of A/B testing is largely a debate about parameterization, not about substance. The paper further claims that Tier 3, which adds per-metric false discovery rate calibration and calibrated shrinkage, is reachable only with an empirical Bayes mixture prior estimated from a representative corpus of historical experiments, and that winner-selected corpora, pooled programs, and heterogeneous metrics prevent calibration regardless of corpus size. Simulations show that flat-prior posterior-probability stopping reproduces naive peeking, that an oracle empirical Bayes prior achieves the lowest estimation error, and that expected-loss stopping minimizes regret only when shipping a null-effect variant is costless.","pith_inferences":["If Bayes factor thresholding is near-optimal for any affine terminal cost, then business costs could be mapped directly into Bayes factor thresholds, letting platforms set stopping rules from customer-impact utilities rather than from a nominal alpha.","The paper's equivalence of the mSPRT mixing distribution and a Bayesian prior suggests that frequentist always-valid procedures could close the mean squared error gap by applying shrinkage estimators at stopping, a practice the paper notes is allowed but does not itself test.","The corpus failure modes imply that any organization-level empirical Bayes prior should be validated for exchangeability across programs and metric types before deployment; otherwise no amount of historical data fixes the inflated false positive rate that winner selection or pooling introduces.","A direct extension would measure the regret crossover point in a real experimentation program, since the simulation's crossover at shipping cost about 0.003 depends on the assumed effect-size distribution."],"forward_implications":["A platform that defaults to flat-prior posterior-probability stopping is running the same procedure as naive peeking, with false positive rate around 0.30 under continuous monitoring, so Bayesian branding alone confers no error-rate protection.","Bayes factor stopping with any proper prior offers both decision-theoretic near-optimality for affine terminal payoffs and unconditional false positive rate control at every stopping time, so for that cost class the Bayesian/frequentist choice is a parameterization choice.","Tier 3 benefits, per-metric false discovery rate control and calibrated shrinkage, require an empirical Bayes prior estimated from a representative corpus of roughly 200 historical experiments; below that, a program should stay at Tier 2 with explicit multiple-testing corrections.","Expected-loss stopping wins on expected regret only when shipping a null-effect variant is essentially free; once the shipping cost exceeds about 2% of a typical true effect, evidence-requiring rules such as Bayes factor thresholding or group-sequential designs dominate.","Relative to group-sequential testing, Bayes factor stopping needs no maximum sample size commitment and yields lower mean squared error from shrinkage, but lower power; the choice between them should follow which operational risk the program controls."],"supporting_citations":[{"why":"Supplies the martingale inequality that converts a Bayes factor threshold into an anytime-valid false positive bound.","marker":"(Ville, 1939)"},{"why":"Establishes the mSPRT as the Bayes factor under a mixing prior and demonstrates the anytime-valid martingale property, forming the frequentist bridge.","marker":"(Johari et al., 2022)"},{"why":"Provides the sequential decision-theoretic result that optimal policies for affine terminal costs are Bayes factor threshold rules.","marker":"(Arrow et al., 1949)"},{"why":"Introduces the empirical Bayes mixture prior for A/B testing that the Tier 3 framework builds on.","marker":"(Deng, 2015)"},{"why":"Extends Bayesian optional-stopping monitoring and supplies the posterior-odds false discovery rate argument.","marker":"(Deng et al., 2016)"},{"why":"Provides Bayes Factor Design Analysis, the simulation-based power planning method used for stopping rules.","marker":"(Schönbrodt and Wagenmakers, 2018)"},{"why":"The always-valid confidence sequence baseline that the simulations compare against.","marker":"(Waudby-Smith and Ramdas, 2024)"},{"why":"Defines the group-sequential testing baseline that requires a maximum sample size commitment.","marker":"(O'Brien and Fleming, 1979)"},{"why":"Defines expected-loss stopping, the Tier 1 decision-theoretic rule whose loss function is not affine in the posterior.","marker":"(Stucchio, 2015)"},{"why":"Supplies the false discovery rate approximation and posterior-odds interpretation used for Tier 3 calibration.","marker":"(Storey, 2003)"}],"fun_headline_variants":["Bayes factor stopping: near-optimal A/B peeking with capped false positives","Three-tier Bayesian A/B testing: pick your risk, then your stopping rule","Empirical Bayes prior: the only route to calibrated A/B testing","Bayes factor stopping is near-optimal for A/B launch decisions","Which risk matters? A/B testing has three tiers of Bayesian control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The exact anytime-valid false positive bound and the Bayes factor closed form assume the per-observation variance is known; in real deployments variance is estimated, so the guarantees hold only asymptotically under regularity conditions.","fun_headline_variants_meta":{"raw":{"variants":["Bayes factor stopping: near-optimal A/B peeking with capped false positives","Three-tier Bayesian A/B testing: pick your risk, then your stopping rule","Empirical Bayes prior: the only route to calibrated A/B testing","Bayes factor stopping is near-optimal for A/B launch decisions","Which risk matters? A/B testing has three tiers of Bayesian control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001297,"raw_usage":{"total_tokens":5325,"prompt_tokens":1006,"completion_tokens":4319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":4223}},"tokens_in":622,"tokens_out":4319,"duration_ms":28038,"temperature":1.0,"reasoning_tokens":4223,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:53:08.249494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate continuous monitoring of the two-group normal test with the variance estimated at each look, stop by $\\mathrm{BF}_{10} > 1/\\alpha$ under the null, and count how often the treatment is declared superior; if the empirical false positive rate exceeds $\\alpha$ by more than Monte Carlo error at finite sample sizes, the anytime-valid guarantee the paper relies on does not hold exactly in the estimated-variance deployment.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the martingale inequality that converts a Bayes factor threshold into an anytime-valid false positive bound."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the mSPRT as the Bayes factor under a mixing prior and demonstrates the anytime-valid martingale property, forming the frequentist bridge."},{"cited_title":"J., Blackwell, D., & Girshick, M","cited_arxiv_id":null,"evidence_quote":"Provides the sequential decision-theoretic result that optimal policies for affine terminal costs are Bayes factor threshold rules."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the empirical Bayes mixture prior for A/B testing that the Tier 3 framework builds on."},{"cited_title":"Continuous Monitoring of A/B Tests without Pain: Optional Stopping in Bayesian Testing","cited_arxiv_id":"1602.05549","evidence_quote":"Extends Bayesian optional-stopping monitoring and supplies the posterior-odds false discovery rate argument."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The always-valid confidence sequence baseline that the simulations compare against."},{"cited_title":"C., & Fleming, T","cited_arxiv_id":null,"evidence_quote":"Defines the group-sequential testing baseline that requires a maximum sample size commitment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines expected-loss stopping, the Tier 1 decision-theoretic rule whose loss function is not affine in the posterior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the false discovery rate approximation and posterior-odds interpretation used for Tier 3 calibration."}],"review_version":1}