Carefree multiple testing with e-processes

Jelle J. Goeman; Rianne de Heide; Yury Tavyrikov

REVIEW 3 major objections 3 minor 1 cited by

Carefree multiple testing with e-processes

T0 review · 3 major / 3 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Carefree multiple testing with e-processes should operate on running maxima, and adjusters are needed to keep e-BH's FDR control under arbitrary dependence.

desk verdict A useful carefree criterion for e-process multiple testing, with a credible but simulation-only counterexample and a clean adjuster fix that needs a proof cleanup. read the letter →

arxiv 2501.19360 v2 pith:6L7Y24CP submitted 2025-01-31 math.ST stat.MEstat.TH

Yury Tavyrikov , Jelle J. Goeman , Rianne de Heide This is my paper · ORCID

classification math.STstat.MEstat.TH MSC 62L1062F0362J15

keywords e-processese-BHprocedurefalsediscoveryrateFDR-suprunningmaximumcarefreedatagatheringadjustersanytime-validinference

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

E-processes let a researcher keep collecting data and still test a hypothesis at any stopping time, but the paper argues that standard multiple testing methods are not 'carefree': gathering more data for one hypothesis can erase an earlier rejection. The paper's solution is to make methods work with the running maximum of each e-process, which forces the rejection set to grow monotonically. Its negative result is that e-BH applied to these raw running maxima does not control the false discovery rate under arbitrary dependence; a two-hypothesis counterexample drives the FDR to about 1.08α instead of α, and the same example breaks averaging-based FWER control and closed e-BH. Its positive result is that an increasing 'adjuster' function maps each running maximum back to an e-process, and e-BH applied to the adjusted maxima controls the FDR-sup at K0α/K for any dependence structure. The price is a measurable loss of power, which the paper suggests can be partly recovered by conditional calibration or stochastic rounding.

What carries the argument

The machinery has three parts. (i) The running maximum M_t^k = sup_{s≤t} E_s^k is what makes data gathering carefree: rejections based on M_t can only grow, and the researcher can always revert an over-collected e-process to its best earlier value. (ii) The e-BH procedure, which rejects the k largest e-values above the threshold K/(α k), provides FDR control for stopped e-values but not for their running maxima. (iii) An admissible adjuster—an increasing right-continuous function A: [1,∞]→[0,∞] with ∫_1^∞ A(E)/$E^{2}$ dE = 1—restores the e-process property, because A applied to a running maximum of an e-process is again an e-process. The paper proves Theorem 1 by plugging adjusted maxima into the e-BH argument and bounding each null's contribution by α/K times E[A(sup_t E_t)] ≤ 1.

What would settle it

Compute, for the Section 3.1 processes, E[E_τ] for every stopping time τ, especially τ that stops when the running maximum first reaches the rejection threshold; if any such expectation exceeds 1, the processes are not valid e-processes under the standard definition and the counterexample does not apply. Alternatively, rerun the same construction with E0=1 fixed for both processes and check whether the simulated FDR of running-maximum e-BH still exceeds α.

Watch

Extended reading notes

Core claim

The discovery is a matched pair of statements about FDR control for sequential e-values. First, the running maximum M_t^k = max_{s≤t} E_s^k is the natural input for a carefree multiple testing method, because rejecting based on M_t makes the rejection set non-decreasing over time. Second, despite the intuitive appeal of replacing each e-process by its running maximum, e-BH on these maxima does not control the FDR under arbitrary dependence: the paper's explicit two-process counterexample gives simulated FDR ≈ 1.08α instead of α. Third, the same counterexample breaks averaging-based FWER control and the improved 'closed e-BH'. The fix is an admissible adjuster A, a right-continuous increasing function with ∫_1^∞ A(E)/$E^{2}$ dE = 1; A(M_t^k) is again an e-process, and e-BH on these adjusted running maxima controls the FDR-sup at K0α/K for any dependence structure.

Load-bearing premise

The counterexample's processes start from random initial values 0 or 1/(2α) rather than from the conventional starting value 1, so the claim that they are valid e-processes rests on accepting the paper's expectation-based definition E[E_τ]≤1 for all stopping times; if the stricter E0=1 definition is required, the counterexample falls outside the class it is supposed to refute.

Editorial extensions

If this is right

Any e-based multiple testing procedure intended for sequential data collection should be evaluated by FDR-sup, the supremum over time of the FDR of the running maxima, rather than by the FDR at each individual stopping time.
The two-hypothesis counterexample shows that raw running-maximum e-BH, averaging, and closed e-BH can all exceed their nominal guarantees under dependence, so none of these is a valid carefree procedure without further correction.
Applying an admissible adjuster to each running maximum before e-BH controls the FDR-sup at K0α/K under arbitrary dependence, making the procedure carefree.
The price of adjustment is real power loss; the paper notes that conditional calibration or stochastic rounding may recover some of that power, so adjustment need not be the final word on efficiency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If 'carefree' is adopted as a general normative standard, then any online or group-sequential FDR method should be judged by its FDR-sup; this reframes the design question from 'which data to collect' to 'which adjuster to choose'.
The counterexample's structure—rare large initial values driving the running maximum—suggests that the violation is driven by the heaviness of the initial jump; one could test whether restricting e-processes to have bounded increments restores raw running-maximum control.
The admissible-adjuster condition has a one-parameter family of choices; a natural next step is to search for adjusters that maximize average power under a given dependence class, rather than the two examples A1 and A2.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper introduces a sensible criterion for multiple testing with e-processes: carefree in the sense that the rejection set never shrinks when more data is collected, formalized as FDR-sup control on running maxima. That framing is genuinely new, and the negative result—e-BH on running maxima can exceed α under arbitrary dependence, roughly 1.08α in their two-hypothesis example—plus the fix via adjusters, is a useful clarification for the e-value community. The positive theorem (e-BH on adjusted running maxima controls FDR-sup at K0α/K) follows in a few lines from the Wang–Ramdas e-BH theorem and the Choe–Ramdas adjuster property. The paper knows this; its contribution is identifying the right criterion and showing that the naive method fails.

The main soft spot is that Proposition 1 rests on a Monte Carlo estimate rather than an exact computation. The simulation is large (1e6 iterations, standard error ~0.001), so the conclusion is credible, but a proof-oriented reader will want either an exact counterexample or a formal statement of what is shown. Relatedly, the counterexample starts from random initial values that can be zero, which conflicts with the introduction's "E0=1" sentence. The formal definition in Section 2.2 only requires E[Eτ]≤1 for every stopping time, and the construction satisfies that by optional stopping, so the counterexample is inside the paper's own class. Still, the inconsistency invites the exact objection your reader raised; the authors should align the intro with the formal definition. The appendix proof of Theorem 1 also drops the adjuster in the key displayed inequality; the prose explains the missing step, so it is fixable, but it should be cleaned up.

On the citation side, the paper is honest about the related literature, including the concurrent work on closed e-BH and the FDR principle. The power loss from adjustment is acknowledged as a drawback, which is good. No overclaiming.

Who is this for? Anyone doing anytime-valid multiple testing or working on e-values. The carefree/FDR-sup perspective is worth taking seriously even if the positive theorem is a short combination. I would send it to a serious referee. With a repaired proof and better exactness it could become a standard reference.

Referee Report

3 major / 3 minor

Summary. The paper argues that multiple testing with e-processes should be 'carefree', meaning that gathering more data never makes a previously rejected hypothesis lose its rejected status, and that this requires working with running maxima of e-processes. It claims that e-BH applied to running maxima can fail to control the FDR under arbitrary dependence (Proposition 1), that closed e-BH and averaging running maxima also fail (Corollaries 1 and 2), and that applying an adjuster to the running maxima restores FDR-sup control at level K0*alpha/K under arbitrary dependence (Theorem 1). The positive result is essentially a combination of the known adjuster property and e-BH FDR control; the negative result is supported by a counterexample evaluated by Monte Carlo simulation rather than by an exact proof.

Significance. If the negative claim were rigorously established and the FDR-sup criterion accepted, the paper would provide a useful caution against naive use of e-BH on running maxima and a simple, implementable fix: adjusted running maxima behave as e-processes and therefore inherit e-BH's FDR guarantee. The paper is clearly written and provides reproducible code for the simulations and for Figure 1. However, the central negative result is not currently proved, and the appendix proof of Theorem 1 is not self-consistent as displayed; for a math.ST paper these gaps need to be repaired before the claims can be accepted.

major comments (3)

[Section 3.1 (Proof of Proposition 1)] The counterexample is not proved. The proof states that 'the FDR is evaluated numerically' and reports a Monte Carlo estimate FDR approximately 1.08*alpha with a standard error of 0.001; a simulation, even with M = 10^6 iterations, does not establish that FDR-sup exceeds alpha for the described processes, so Proposition 1 as a mathematical claim lacks support. In addition, the constructed processes start from (X0^1, X0^2) equal to (0,0) with probability 1-2*alpha and (1/(2*alpha), 1/(2*alpha)) with probability 2*alpha, so E0^1 is not equal to 1 almost surely, contrary to the definition in the Introduction that an e-process 'starts at a value of E0 = 1'; if the looser formal definition in Section 2.2 (E[E_tau] <= 1 for all stopping times) is intended, the paper should say so and reconcile the two definitions. Finally, the text verifies only E[E_t] = 1 at fixed times, not the required E[E_tau] <= 1 for arbitrary stopping times, so even under the Section 2.2 definition the e-process validity is asserted without proof.
[Appendix A (Proof of Theorem 1)] The displayed chain (6)-(8) is written for the unadjusted running maximum sup_{t<=s} e^j_t; the indicator in (6) and the expectation in (8) do not contain the adjuster A. The prose then says the last expectation would be bounded if an adjuster were applied, but the displayed argument is not rewritten accordingly. As printed, the appendix does not prove Theorem 1. The theorem is true by the known facts that A(M^k_t) is an e-process and e-BH controls the FDR for e-values at each fixed time; the authors should replace the display with this argument or carefully insert the adjuster throughout the proof.
[Sections 3.2 and 3.3] Corollaries 1 and 2 inherit the gap in Proposition 1. The uniform improvement result of Goeman et al. and the averaging argument would transfer a proven violation, but since the violation is currently only a Monte Carlo simulation rather than a proof, these corollaries are not established. The authors should either provide an exact proof of the counterexample or explicitly present the negative results as numerical evidence rather than as propositions/corollaries.

minor comments (3)

[Section 5] The conclusion that 'carefree multiple testing procedures must operate on the running maxima' is stronger than the analysis supports; the paper shows that running maxima yield a sufficient carefree property and that e-BH fails on them in the proposed counterexample, not that every carefree procedure must use running maxima.
[Various] There are typographical slips: 'Acknowlegdgemenets' in the acknowledgements, 'counterexpample.py' in Appendix B.1, and 'The stopped e-BH procedure' in Procedure 2 should read 'running maximum e-BH procedure'.
[Appendix A, Eq. (6)] Equation (6) contains an unexplained 1/n factor in the displayed FDR-sup formula; this appears to be a typographical artifact and should be removed or explicitly defined.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central results rest on externally established theorems and a constructed counterexample, not on fitted inputs or self-referential definitions.

full rationale

The paper's main derivation chain contains no circular step. Theorem 1 controls FDR-sup by combining two externally established ingredients: the e-BH FDR guarantee of Wang and Ramdas (2022) and the Choe and Ramdas (2024) adjuster theorem that turns a running maximum of an e-process into an e-process. Neither ingredient is assumed from the paper's own conclusions, and no fitted parameter is relabeled as a prediction. Proposition 1 is a counterexample rather than a fitted claim; its possible weakness regarding the random initial value E0 and its Monte Carlo confirmation are correctness or rigor concerns, not circularity, because the negative claim is not defined in terms of its own conclusion and does not rely on the paper's own theorems. The only self-citation appears in Corollary 1, where the uniform-improvement result of Goeman et al. (2025), coauthored by de Heide, is used to extend Proposition 1 to closed e-BH. That step is secondary and does not support the central FDR-sup theorem; moreover, the cited result is presented as a theorem with its own proof, not as an unverified premise imported to force the conclusion. There is no renaming of a known result as organization, and no ansatz is smuggled in via citation. The derivation is therefore self-contained with respect to the main claims, and no significant circularity is found.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central theorem is assembled from two external pillars: e-BH FDR control under arbitrary dependence and the adjuster property that A(M) is an e-process. The common filtration assumption is explicit. The counterexample is hand-constructed with no data fitting; its parameters are chosen to show the failure. No new entities are introduced. Because the core support is external and independently established, the non-circularity of the main claim is high despite minor self-citation in Corollary 1.

free parameters (3)

initial nonzero probability 2 alpha = 2 alpha
Hand-selected in the counterexample so the two processes are occasionally initialized at 1/(2 alpha); this scaling drives the FDR excess.
increment probabilities 1/3 each = 1/3
Chosen to give each coordinate a nonnegative martingale with mean 1 and to create the dependence that breaks e-BH on running maxima.
initial nonzero value 1/(2 alpha) = 1/(2 alpha)
Set so that the e-BH rejection thresholds are hit with the probabilities that make the FDR exceed alpha.

assumptions (5)

standard math e-BH controls FDR under arbitrary dependence among e-values
Used in Fact 1 and Theorem 1; from Wang and Ramdas (2022), cited.
domain assumption For an admissible adjuster A satisfying the integral condition, A(M_t) is an e-process when M_t is the running maximum of an e-process
Core of Theorem 1; from Choe and Ramdas (2024), cited, not re-proven.
domain assumption All e-processes are adapted to a common filtration
Stated in Section 2.2 as required for joint validity under stopping.
domain assumption The constructed processes E^1_t and E^2_t satisfy E[E_tau] <= 1 for arbitrary stopping times
Asserted in Section 3.1 by checking E[E_t] = 1; relies on optional stopping for nonnegative martingales and does not address E0 = 1.
domain assumption Closed e-BH uniformly improves standard e-BH
Used in Corollary 1; from Goeman et al. (2025), a paper with overlapping authorship.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Carefree multiple testing with e-processes." pith.science (2026). https://pith.science/paper/6L7Y24CP

@misc{pith2026250119360,
  author       = {Pith},
  title        = {Pith review of: Carefree multiple testing with e-processes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6L7Y24CP}},
  note         = {Machine review of arXiv:2501.19360}
}

abstract

E-processes enable hypothesis testing with ongoing data collection while maintaining Type I error control. However, when testing multiple hypotheses simultaneously, current $e$-value based multiple testing methods such as e-BH are not invariant to the order in which data are gathered for the different $e$-processes. This can lead to undesirable situations, e.g., where a hypothesis rejected at time $t$ is no longer rejected at time $t+1$ after choosing to gather more data for one or more $e$-processes unrelated to that hypothesis. We argue that multiple testing methods should always work with suprema of $e$-processes. We provide an example to illustrate that e-BH does not control the FDR, at level $\alpha$ when applied to suprema of $e$-processes. From the same example we see that the FWER is not controlled with averaging, and also closed e-BH does not control the FDR. We show that adjusters can be used to ensure FDR-sup control with e-BH under arbitrary dependence.

Figures

Figures reproduced from arXiv: 2501.19360 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Anytime-valid FDR control with the stopped e-BH procedure
stat.ME 2025-02 accept novelty 6.0 of 10

Stopped e-BH controls FDR at all stopping times when the underlying e-processes are global, and local e-processes become global under a no-unobserved-confounding Markov condition.

Reference graph

Works this paper leans on

20 extracted references · 11 canonical work pages · cited by 1 Pith paper

[1]

Y. J. Choe and A. Ramdas. Combining Evidence Across Filtrations . 0 (arXiv:2402.09698), Feb. 2024

arXiv 2024
[2]

A. P. Dawid, S. de Rooij , P. Grunwald, W. M. Koolen, G. Shafer, A. Shen, N. Vereshchagin, and V. Vovk. Probability-free pricing of adjusted American lookbacks. 0 (arXiv:1108.4113), Aug. 2011 a . doi:10.48550/arXiv.1108.4113

work page Pith review arXiv doi:10.48550/arxiv.1108.4113 2011
[3]

A. P. Dawid, S. de Rooij , G. Shafer, A. Shen, N. Vereshchagin, and V. Vovk. Insuring against loss of evidence in game-theoretic probability. Statistics & Probability Letters, 81 0 (1): 0 157--162, Jan. 2011 b . ISSN 0167-7152. doi:10.1016/j.spl.2010.10.013

work page doi:10.1016/j.spl.2010.10.013 2011
[4]

Fischer and A

L. Fischer and A. Ramdas. Online closed testing with e-values. 0 (arXiv:2407.15733), July 2024 a

arXiv 2024
[5]

Fischer and A

L. Fischer and A. Ramdas. An online generalization of the e- BH procedure. 0 (arXiv:2407.20683), July 2024 b . doi:10.48550/arXiv.2407.20683

work page doi:10.48550/arxiv.2407.20683 2024
[6]

Fischer, M

L. Fischer, M. Bofill Roig, and W. Brannath. The online closure principle. The Annals of Statistics, 52 0 (2), Apr. 2024. ISSN 0090-5364. doi:10.1214/24-AOS2370

work page doi:10.1214/24-aos2370 2024
[7]

The e-Partitioning Principle of False Discovery Rate Control

J. Goeman, R. de Heide, and A. Solari. The e-partitioning principle of false discovery rate control, 2025. URL https://arxiv.org/abs/2504.15946

work page Pith review arXiv 2025
[8]

P. D. Gr \"u nwald. Beyond neyman--pearson: E-values enable hypothesis testing with a data-driven alpha. Proceedings of the National Academy of Sciences, 121 0 (39): 0 e2302098121, 2024

work page 2024

Show all 20 references

[9]

Hartog and L

W. Hartog and L. Lei. Family-wise error rate control with e-values, 2025. URL https://arxiv.org/abs/2501.09015

2025 arXiv
[10]

N. W. Koning. Post-hoc \ \ Hypothesis Testing and the Post-hoc \ p\ -value. 0 (arXiv:2312.08040), Sept. 2024. doi:10.48550/arXiv.2312.08040

2024 doi
[11]

Lee and Z

J. Lee and Z. Ren. Boosting e- BH via conditional calibration. 0 (arXiv:2404.17562), Apr. 2024. doi:10.48550/arXiv.2404.17562

2024 doi arXiv
[12]

Ramdas and R

A. Ramdas and R. Wang. Hypothesis testing with e-values. 0 (arXiv:2410.23614), Nov. 2024. doi:10.48550/arXiv.2410.23614

2024 doi arXiv
[13]

S. K. Sarkar, A. Chen, L. He, and W. Guo. Group sequential BH and its adaptive versions controlling the FDR . Journal of Statistical Planning and Inference, 199: 0 219--235, 2019. ISSN 0378-3758. doi:https://doi.org/10.1016/j.jspi.2018.07.001. URL https://www.sciencedirect.com...

2019 doi
[14]

Shafer, A

G. Shafer, A. Shen, N. Vereshchagin, and V. Vovk. Test Martingales , Bayes Factors and p- Values . Statistical Science, 26 0 (1): 0 84--101, Feb. 2011. ISSN 0883-4237, 2168-8745. doi:10.1214/10-STS347

2011 doi
[15]

H. Wang, S. Dandapanthula, and A. Ramdas. Anytime-valid fdr control with the stopped e-bh procedure, 2025. URL https://arxiv.org/abs/2502.08539

2025 arXiv
[16]

R. Wang. The only admissible way of merging arbitrary e-values. arXiv preprint arXiv:2409.19888, 2024

2024 arXiv
[17]

Wang and A

R. Wang and A. Ramdas. False Discovery Rate Control with E-values . Journal of the Royal Statistical Society Series B: Statistical Methodology, 84 0 (3): 0 822--852, July 2022. ISSN 1369-7412. doi:10.1111/rssb.12489

2022 doi
[18]

Xu and A

Z. Xu and A. Ramdas. Online multiple testing with e-values. 0 (arXiv:2311.06412), Nov. 2023

2023 arXiv
[19]

Xu and A

Z. Xu and A. Ramdas. More powerful multiple testing under dependence via randomization. 0 (arXiv:2305.11126), Apr. 2024. doi:10.48550/arXiv.2305.11126

2024 doi
[20]

Z. Xu, L. Fischer, and A. Ramdas. Bringing closure to fdr control: beating the e-benjamini-hochberg procedure, 2025. URL https://arxiv.org/abs/2504.11759

2025 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Y. J. Choe and A. Ramdas. Combining Evidence Across Filtrations . 0 (arXiv:2402.09698), Feb. 2024

arXiv 2024

[2] [2]

A. P. Dawid, S. de Rooij , P. Grunwald, W. M. Koolen, G. Shafer, A. Shen, N. Vereshchagin, and V. Vovk. Probability-free pricing of adjusted American lookbacks. 0 (arXiv:1108.4113), Aug. 2011 a . doi:10.48550/arXiv.1108.4113

work page Pith review arXiv doi:10.48550/arxiv.1108.4113 2011

[3] [3]

A. P. Dawid, S. de Rooij , G. Shafer, A. Shen, N. Vereshchagin, and V. Vovk. Insuring against loss of evidence in game-theoretic probability. Statistics & Probability Letters, 81 0 (1): 0 157--162, Jan. 2011 b . ISSN 0167-7152. doi:10.1016/j.spl.2010.10.013

work page doi:10.1016/j.spl.2010.10.013 2011

[4] [4]

Fischer and A

L. Fischer and A. Ramdas. Online closed testing with e-values. 0 (arXiv:2407.15733), July 2024 a

arXiv 2024

[5] [5]

Fischer and A

L. Fischer and A. Ramdas. An online generalization of the e- BH procedure. 0 (arXiv:2407.20683), July 2024 b . doi:10.48550/arXiv.2407.20683

work page doi:10.48550/arxiv.2407.20683 2024

[6] [6]

Fischer, M

L. Fischer, M. Bofill Roig, and W. Brannath. The online closure principle. The Annals of Statistics, 52 0 (2), Apr. 2024. ISSN 0090-5364. doi:10.1214/24-AOS2370

work page doi:10.1214/24-aos2370 2024

[7] [7]

The e-Partitioning Principle of False Discovery Rate Control

J. Goeman, R. de Heide, and A. Solari. The e-partitioning principle of false discovery rate control, 2025. URL https://arxiv.org/abs/2504.15946

work page Pith review arXiv 2025

[8] [8]

P. D. Gr \"u nwald. Beyond neyman--pearson: E-values enable hypothesis testing with a data-driven alpha. Proceedings of the National Academy of Sciences, 121 0 (39): 0 e2302098121, 2024

work page 2024