REVIEW 2 major objections 5 minor
False discovery rate control with compound p-values
T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper proves that independent compound p-values keep the BH procedure's false discovery rate at most $1.93\alpha$, and that the constant cannot drop below $\frac{7}{6}$.
desk verdict Genuinely new results on FDR for compound p-values, with one unproved numerical step that should be fixed before the advertised constant is taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine of the proof is a leave-one-out reformulation of the BH procedure (Lemma 12): for each possible number $i$ of rejected nulls, $k_i$ is the number of rejections the procedure would make if exactly $i$ true nulls were rejected, and $I$ is the actual number of rejected nulls, so the false discovery proportion is $I/(1\vee k_I)$. Conditional on the non-null p-values, independence is used through a positive-correlation inequality for independent variables (Lemma 13) to bound the conditional probabilities, and through $B_i(t)$, the worst-case probability that a sum of independent Bernoulli variables with mean at most $t$ reaches $i$. The remaining task is a deterministic optimization problem, solved by the recursively defined sequence $c_1=1$, $c_2=1.5$, and $c_\ell = c_{\ell-1} + P\{\operatorname{Pois}((\ell-1)/c_{\ell-1}) \ge \ell-1\} - c_{\ell-1}P\{\operatorname{Pois}((\ell-1)/c_{\ell-1}) \ge \ell\}$, whose limit is verified numerically to be at most $1.9227$. That constant is what converts the tail bounds into the $1.93\alpha$ FDR guarantee.
What would settle it
Compute the sequence $c_\ell$ from Lemma 14 to a large index, say $10^6$ or $10^7$ terms, and compare its limiting value with $1.9227$; if the limit exceeds $1.93$, the constant in Theorem 1 is false, while reproducing the limit confirms the lemma's numerical claim.
Extended reading notes
Core claim
The central claim is that independence protects the BH procedure from the worst effects of compound p-values. Theorem 1 states that for independent compound p-values, the FDR of the BH procedure at level $\alpha$ is at most $1.93\alpha$ for every $m$ and every set of true nulls; Proposition 2 shows the constant is genuinely needed, with a distribution whose FDR is exactly $\frac{7}{6}\alpha$. Under the global null (all hypotheses true) the bound improves to $\alpha + 2\alpha^2$, with a matching lower bound of $\alpha + \alpha^2/4$. Under the PRDS positive-dependence condition, however, the paper constructs compound p-values whose FDR is at least $\frac{3}{8}\min\{\alpha h_m, 1\}$, so the $O(\log m)$ inflation cannot be avoided in that setting. The same techniques extend to approximate compound p-values, and the paper gives examples where compound p-values arise from empirical Bayes, decreasing-density, permutation, and Monte Carlo testing settings.
Load-bearing premise
The argument for the $1.93\alpha$ bound relies on Lemma 14, whose final step states without a written derivation that the recursively defined sequence $c_\ell$ has limit at most $1.9227$; if that numerical verification is wrong or unreproducible, the stated constant is not established by the proof as written.
Editorial extensions
If this is right
- For any number $m$ of hypotheses, independent compound p-values can be fed into the BH procedure with only a constant-factor FDR inflation, so the harmonic-factor correction required under arbitrary dependence is unnecessary.
- The worst-case constant lies between $\frac{7}{6}$ and $1.93$, so exact FDR control at level $\alpha$ is impossible for this procedure without extra assumptions on the marginal distributions.
- Under the global null the excess FDR is $O(\alpha^2)$, meaning that for small $\alpha$ the BH procedure is nearly an exact level-$\alpha$ test even though individual p-values are only valid on average.
- The PRDS condition, which restores level-$\alpha$ control for ordinary p-values, does not rescue compound p-values: FDR can still grow like $\log m$, so independence is what is doing the real work.
- Approximate compound p-values inherit the same type of guarantee, with the $1.93\alpha$ bound inflated by a factor $1+\epsilon$ when the approximation error is of the multiplicative type.
Reading between the lines
- A natural next step, left open by the paper, is to optimize the deterministic problem in Lemma 14 directly; the true worst-case constant may well be closer to $\frac{7}{6}$ than to $1.93$.
- The conditional-independence structure in the permutation and Monte Carlo examples suggests a transferable recipe: if a pooled statistic is exchangeable after conditioning on empirical distributions, the resulting p-values become independent compound p-values, so the $1.93\alpha$ bound applies in many designs beyond those listed.
- The contrast with the e-value analogue of BH, which does allow level-$\alpha$ FDR control even under dependence, indicates that the logarithmic penalty for compound p-values may be intrinsic to p-value aggregation rather than to the average-validity idea itself.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies false discovery rate control of the Benjamini–Hochberg (BH) procedure when the input p-values are compound p-values, i.e., they satisfy average superuniformity over the true nulls. Under independence, Theorem 1 gives the central bound FDR ≤ 1.93α, Theorem 3 extends this to approximate compound p-values, and Theorem 4 gives the sharper global-null bound FDR ≤ α + 2α². These upper bounds are complemented by lower bounds showing that a constant-factor inflation is unavoidable under independence (Proposition 2, FDR ≥ 7α/6) and that the global-null excess is at least α²/4 (Proposition 5). Proposition 6 shows that, under PRDS, the O(log m) upper bound of Armstrong cannot be improved beyond a constant factor. The paper also presents several motivating constructions of compound p-values (empirical Bayes, decreasing densities, permutation tests, Monte Carlo p-values, data alignment, Gaussian means) and an application to Upworthy headline data. Proofs are deferred to appendices and are largely detailed and self-contained.
Significance. If the main results are correct, this is a substantial contribution: it shows that independence, rather than positive dependence, is the assumption that rescues BH from the O(log m) inflation seen under arbitrary dependence for compound p-values, and it pins down the worst-case constant-factor inflation up to the factor 7/6 versus 1.93. The lower-bound constructions are explicit and checkable, and the examples and data analysis demonstrate practical relevance. The paper is also careful to benchmark against the existing results of Armstrong, Benjamini–Yekutieli, and Guo–Rao. The main weakness is that the advertised constant 1.93 in Theorem 1 rests on an unverified numerical claim in Lemma 14; until a reproducible numerical verification or a rigorous analytic bound is supplied, the exact constant is not established, although the qualitative O(α) conclusion is very likely sound.
major comments (2)
- [Appendix A.2, Lemma 14] The proof of Lemma 14 asserts 'We can verify numerically that lim_{\ell\to\infty} c_\ell \le 1.9227' and provides no computation, code, certified interval bound, or tail estimate. This numerical assertion is the only step that converts the recursive inequality (23) into the constant 1.93 in Theorem 1, so it is load-bearing: without a reproducible verification (or an analytic bound replacing the numerical limit), the advertised constant is not proved, even though a qualitative O(α) bound might survive by a different route.
- [Appendix B.1, Proposition 2] The formal definition of the lower-bound construction says 'p1 ∼ Unif(1.5/m, 1)', but the subsequent compound-p-value check and the FDR calculation use the two-point distribution P(p1 = 1.5/m) = P(p1 = 1) = 1/2. Under the uniform distribution, the stated value of \(\sum_{i\in H_0} P(p_i \le 1.5/m)\) would be 1 rather than 1.5, so the proof as written does not match the construction. Please correct the displayed definition to the two-point distribution that is actually used in the calculation.
minor comments (5)
- [Appendix F.4, Lemma 24] The first identity in Lemma 24 is numerically false: for t = 0.5, the right-hand side equals 0 while the left-hand side is at least P(Pois(0.5) ≥ 1) > 0.393. The second identity is the one used in Theorem 4, so the main result is not affected, but the lemma should be corrected or the first identity removed.
- [Appendix B.3, Proposition 6] In the compound-p-value verification, the displayed sum is concluded to equal αℓ, but the computation actually yields α′ℓ. Since α′ ≤ α, the inequality still verifies the compound-p-value condition; please fix the displayed equality.
- [Appendix B.3, Proposition 6] The claim that the construction satisfies the PRDS condition (9) is asserted without proof. A short verification would be helpful, since this is a central lower-bound result and the conditioning event changes in a non-obvious way when t crosses the atom of p_i.
- [Appendix A.4, Lemma 15] The notation is confusing because U1(s1) is used both for the event {P_m j=1 1{U_j ≤ s_{1,j}} ≥ 1} and for the random variable U1. Please introduce distinct notation for the events, for instance E_i(s_i).
- [Figures 1 and 2] The captions refer to the 'Uworthy' data set; the correct name is 'Upworthy'. There is also a duplicated article 'the' on page 7 ('estimate the the false discovery proportion') that should be corrected.
Circularity Check
No circularity: the main FDR bounds are derived from the compound-p-value definition and external lemmas; the numerical verification in Lemma 14 is a reproducibility gap, not a circular step.
full rationale
All main claims are proven from the defining inequality (2), the BH reformulation (Lemma 12), and external named results (Hoeffding 1956, Benjamini-Hochberg, Benjamini-Yekutieli, Armstrong 2022). Theorem 1's bound is not obtained by fitting any parameter to the target; the constant 1.9227 is the output of a worst-case optimization in Lemma 14, independent of the data and of Theorem 1's statement. Lower bounds (Propositions 2, 5, 6) are explicit adversarial constructions with computed FDR; they are not derived from the upper bounds. Examples are auxiliary and validated by direct CDF calculations. The only in-scope weakness is that Lemma 14's proof asserts 'We can verify numerically that lim c_l ≤ 1.9227' without giving the computation or an interval certificate, and Lemma 24's first identity is numerically false as stated (for t=0.5 the right-hand side is 0 while the left-hand side already exceeds 0.393); the latter is not used in Theorem 4. These are correctness/reproducibility issues, not circularity: no step defines its conclusion into its assumptions, and no load-bearing premise rests on a self-citation. The self-citations present (Ramdas et al. 2019; Pananjady and Samworth 2022) occur only in illustrative examples or related work and are not load-bearing.
Assumptions & free parameters
assumptions (7)
- domain assumption Definition of compound p-values (condition (2)) introduced by Armstrong (2022)
- domain assumption Independence of true-null p-values and their independence from non-null p-values (footnote 2)
- domain assumption PRDS positive dependence condition (9) from Benjamini and Yekutieli (2001)
- standard math Hoeffding's inequality comparing sums of independent Bernoullis with Poisson tails (Hoeffding 1956, Theorem 4)
- standard math Rearrangement inequality (Schmidt 2014, Theorem 2.2)
- standard math Gamma-Poisson relationship and Stirling number identity (Boyadzhiev 2009; Paris 2010)
- ad hoc to paper Numerical verification that lim_{\ell\to\infty} c_\ell <= 1.9227 (Lemma 14)
Cite this review
Pith. "Pith review of False discovery rate control with compound p-values." pith.science (2026). https://pith.science/paper/2SO7NBHQ
@misc{pith2026250721465,
author = {Pith},
title = {Pith review of: False discovery rate control with compound p-values},
year = {2026},
howpublished = {\url{https://pith.science/paper/2SO7NBHQ}},
note = {Machine review of arXiv:2507.21465}
}
abstract
In the setting of multiple testing, compound p-values generalize p-values by asking for superuniformity to hold only \emph{on average} across all true nulls. We study the properties of the Benjamini--Hochberg procedure applied to compound p-values. Under independence, we show that the false discovery rate (FDR) is at most $1.93\alpha$, where $\alpha$ is the nominal level, and exhibit a distribution for which the FDR is $\frac{7}{6}\alpha$. If additionally all nulls are true, then the upper bound can be improved to $\alpha + 2\alpha^2$, with a corresponding worst-case lower bound of $\alpha + \alpha^2/4$. Under positive dependence, on the other hand, we demonstrate that FDR can be inflated by a factor of $O(\log m)$, where~$m$ is the number of hypotheses. We provide numerous examples of settings where compound p-values arise in practice, either because we lack sufficient information to compute non-trivial p-values, or to facilitate a more powerful analysis.
Figures
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.