Flexible Bayesian Multiple Comparison Adjustment Using Dirichlet Process and Beta-Binomial Model Priors

Don van den Bergh; Fabian Dablander

arxiv: 2208.07086 · v3 · submitted 2022-08-15 · 📊 stat.ME · math.ST· stat.TH

Flexible Bayesian Multiple Comparison Adjustment Using Dirichlet Process and Beta-Binomial Model Priors

Don van den Bergh , Fabian Dablander This is my paper

Pith reviewed 2026-05-24 11:47 UTC · model grok-4.3

classification 📊 stat.ME math.STstat.TH

keywords Bayesian multiple comparisonspartition priorsbeta-binomial modelDirichlet process priorequality constraintsstochastic searchgroup comparisons

0 comments

The pith

Beta-binomial priors over partitions let Bayesian models test all possible group equalities at once.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a class of beta-binomial priors that place probability directly on every possible partition of groups, where each partition encodes a unique pattern of which groups are equal and which are not. This construction supplies a prior for multiple-comparison adjustment that considers the full set of equality configurations rather than only pairwise tests. The priors are contrasted with the Dirichlet process approach, and a stochastic search algorithm is introduced to traverse the rapidly expanding partition space. The method is shown on examples that compare means, standard deviations, and proportions.

Core claim

A class of flexible beta-binomial priors for multiple comparison adjustment allows researchers to assess pairwise equality constraints but simultaneously all possible equalities among all groups, by constructing a prior distribution over all possible partitions of groups.

What carries the argument

Beta-binomial priors placed directly on the space of partitions of groups, which represent all configurations of equality and inequality constraints.

If this is right

Pairwise equality tests can be performed while accounting for every other possible equality pattern among the groups.
The framework applies directly to comparisons of means, standard deviations, and proportions.
Computation remains practical for ten or more groups because the stochastic search navigates more than 100,000 partitions without exhaustive listing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same partition-prior construction could be transferred to other Bayesian selection problems whose configuration space is combinatorial.
Hierarchical or covariate-dependent extensions of the beta-binomial partition prior would be a direct next step.
Applied researchers who currently rely on post-hoc corrections could replace those corrections with a single prior over partitions.

Load-bearing premise

The stochastic search algorithm efficiently explores the rapidly growing space of partitions without missing important configurations or introducing bias in the posterior.

What would settle it

For a small number of groups where exhaustive enumeration of partitions is feasible, compare the posterior probabilities obtained from the stochastic search against the exact posteriors from full enumeration.

Figures

Figures reproduced from arXiv: 2208.07086 by Don van den Bergh, Fabian Dablander.

**Figure 1.** Figure 1: All 52 possible models given K = 5, represented as partitions. Circles represent individual parameters and shaded regions indicate which parameters are equal. specifically, going beyond classical testing, we consider the problem of assessing all possible equalities and inequalities between the groups. In general terms, the inference problem is: ρ ∼ πρ(.) ⃗θ | ρ ∼ πθ⃗(.) f(⃗y; ⃗θ, ρ) = Y K j=1 g(⃗yj ; θj , … view at source ↗

**Figure 2.** Figure 2: Top: Dirichlet process (left), beta-binomial (middle), and uniform prior (right) across [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Figure 3: Left: Probability of making at least one false claim about a difference between two [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

**Figure 4.** Figure 4: Familywise error rate across priors and sample sizes under a model with 0 (top left), [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗

**Figure 5.** Figure 5: Proportion of falsely claiming a difference between two groups when there is none [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

**Figure 6.** Figure 6: Left: Posterior means of the full model where all proportions are assumed to be [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

**Figure 7.** Figure 7: Left: Posterior means of the full model where all standard deviations are assumed to be [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗

**Figure 8.** Figure 8: Familywise error rate across priors and sample sizes under a model with 0 (top left), [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗

**Figure 9.** Figure 9: Proportion of falsely claiming a difference between two groups when there is none [PITH_FULL_IMAGE:figures/full_fig_p025_9.png] view at source ↗

read the original abstract

Researchers frequently wish to assess the equality or inequality of groups, but this poses the challenge of adequately adjusting for multiple comparisons. Statistically, all possible configurations of equality and inequality constraints can be uniquely represented as partitions of groups, where any number of groups are equal if they are in the same subset of the partition. In a Bayesian framework, one can adjust for multiple comparisons by constructing a suitable prior distribution over all possible partitions. Inspired by work on variable selection in regression, we propose a class of flexible beta-binomial priors for multiple comparison adjustment. We compare this prior setup to the Dirichlet process prior suggested by Gopalan and Berry (1998) and multiple comparison adjustment methods that do not specify a prior over partitions directly. Our approach not only allows researchers to assess pairwise equality constraints but simultaneously all possible equalities among all groups. Since the space of possible partitions grows rapidly -- for ten groups, there are already 115,975 possible partitions -- we use a stochastic search algorithm to efficiently explore the space. Our method is implemented in the Julia package EqualitySampler, and we illustrate it on examples related to the comparison of means, standard deviations, and proportions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Beta-binomial priors over partitions give a flexible alternative to the 1998 Dirichlet process for Bayesian multiple comparisons, backed by Julia code, but the stochastic search over large partition spaces lacks shown convergence checks.

read the letter

This paper introduces beta-binomial priors on the space of partitions as a way to adjust for multiple comparisons when comparing groups in a Bayesian model. It sets those priors against the Dirichlet process prior from Gopalan and Berry and against methods that do not put a prior directly on partitions. The beta-binomial construction comes from variable selection work and lets the user tune how much prior mass sits on different numbers of distinct groups. The claim is that this gives the full posterior over all equality patterns at once, not just pairwise tests. They also release the EqualitySampler package in Julia and walk through examples on means, standard deviations, and proportions. That package and the direct comparison to the older prior are the concrete additions. The framing is useful because the partition view automatically handles the full set of equality constraints without separate adjustments. The soft spot is the sampling step. For ten groups the partition space already holds 115975 elements, so the method depends on stochastic search to explore it. The abstract calls the search efficient but supplies no effective sample sizes, recovery simulations, or bounds on how close the sampled posterior comes to the target. If those checks are missing from the full paper, the reported probabilities on equality configurations could be biased or incomplete. This work is for Bayesian statisticians who already use partition-based priors or who want a tunable alternative for group comparisons. An applied reader looking for off-the-shelf multiple comparison tools will find the software helpful but will still need to verify the search behavior on their own data. I would send it for peer review. The idea is straightforward, the software lowers the cost of trying it, and the comparison to the Dirichlet process is worth having in print even if the search diagnostics need strengthening.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes a class of flexible beta-binomial priors over the space of partitions of groups as a Bayesian approach to multiple-comparison adjustment. This allows simultaneous inference on all possible equality configurations (not merely pairwise), is compared to the Dirichlet-process prior of Gopalan & Berry (1998) and to non-prior-based methods, and is implemented via a stochastic search algorithm whose results are illustrated on examples involving means, standard deviations, and proportions; the method is released in the Julia package EqualitySampler.

Significance. If the stochastic search is shown to produce reliable posterior probabilities over partitions, the beta-binomial construction supplies a tunable, partition-level prior that directly encodes beliefs about the number and sizes of equal groups, extending existing Bayesian multiple-testing tools and providing a reproducible software implementation that other researchers can apply directly.

major comments (2)

[Abstract / stochastic search description] Abstract and § on stochastic search: the claim that the algorithm 'efficiently explore[s] the space' for 115975 partitions (n=10) is load-bearing for all reported posterior probabilities on equality constraints, yet no effective sample sizes, Gelman-Rubin statistics, or total-variation bounds are supplied to demonstrate that the Metropolis-Hastings chain mixes and is unbiased with respect to the beta-binomial target.
[Comparison to Dirichlet process prior] § comparing beta-binomial to Dirichlet process: the manuscript asserts greater flexibility, but without simulation results that quantify calibration of posterior probabilities on global and pairwise equalities (or frequentist error-rate control) under both priors, the practical advantage over Gopalan & Berry (1998) remains unverified.

minor comments (2)

[Prior specification] Notation for the beta-binomial parameters (a, b) and the partition probability mass function should be introduced once with an explicit equation rather than only in prose.
[Abstract] The abstract states that the approach 'allows researchers to assess pairwise equality constraints but simultaneously all possible equalities'; a short clarifying sentence distinguishing marginal pairwise probabilities from joint partition probabilities would help readers.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which identify key areas where additional validation would strengthen the manuscript. We address each major comment below and agree that revisions are warranted.

read point-by-point responses

Referee: [Abstract / stochastic search description] Abstract and § on stochastic search: the claim that the algorithm 'efficiently explore[s] the space' for 115975 partitions (n=10) is load-bearing for all reported posterior probabilities on equality constraints, yet no effective sample sizes, Gelman-Rubin statistics, or total-variation bounds are supplied to demonstrate that the Metropolis-Hastings chain mixes and is unbiased with respect to the beta-binomial target.

Authors: We acknowledge that the current manuscript does not report formal convergence diagnostics such as effective sample sizes, Gelman-Rubin statistics, or total-variation distance bounds for the stochastic search procedure. Although the algorithm is a Metropolis-Hastings sampler targeting the beta-binomial distribution over partitions and we observed stable results across independent runs in the reported examples, the absence of these diagnostics leaves the mixing properties unverified. In the revised version we will add effective sample size estimates, multiple-chain Gelman-Rubin statistics, and a brief total-variation assessment for the n=10 case to substantiate the claim of efficient exploration. revision: yes
Referee: [Comparison to Dirichlet process prior] § comparing beta-binomial to Dirichlet process: the manuscript asserts greater flexibility, but without simulation results that quantify calibration of posterior probabilities on global and pairwise equalities (or frequentist error-rate control) under both priors, the practical advantage over Gopalan & Berry (1998) remains unverified.

Authors: The manuscript provides a theoretical comparison of the two priors, emphasizing that the beta-binomial construction permits direct control over the distribution of partition sizes and the number of clusters, whereas the Dirichlet process induces a specific Ewens sampling formula. We agree, however, that this flexibility claim would be more convincing if accompanied by simulation evidence on posterior calibration and frequentist operating characteristics. We will therefore include a new simulation study in the revision that generates data under known equality configurations, computes posterior probabilities of global and pairwise equalities under both priors, and reports calibration and error-rate results. revision: yes

Circularity Check

0 steps flagged

No significant circularity; proposal rests on external priors and search algorithm

full rationale

The abstract and provided text propose beta-binomial priors inspired by external variable-selection literature and compare them to the 1998 Gopalan-Berry Dirichlet process prior. No equations, fitted parameters, or self-citations appear in the load-bearing claims. The stochastic search is described as efficient but is not shown to reduce to a self-defined quantity or fitted input. The derivation chain is therefore self-contained against external benchmarks and receives the default non-finding.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central modeling choice rests on representing all equality/inequality patterns as partitions and placing a prior directly on that discrete space; no free parameters or invented entities are named in the abstract.

axioms (1)

domain assumption All possible configurations of equality and inequality constraints can be uniquely represented as partitions of groups.
Stated explicitly in the abstract as the statistical representation used.

pith-pipeline@v0.9.0 · 5736 in / 1141 out tokens · 28680 ms · 2026-05-24T11:47:46.181244+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages · 1 internal anchor

[1]

A comparison of the Benjamini-Hochberg procedure with some Bayesian rules for multiple testing

Barbieri, M. M., Berger, J. O., George, E. I., & Roˇ ckov´ a, V. (2021). The median probability model and correlated variables. Bayesian Analysis, 16(4), 1085–1112. Barbieri, M. M., & Berger, J. O. (2004). Optimal predictive model selection. The Annals of Statistics, 32(3), 870–897. Bayarri, M. J., Berger, J. O., Forte, A., & Garc´ ıa-Donato, G. (2012). C...

work page internal anchor Pith review Pith/arXiv arXiv 2021
[2]

Chang, S., & Berger, J. O. (2020). Frequentist properties of Bayesian multiplicity control for multiple testing of normal means. Sankhya A, 82, 310–329. Consonni, G., Fouskakis, D., Liseo, B., Ntzoufras, I., et al. (2018). Prior distributions for ob- jective Bayesian analysis. Bayesian Analysis, 13(2), 627–679. Dablander, F., van den Bergh, D., Ly, A., & ...

work page 2020
[3]

Lewandowski, D., Kurowicka, D., & Joe, H. (2009). Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis , 100(9), 1989–2001. Liang, F., Paulo, R., Molina, G., Clyde, M. A., & Berger, J. O. (2008). Mixtures of g priors for Bayesian variable selection. Journal of the American Statistical Associatio...

work page 2009
[4]

Rao, C., & Swarupchand, U. (2009). Multiple comparison procedures-a note and a bibliography. Journal of Statistics , 16(1), 66–109. Rasmussen, C. E., et al. (1999). The infinite Gaussian mixture model. NIPS, 12, 554–560. Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. M. (2012). Default Bayes factors for ANOVA designs. Journal of Mathematical...

work page 2009
[5]

simulations

https://mc-stan.org Teh, Y. W. (2010). Dirichlet process. Encyclopedia of Machine Learning, 1063, 280–287. Westfall, P. H., Johnson, W. O., & Utts, J. M. (1997). A Bayesian perspective on the Bonferroni adjustment. Biometrika, 84(2), 419–427. Wilson, M. A., Iversen, E. S., Clyde, M. A., Schmidler, S. C., & Schildkraut, J. M. (2010). Bayesian model search ...

work page 2010
[6]

The prior density of the beta-binomial distribution over partitions is decreasing for α = 1 and β ≥ K 2 , and strictly decreasing for α = 1 and β > K 2 . Proof. The prior density of the Beta-binomial over partitions is given by: π (ρ | K, α, β) = K − 1 |ρ| − 1 B (|ρ| − 1 + α, K − |ρ| + β) B (α, β ) K |ρ| . To examine the ratio of two consecutive model siz...

work page 1975

[1] [1]

A comparison of the Benjamini-Hochberg procedure with some Bayesian rules for multiple testing

Barbieri, M. M., Berger, J. O., George, E. I., & Roˇ ckov´ a, V. (2021). The median probability model and correlated variables. Bayesian Analysis, 16(4), 1085–1112. Barbieri, M. M., & Berger, J. O. (2004). Optimal predictive model selection. The Annals of Statistics, 32(3), 870–897. Bayarri, M. J., Berger, J. O., Forte, A., & Garc´ ıa-Donato, G. (2012). C...

work page internal anchor Pith review Pith/arXiv arXiv 2021

[2] [2]

Chang, S., & Berger, J. O. (2020). Frequentist properties of Bayesian multiplicity control for multiple testing of normal means. Sankhya A, 82, 310–329. Consonni, G., Fouskakis, D., Liseo, B., Ntzoufras, I., et al. (2018). Prior distributions for ob- jective Bayesian analysis. Bayesian Analysis, 13(2), 627–679. Dablander, F., van den Bergh, D., Ly, A., & ...

work page 2020

[3] [3]

Lewandowski, D., Kurowicka, D., & Joe, H. (2009). Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis , 100(9), 1989–2001. Liang, F., Paulo, R., Molina, G., Clyde, M. A., & Berger, J. O. (2008). Mixtures of g priors for Bayesian variable selection. Journal of the American Statistical Associatio...

work page 2009

[4] [4]

Rao, C., & Swarupchand, U. (2009). Multiple comparison procedures-a note and a bibliography. Journal of Statistics , 16(1), 66–109. Rasmussen, C. E., et al. (1999). The infinite Gaussian mixture model. NIPS, 12, 554–560. Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. M. (2012). Default Bayes factors for ANOVA designs. Journal of Mathematical...

work page 2009

[5] [5]

simulations

https://mc-stan.org Teh, Y. W. (2010). Dirichlet process. Encyclopedia of Machine Learning, 1063, 280–287. Westfall, P. H., Johnson, W. O., & Utts, J. M. (1997). A Bayesian perspective on the Bonferroni adjustment. Biometrika, 84(2), 419–427. Wilson, M. A., Iversen, E. S., Clyde, M. A., Schmidler, S. C., & Schildkraut, J. M. (2010). Bayesian model search ...

work page 2010

[6] [6]

The prior density of the beta-binomial distribution over partitions is decreasing for α = 1 and β ≥ K 2 , and strictly decreasing for α = 1 and β > K 2 . Proof. The prior density of the Beta-binomial over partitions is given by: π (ρ | K, α, β) = K − 1 |ρ| − 1 B (|ρ| − 1 + α, K − |ρ| + β) B (α, β ) K |ρ| . To examine the ratio of two consecutive model siz...

work page 1975