REVIEW 4 major objections 5 minor 41 references
A scalable Bayesian double machine learning framework, with application to racial disproportionality assessment
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper claims that a Bayesian empirical likelihood combined with double machine learning yields a coverage-valid posterior for high-dimensional semiparametric treatment effects, and that applying it to London stop-and-search data…
desk verdict The Bayesian DML wrapper is a plausible incremental method with decent simulations, but the London application's headline negative effect is likely built into the outcome definition, so the empirical claim should not be trusted as stated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Bayesian generalized empirical likelihood posterior combined with sample-splitting double machine learning. The treatment effect β solves E[ψ(Z;β)] = 0 with ψ = [D−π(X)][Y−βD−µ(X)], where π and µ are nuisance functions estimated out-of-fold by Lasso, random forest, or neural network. The likelihood is replaced by profile empirical-likelihood weights p_i that minimize a Cressie-Read divergence subject to the moment condition, and β is updated by MCMC. Validity is certified by the Monahan-Boos H statistic, which checks that the posterior cumulative distribution at the true β is uniform over prior-predictive draws.
What would settle it
Re-estimate the effect using a leave-one-out disproportionality index where each borough's rate is divided by the London average computed from the other 32 boroughs only; if the negative posterior mean disappears or flips sign, the headline result is an artifact of the denominator. A sharper test would substitute an individual-level outcome such as the probability of being stopped given presence, or add borough-level crime counts to the confounder set and re-run the analysis.
Extended reading notes
Core claim
The central claim is that a Bayesian posterior can be built around the double machine learning estimating equation and still be a proper posterior. The posterior uses weights p_i from a generalized empirical likelihood (EL, ETEL, or Hellinger distance) subject to the moment condition, giving π(β|z) ∝ π0(β) ∏ p_i; the paper demonstrates by simulation that the resulting intervals achieve nominal coverage when checked with the Monahan-Boos H statistic. On the applied side, the paper claims that 'accounting for the proportion of the Black population can help alleviate disproportionality' in expressive-crime stop and search. Across all specifications, the posterior mean for β is negative, between -0.55 and -0.31; the 95% credible intervals from random-forest and neural-network nuisance models exclude zero, while Lasso intervals are wider and contain zero.
Load-bearing premise
The causal reading rests on no unmeasured confounding at the borough level: after conditioning on the 31 observed covariates, boroughs with larger Black population shares must not differ systematically in unmeasured drivers of stop-and-search disproportionality, such as actual crime rates, police deployment, or reporting practices; in addition, the disproportionality index itself divides by a London average that includes each borough, so part of the negative association may be built into the outcome definition.
Editorial extensions
If this is right
- If the coverage result holds beyond the simulation setting, researchers can report Bayesian credible intervals for high-dimensional semiparametric treatment effects without relying on asymptotic normal approximations.
- The negative posterior mean implies that, within London, stop-and-search disproportionality for expressive crimes is lower in boroughs with larger Black population shares, a pattern the authors read as more aggressive policing of Black individuals in predominantly white areas.
- The proposed algorithm is computationally about five times faster than the Gaussian-process benchmark (BDR-HD) in the n=50 simulation while matching its bias and RMSE, so it offers a scalable route for small-area Bayesian causal inference.
- The framework directly extends the double machine learning toolkit to settings where policymakers hold informative priors, since the prior enters only through β and the nuisance functions remain nonparametric.
Reading between the lines
- A leave-one-out recomputation of the DI, dividing each borough's rate by the London average excluding that borough, would directly test whether the negative coefficient is partly mechanical; the paper does not report that check.
- Given n=33 boroughs and 31 covariates, the wide Lasso credible intervals that include zero are arguably the more honest uncertainty statement; the tighter random-forest intervals still depend on the same identifying assumption of no unmeasured confounding.
- A natural out-of-sample prediction is that longitudinal changes in borough Black population share between Censuses should track decreases in the expressive-crime DI if the negative effect is causal; this is testable with future data.
- The Monahan-Boos validation is performed under the simulation's known data-generating process; a misspecified nuisance model or an incorrectly specified moment condition could break the coverage guarantee, and the paper does not probe this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a Bayesian extension of double/debiased machine learning for the partially linear model Y=µ(X)+βD+U. The likelihood is replaced by a generalized empirical likelihood (EL, ETEL, Hellinger) subject to the Neyman moment condition, with K-fold sample splitting for nuisance estimation. The authors argue via Monahan-Boos H-statistic simulations that the resulting posterior has valid coverage, show simulation comparisons for binary and continuous treatments, and apply the method to 33 London boroughs with outcome the Black disproportionality index for expressive crimes and treatment the borough Black population share, reporting a negative posterior mean.
Significance. If the method's coverage calibration holds beyond the simulated DGP, this is a useful contribution: it gives applied researchers a Bayesian posterior for a semiparametric causal parameter while retaining DML's robustness to high-dimensional nuisance estimation, and the code is public. The empirical question is important and the negative association is policy-relevant. However, the empirical inference is currently undermined by the definition of the outcome, which appears to be mechanically coupled to the treatment, and by the very small number of boroughs relative to confounders.
major comments (4)
- [Section 2, Table 3] As written, the outcome DI_i is the borough rate of Black people in Stop and Search for expressive crime divided by the London rate. If 'rate' is stops per Black resident, then DI_i = (B_i/(d_i P_i)) / (Σ_j B_j / Σ_j d_j P_j), so the treatment variable d_i (percentage of Black population) appears in the denominator of the outcome. Under a colorblind null in which B_i/(d_iP_i) is constant, E[DI_i]=1, but small-d_i boroughs have enormous sampling noise, which mechanically produces a negative regression slope of DI on d_i. This is exactly the pattern in Table 4, and the observed maximum DI 12.58 in a borough with only 1.89% Black residents is consistent with the artifact. Please give the exact formula for DI; if it is the one above, redefine the outcome so that the treatment does not enter its construction, or provide a quantitative demonstration that the negative association is not a ratio artifact.
- [Section 5, Table 4] The application has n=33 boroughs and p=31 covariates; with two-fold sample splitting, each nuisance model is estimated on roughly 16 units. The posterior credible intervals are extremely sensitive to the ML estimator: Lasso intervals all include zero (-2.17,0.95; -2.21,0.99; -2.18,1.00), while random forest and neural network intervals exclude zero. The claim that 'all methods suggest a negative relationship' is based only on posterior means. The paper should report the instability across folds, add a spatial component or at least a spatial residual analysis, and show sensitivity to the informative prior N(0,2). As it stands, the empirical conclusion is not robust.
- [Section 3.5, Figure 3] The Monahan-Boos validation produces Kolmogorov-Smirnov p-values of 0.07 (ETEL) and 0.09 (EL) on 10,000 simulations. These are borderline rather than clear evidence of uniformity, so the statement that the method 'yields a valid posterior' overstates what is shown. Report coverage at several nominal levels (e.g., 50%, 80%, 95%) with Monte Carlo standard errors, and restrict the claim to the DGP studied; no theoretical proof of general posterior validity is given.
- [Section 3.1] The causal estimand β=E[∂Y(d)/∂d] is not well matched to the DI outcome because DI_i is defined relative to the London-wide average rate, which itself depends on all borough treatments. Changing d_i for one borough changes the denominator of DI_j for every j, so the stable-unit-treatment-value assumption is strained even before considering the denominator coupling in the first comment. Please clarify what hypothetical intervention the estimand corresponds to, or reframe the analysis as descriptive.
minor comments (5)
- [Throughout] The manuscript contains several typographical errors, including 'dispropotionality', 'Alogrithm', 'the the empirical Cressie-Read statistic', 'counfounders', and 'may raises'. A careful proofread is needed.
- [Algorithm 2, Step 7] The algorithm samples β^(j) from a product that uses p_i^(j) computed from β^(j-1); please clarify whether this is a Metropolis-within-Gibbs step and report acceptance rates or a convergence diagnostic.
- [Figure 2] The histograms appear to be of posterior means, while the red curve is labeled the standard normal density; the comparison target should be stated explicitly.
- [Section 2, Table 3] The observed DI values range from 1.66 to 12.58, with no borough below 1, even though the text describes DI<1 as below expectation; please discuss whether the DI is restricted by construction and what this implies for the interpretation.
- [Discussion] The sentence 'police are more aggressively policing Black individuals in areas that are predominantly white' is a causal claim that goes beyond the estimated association; consider softening or removing it.
Circularity Check
The application's negative DI-Black-share relationship is partly built into the outcome: the DI is a rate whose denominator is the treatment (Black population share).
-
self definitional
[Section 2 (outcome and treatment definitions); Section 5, Table 4]
"Therefore, we define the outcome variable as the disproportionality index (DI) for expressive crime targeting Black people in each Borough (total 33 Boroughs) and it can be calculated as: DIi = rate of Black people in Stop and Search for expressive crime in Boroughi / rate of average Black people in Stop and Search for expressive crime in London. ... To evaluate the effect of disproportionality, we use the percentage of Black people in each borough as the treatment variable."
The DI is a rate ratio. With London's Black stop share around 38% and Black population share 13.5%, the only reading consistent with DI values up to 12.58 (Table 3) is that the borough rate is Black expressive stops divided by the borough's Black population, so the treatment D_i (Black population share) is the denominator of Y_i: Y_i = [B_i/(D_i P_i)] / [sum_j B_j/(sum_j D_j P_j)]. Hence log Y_i contains -log D_i by construction; small-D boroughs have mechanically large, high-variance DI values. Regressing this outcome on D therefore yields a negative coefficient even under a colorblind null, exactly the sign reported in Table 4 (-0.55 to -0.31).
full rationale
The methodological development is not circular: the empirical likelihood weights are solved from the Neyman moment condition, the sample-splitting algorithm is standard DML, and the posterior-coverage claim is checked by the Monahan-Boos H-statistic simulation, an external criterion. The citations to the authors' own ETEL work (Luo et al., 2023) are background and not load-bearing. However, the paper's headline empirical finding is partially circular: the DI outcome is defined using the treatment (Black population share) in its denominator, so the negative posterior means in Section 5 are in part a mathematical property of the outcome construction. The Discussion flags residual spatial autocorrelation but not this denominator coupling. Because the method itself stands independently but the application's central conclusion is partially forced by definition, the circularity score is 6 rather than 0-2.
Assumptions & free parameters
free parameters (1)
- Prior hyperparameters for treatment effect beta in application =
N(0,2)
assumptions (5)
- domain assumption No unmeasured confounding (NUC): given X, treatment D is independent of potential outcomes
- domain assumption Consistency, SUTVA, and positivity
- standard math Regularity conditions for double machine learning: nuisance estimates converge fast enough and Neyman orthogonality holds
- standard math Existence of empirical likelihood weights: zero lies in the convex hull of the moment function values
- domain assumption The DI outcome is a valid measure of disproportionality
Cite this review
Pith. "Pith review of A scalable Bayesian double machine learning framework, with application to racial disproportionality assessment." pith.science (2026). https://pith.science/paper/WBJFIIUS
@misc{pith2026250207695,
author = {Pith},
title = {Pith review of: A scalable Bayesian double machine learning framework, with application to racial disproportionality assessment},
year = {2026},
howpublished = {\url{https://pith.science/paper/WBJFIIUS}},
note = {Machine review of arXiv:2502.07695}
}
read the original abstract
Racial disproportionality in stop and search practices elicits substantial concerns about its societal and behavioral impacts. In London, Black individuals are about four times more likely to be stopped and searched than White individuals. Using data on stop and search events in London from January 2019 to December 2023, this paper aims to investigate disproportionality in the volume of stops for expressive crimes involving Black individuals compared to other ethnicities. We employ a semi-parametric partially linear structural regression method and introduce a Bayesian empirical likelihood procedure combined with double machine learning techniques to control for high-dimensional confounding and to accommodate strong prior assumptions. In addition, we show that the proposed procedure yields a valid posterior in terms of coverage. Applying this approach to the stop and search dataset, we find that racial disproportionality aimed at the Black community may be influenced by the borough racial composition when focusing on expressive crimes.
Figures
Reference graph
Works this paper leans on
-
[1]
Antonelli, J., G. Papadogeorgou, and F. Dominici (2022). Causal inference in high dimensions: A marriage between B ayesian modeling and good frequentist properties. Biometrics\/ 78\/ (1), 100--114
work page 2022
-
[2]
Baggerly, K. A. (1998). Empirical likelihood as a goodness-of-fit measure. Biometrika\/ 85\/ (3), 535--547
work page 1998
-
[3]
Bowling, B. and C. Phillips (2007). Disproportionate and discriminatory: Reviewing the evidence on police stop and search. The Modern Law Review\/ 70\/ (6), 936--961
work page 2007
-
[4]
Chamberlain, G. and G. W. Imbens (2003). Nonparametric applications of B ayesian inference. Journal of Business & Economic Statistics\/ 21\/ (1), 12--18
2003
-
[5]
Chetverikov, M
Chernozhukov, V., D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal\/ 21\/ (1), C1--C68
2018
-
[6]
Chernozhukov, V. and H. Hong (2003). An MCMC approach to classical estimation. Journal of Econometrics\/ 115\/ (2), 293--346
2003
-
[7]
Chib, S., M. Shin, and A. Simoni (2018). Bayesian estimation and comparison of moment condition models. Journal of the American Statistical Association\/ 113\/ (524), 1656--1668
work page 2018
-
[8]
Gaebler, J., W. Cai, G. Basse, R. Shroff, S. Goel, and J. Hill (2022). A causal framework for observational studies of discrimination. Statistics and Public Policy\/ 9\/ (1), 26--48
work page 2022
Show all 41 references
-
[9]
Graham, D. J., E. J. McCoy, and D. A. Stephens (2016). Approximate B ayesian inference for doubly robust estimation. Bayesian Analysis\/ 11\/ (1), 47--69
2016
-
[10]
Hahn, J. (1998). On the role of the propensity score in efficient semiparametric estimation of average treatment effects. Econometrica\/ , 315--331
1998
-
[11]
State of policing: The annual assessment of policing in england and wales 2022
HM Inspectorate of Constabulary and Fire & Rescue Services (2022). State of policing: The annual assessment of policing in england and wales 2022. HMICFRS\/
2022
-
[12]
Beck, and J
Huang, Z., B. Beck, and J. Antonelli (2024). Causal inference and racial bias in policing: New estimands and the importance of mobility data. arXiv:2409.08059\/
2024 arXiv
-
[13]
Johnson, and R
Imbens, G., P. Johnson, and R. H. Spady (1998). Information theoretic approaches to inference in moment condition models. Econometrica\/ 66\/ (2), 333--357
1998
-
[14]
Corbett-Davies, J
Jung, J., S. Corbett-Davies, J. D. Gaebler, R. Shroff, and S. Goel (2018). Mitigating included-and omitted-variable bias in estimates of disparate impact. arXiv:1809.05651\/
2018 arXiv
-
[15]
Kaplan, D. and J. Chen (2012). A two-step B ayesian approach for propensity score analysis: Simulations and case study. Psychometrika\/ 77\/ (3), 581--609
2012
-
[16]
Otsu, and K
Kitamura, Y., T. Otsu, and K. Evdokimov (2013). Robustness, infinitesimal neighborhoods, and moment restrictions. Econometrica\/ 81\/ (3), 1185--1201
2013
-
[17]
Lowe, and J
Knox, D., W. Lowe, and J. Mummolo (2020). Administrative records mask racially biased policing. American Political Science Review\/ 114\/ (3), 619--637
2020
-
[18]
Lee, M.-J. (2018). Simple least squares estimator for treatment effects using propensity score residuals. Biometrika\/ 105\/ (1), 149--164
2018
-
[19]
Ding, and F
Li, F., P. Ding, and F. Mealli (2023). Bayesian causal inference: A critical review. Philosophical Transactions of the Royal Society A\/ 381\/ (2247), 20220153
2023
-
[20]
Saarela, B
Liu, K., O. Saarela, B. M. Feldman, and E. Pullenayegum (2020). Estimation of causal effects with repeatedly measured outcomes in a B ayesian framework. Statistical Methods in Medical Research\/ 29\/ (9), 2507--2519
2020
-
[21]
Luo, Y., D. J. Graham, and E. J. McCoy (2023). Semiparametric B ayesian doubly robust causal estimation. Journal of Statistical Planning and Inference\/ 225 , 171--187
2023
-
[22]
Luo, Y., D. A. Stephens, D. J. Graham, and E. J. McCoy (2023). Assessing the validity of B ayesian inference using loss functions. arXiv:2103.04086\/
2023
-
[23]
McCandless, L. C., I. J. Douglas, S. J. Evans, and L. Smeeth (2010). Cutting feedback in B ayesian regression adjustment for the propensity score. The International Journal of Biostatistics\/ 6\/ (2), 16
2010
-
[24]
Meng, Y. (2017). Profiling minorities: Police stop-and-search practices in contemporary London . Human Geographies\/ 11\/ (1), 5--22
2017
-
[25]
Millner, N. (2020). As the drone flies: Configuring a vertical politics of urban policing. Political Geography\/ 80 , 102163
2020
-
[26]
Monahan, J. F. and D. D. Boos (1992). Proper likelihoods for B ayesian analysis. Biometrika\/ 79\/ (2), 271--278
1992
-
[27]
Disproportionality board data pack
MOPAC (2023). Disproportionality board data pack. Disproportionality Board Data Pack 2023\/
2023
-
[28]
Newey, W. K. and R. J. Smith (2004). Higher order properties of GMM and generalized empirical likelihood estimators . Econometrica\/ 72\/ (1), 219--255
2004
-
[29]
Oberwittler, D. and S. Roché (2022). How institutional contexts shape police-adolescent relations in France and Germany: Spatial and social disparities . Policing and Society\/ 32\/ (3), 378--410
2022
-
[30]
Owen, A. B. (2001). Empirical likelihood . CRC press
2001
-
[31]
Qin, J. and J. Lawless (1994). Empirical likelihood and general estimating equations. The Annals of Statistics\/ 22\/ (1), 300--325
1994
-
[32]
Read, T. R. and N. A. Cressie (2012). Goodness-of-fit statistics for discrete multivariate data . Springer Science & Business Media
2012
-
[33]
Robinson, P. M. (1988). Root- n -consistent semiparametric regression. Econometrica\/ 56\/ (4), 931--954
1988
-
[34]
Rosenbaum, P. R. and D. B. Rubin (1983). The central role of the propensity score in observational studies for causal effects. Biometrika\/ 70\/ (1), 41--55
1983
-
[35]
Rubin, D. B. (1981). The Bayesian bootstrap . The Annals of Statistics\/ 9\/ (1), 130--134
1981
-
[36]
Schennach, S. M. (2007). Point estimation with exponentially tilted empirical likelihood. The Annals of Statistics\/ 35\/ (2), 634--672
2007
-
[37]
Suss, J. and T. Oliveira (2023). Economic inequality and the spatial distribution of stop-and-search in london. British Journal of Criminology\/ 63\/ (4), 828--847
2023
-
[38]
Betancourt, D
Talts, S., M. Betancourt, D. Simpson, A. Vehtari, and A. Gelman (2018). Validating B ayesian inference algorithms with simulation-based calibration. arXiv:1804.06788\/
2018 arXiv
-
[39]
Quinton, and B
Tiratelli, M., P. Quinton, and B. Bradford (2018). Does stop-and-search deter crime? Evidence from ten years of London-wide data . The British Journal of Criminology\/ 58\/ (5), 1212--1231
2018
-
[40]
Yiu, A., R. J. B. Goudie, and B. D. M. Tom (2020). Inference under unequal probability sampling with the B ayesian exponentially tilted empirical likelihood. Biometrika\/ 107\/ (4), 857--873
2020
-
[41]
Zhao, Q., L. J. Keele, D. S. Small, and M. M. Joffe (2022). A note on posttreatment selection in studying racial discrimination in policing. American Political Science Review\/ 116\/ (1), 337--350
2022
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.