REVIEW 5 major objections 6 minor 30 references
Predictive Coresets
T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read A predictive coreset matches the posterior predictive distribution of a reweighted subsample to that of the full data, making coresets available for nonparametric and likelihood-free Bayesian models.
desk verdict Genuinely new idea for coresets by matching posterior predictives, but the main algorithm's convergence is unproven due to an unjustified argmin/expectation swap. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machine is predictive recursion under a Dirichlet process prior. The unknown posterior predictive $p(\tilde y_{N+1:\infty} \mid y_{1:N})$ is approximated by $M$ draws from the Pólya urn, the DP's predictive rule; the predictive given a weighted coreset is approximated by $M$ draws from a weighted Pólya urn. The coreset transform $T$ is restricted to linear maps $y^\star \mapsto w \odot y^\star$ on the observed subsample, with imputed future points left untouched, and the discrepancy $d$ is the $p$-Wasserstein distance between the two empirical predictive samples. Each Monte Carlo trajectory yields one optimization, and the returned transform averages those per-trajectory optimal weights.
What would settle it
Take a small conjugate Bayesian model, such as normal data with a normal prior, where the full-data posterior predictive is available in closed form. Compute the predictive coreset by the paper's Algorithm 1, evaluate the discrepancy in Definition 3.1 at the returned averaged weights, and compare it with the minimum discrepancy over weights computed from the exact predictive. If the returned weights miss that minimum, or if the averaged per-trajectory argmins differ from the argmin of the averaged objective on the same Monte Carlo draws, the algorithm is not solving the stated problem.
Extended reading notes
Core claim
The central claim is that a good coreset can be defined by prediction, not likelihood. Given a sample $y_{1:N}$ and a subsample $y^\star_{1:n}$, a predictive coreset is the empirical distribution of $\tilde y^\star_{1:n} = T(y^\star_{1:n})$, where $T$ minimizes a discrepancy $d$ between the posterior predictive of future observations conditioned on the transformed coreset and the posterior predictive conditioned on the full data. The paper proposes a general algorithm: sample $M$ steps ahead from the Pólya-urn predictive under a Dirichlet process prior for both the full data and the weighted coreset, solve for linear weights $w$ by minimizing a Wasserstein distance between the two empirical predictive clouds, and average the optimal weights over Monte Carlo iterations. The paper reports that this improves posterior inference over uniform subsampling in the large majority of simulated repeats for density estimation, logistic regression, and random partitions, and it establishes that the weights converge almost surely to the minimizer of the predictive discrepancy.
Load-bearing premise
The load-bearing premise is that the approximate future-data draws used inside the algorithm stand in for the true predictive distributions, and that averaging the best weight vectors found on each draw gives the single best weight vector overall.
Editorial extensions
If this is right
- Nonparametric priors become viable for coreset construction, since the algorithm needs only predictive draws, not a tractable likelihood.
- Data in non-Euclidean spaces, such as random partitions, can be core-setted by putting a ground metric on the augmented space and transporting mixture draws; the partition experiments illustrate this.
- The predictive coreset improves over uniform subsampling in 81% of density-estimation repeats, 71% of logistic-regression repeats, and 91% of random-partition repeats in the paper's simulations.
- As the number of Monte Carlo iterations grows, the returned weights converge almost surely to the minimizer of the predictive discrepancy (Theorem 2), so approximation error can be driven down computationally.
- With the adaptive extension, hyperparameter values are sampled near the posterior rather than uniformly from the prior, which should make the matching objective track the true posterior predictives and cut wasted computation in hierarchical models.
Reading between the lines
- Because the construction only needs predictive draws, the Dirichlet process can likely be replaced by any prior with a closed-form predictive rule, such as normalized random measures with independent increments; the paper uses the DP as one workable instance.
- The reported gains are against uniform subsampling; a head-to-head comparison with likelihood-based coresets on a tractable parametric model would show whether predictive matching adds accuracy or mainly adds model applicability.
- The average-of-argmins step is a Monte Carlo estimator for the argmin of an expected discrepancy; using one pooled stochastic optimization across all trajectories would be a natural alternative and a way to test stability.
- Algorithm 3 suggests a direct experiment: on a hierarchical model, compare weights from prior-sampling and posterior-sampling versions; large differences would indicate that the prior draws, not the transport step, dominate the approximation error.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes predictive coresets, a construction in which a small weighted subsample is chosen to make the posterior predictive distribution of the transformed subsample match the posterior predictive distribution of the full data, rather than matching likelihoods as in standard KL-based coresets. A Dirichlet-process prior is used as a tractable surrogate for the posterior predictive via Pólya-urn sampling, and the weights are obtained by minimizing a Wasserstein-type discrepancy between M-step empirical predictive draws and averaging the per-draw minimizers over Monte Carlo iterations. Three experiments—density estimation, logistic regression, and random partitions—report that the proposed coreset improves on uniform subsampling in 71–91% of repeat simulations. The paper also states posterior-contraction-based convergence results and sketches an ABC-style adaptive extension for hyperparameter sampling.
Significance. If the method worked as described, it would extend coresets from likelihood-based KL objectives to predictive distributions and to non-Euclidean sample spaces, which would be a useful contribution to Bayesian coreset methodology. The definition in Eq. (2) is original, the paper is clearly written, and the public code plus quantitative comparisons to uniform subsampling are positive features. However, the main theoretical result does not establish the convergence of the algorithm as implemented, and the operational procedure diverges from the stated objective in two load-bearing places: the per-draw argmin averaging and the prior-sampling of hyperparameters. The empirical evaluation is internally coherent but does not yet support the model-agnostic claim, because the validation uses the same DP-mixture modeling assumptions as the coreset objective. The contribution is promising but not established in this version.
major comments (5)
- [§3.1, Eq. (2); Algorithm 1, lines 7–9] The core optimization in Definition 3.1 is a single minimization of d between two random posterior predictive laws, but Algorithm 1 replaces it by per-draw empirical minimizations and then averages the argmins, returning \barω = (1/niter)Σ_t ω_t. For a nonlinear discrepancy such as the Wasserstein distance, argmin_ω E[d(·,ω)] is not equal to E[argmin_ω d(·,ω)] in general, so the returned weights need not minimize the defining objective. Theorem 2 claims a.s. convergence of this average to the population argmin, but its proof only asserts convergence of the predictives to p0 and continuity of the weighting map; it never proves an argmin/expectation interchange, and increasing niter does not make the finite-N and finite-n predictive distributions converge to p0. All reported improvements in Sections 4 and 5 are produced by this unvalidated averaging step, so the empirical claims do not distinguish the proposed method from an ad hoc weighting rule.
- [Algorithm 1, Step 5; §7] Hyperparameters are drawn θ_t ∼ π(dθ) from the prior, not from the posterior π(dθ | y_{1:N}). Averaging Pólya-urn predictives over θ_t ∼ π therefore gives ∫ p(· | y_{1:N}, θ) π(dθ), which is not the posterior predictive p(· | y_{1:N}) of model (3) unless the posterior equals the prior. Thus the quantity matched in the algorithm is not the quantity in Definition 3.1. Section 7 acknowledges the prior-sampling limitation, but Algorithm 3 is only a sketch, has no convergence or acceptance-rate analysis, and is not used in the simulation studies, so the main reported results remain affected by this mismatch.
- [§6, Theorem 3] Theorem 3 bounds d(P_N, P_n) for the unweighted subsample predictive and does not involve the coreset weights returned by Algorithm 1. The proof also repeats d(P_N, p0) instead of using d(P_N, p0) and d(P_n, p0) in the triangle inequality, and the rate ϵ_{N,n} = ϵ_N + ϵ_n does not vanish for fixed coreset size n as N → ∞. Consequently, Theorem 3 does not establish that weighting improves over uniform subsampling; a statement bounding d(P_N, P_n^{ω}) for ω = \barω is needed.
- [§3.2 and §4.2] The chosen transformation family T(y⋆_{1:n}) = (w_1 y⋆_1, ..., w_n y⋆_n) is not closed on the sample spaces used in the examples. In the logistic regression application the response is binary, so scaling y_i ∈ {0,1} by a general weight w_i produces values outside the original space; the product ground metric d((x1,y1),(x2,y2))² = d_L2(x1,x2)² + 1(y1≠y2) is then evaluated on points that are not elements of the original space. The linear-map parametrization therefore needs to be justified separately for categorical or discrete components, and the claim that the method applies to arbitrary non-Euclidean spaces is not supported by the implemented transform.
- [§4.1–§5] The simulations validate the coreset against full-data inference under the same DP-mixture model class that defines the coreset objective (DP-based predictive matching). This makes the reported 71–91% improvement rates partly circular: they show that the coreset can reproduce the model's own predictive approximation, but they do not demonstrate preservation of inference under a different model, which is the model-agnostic claim. The comparison to uniform subsampling is a useful baseline, but an independent test with a different kernel, a different hyperprior, or a likelihood-free simulator is needed to support the claimed generality.
minor comments (6)
- [Algorithm 2, line 9; Algorithm 3, line 10] Both algorithms return \barω = (1/M)Σ_t ω_t, but M is the posterior predictive sample size; the averaging index should be niter as in Algorithm 1.
- [§6] Theorem 2 uses nt for the number of simulations while Algorithm 1 uses niter and Theorem 3 uses Mn; the roles of n, N, M, and niter should be defined consistently and unambiguously.
- [Proof of Theorem 2] The line '∥w1 − w2∥2 < δ implies , d(w1 ⊙ x, w2 ⊙ x) < ε' contains a stray comma and should be written as a proper implication with an explicit dependence δ(ε).
- [Figure 1] The text and caption disagree about which panel is the histogram and which shows the posterior mean densities; please correct the cross-reference and the caption ordering.
- [Figure 4 caption] The caption refers to 'Algorithm 3' for the partition coreset, but Algorithm 3 is introduced only in Section 7; the partition procedure is Algorithm 2, so the caption is inconsistent with the text.
- [Definition 3.1] Definition 3.1 does not specify the family of transformations T or the discrepancy d, and no measurability, existence, or uniqueness conditions are given for the argmin; these should be stated before the theoretical analysis.
Circularity Check
No significant circularity: the predictive-coreset objective and its DP approximation are independent of the reported evaluations, and no load-bearing self-citation is used.
full rationale
The core construction in Definition 3.1 and Eq. (2) is a variational minimization over transforms T; the target distributions are the posterior predictives, which the paper approximates via an external NPL/DP model (Lyddon et al., 2018). This approximation is a stated modeling choice, not a consequence of the coreset definition. Algorithm 1 solves per-draw empirical Wasserstein problems and averages the argmins; Theorem 2 asserts convergence to the population argmin, and while the proof is incomplete because it does not establish an argmin/expectation interchange, this is a rigor gap rather than a self-referential reduction. The simulation studies compare coresets against uniform subsampling under the same DP mixture model, so construction and evaluation share modeling assumptions; however, the reported metrics (KL divergence of estimated densities, L2 distance of logits, variation of information) are not identical to the training objective, and in 19-29% of repeats the uniform subsample wins, so the empirical claim is not forced by construction. References to contraction rates (Camerlenghi et al., 2022; Nguyen, 2016) are external mathematical results with stated assumptions, not self-citations. No step in the paper defines the target in terms of the output or renames a fitted parameter as a prediction.
Assumptions & free parameters
free parameters (4)
- Coreset size n =
n = 50 (density), 20 (logistic), 50 (partitions)
- Posterior predictive sample size M =
M = 200 (density), 100 (logistic), 500 (partitions)
- DP concentration parameter α =
Not reported
- Monte Carlo iterations niter =
Not reported
assumptions (6)
- domain assumption The data sequence is exchangeable.
- domain assumption The DP prior in (3) provides a tractable and sufficiently accurate substitute for the true posterior predictive of the inference model.
- domain assumption The empirical distribution of M Pólya-urn samples faithfully represents the posterior predictive distribution.
- ad hoc to paper Averaging the per-draw argmin over t approximates the argmin of the expected discrepancy in (2).
- ad hoc to paper Linear maps y ↦ w⊙y are sufficiently expressive to match the relevant predictive distributions.
- domain assumption The base metric space (X,d) is totally bounded with packing number Nδ ∼ δ^{-a} and enough finite moments for p0.
Cite this review
Pith. "Pith review of Predictive Coresets." pith.science (2026). https://pith.science/paper/3SOW4HNO
@misc{pith2026250205725,
author = {Pith},
title = {Pith review of: Predictive Coresets},
year = {2026},
howpublished = {\url{https://pith.science/paper/3SOW4HNO}},
note = {Machine review of arXiv:2502.05725}
}
read the original abstract
Modern data analysis often involves massive datasets with hundreds of thousands of observations, making traditional inference algorithms computationally prohibitive. Coresets are selection methods designed to choose a smaller subset of observations while maintaining similar learning performance. Conventional coreset approaches determine these weights by minimizing the Kullback-Leibler (KL) divergence between the likelihood functions of the full and weighted datasets; as a result, this makes them ill-posed for nonparametric models, where the likelihood is often intractable. We propose an alternative variational method which employs randomized posteriors and finds weights to match the unknown posterior predictive distributions conditioned on the full and reduced datasets. Our approach provides a general algorithm based on predictive recursions suitable for nonparametric priors. We evaluate the performance of the proposed coreset construction on diverse problems, including random partitions and density estimation.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
E. Bernton, P. E. Jacob, M. Gerber, and C. P. Robert. Approximate B ayesian computation with the W asserstein distance. Journal of the Royal Statistical Society Series B: Statistical Methodology, 81 0 (2): 0 235--269, 02 2019
work page 2019
- [2]
-
[3]
T. Broderick, N. Boyd, A. Wibisono, A. C. Wilson, and M. I. Jordan. Streaming variational B ayes. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013 a
work page 2013
-
[4]
T. Broderick, J. Pitman, and M. I. Jordan. Feature allocations, probability functions, and paintboxes. Bayesian Analysis, 8: 0 801--836, 2013 b
work page 2013
-
[5]
Wasserstein posterior contraction rates in non-dominated Bayesian nonparametric models
F. Camerlenghi, E. Dolera, S. Favaro, and E. Mainini. Wasserstein posterior contraction rates in non-dominated B ayesian nonparametric models. 2022. URL arxiv:2201.12225
work page Pith review arXiv 2022
-
[6]
T. Campbell and T. Broderick. Automated Scalable Bayesian Inference via Hilbert Coresets . Journal of Machine Learning Research, 20 0 (15): 0 1--38, 2019
work page 2019
-
[7]
M. Catalano and H. Lavenant. Hierarchical integral probability metrics: A distance on random probability measures with low sample complexity. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Le...
work page 2024
- [8]
Show all 30 references
-
[9]
Clyde and H
M. Clyde and H. Lee. Bagging and the B ayesian bootstrap. In T. S. Richardson and T. S. Jaakkola, editors, Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics, volume R3 of Proceedings of Machine Learning Research, pages 57--62. PMLR, 01 2001
2001
-
[10]
Dolera and E
E. Dolera and E. Regazzini. Uniform rates of the G livenko– C antelli convergence and their use in approximating B ayesian inferences. Bernoulli, 25 0 (4A): 0 2982--3015, 11 2019
2019
-
[11]
D. Feldman. Core-sets: An updated survey. WIREs Data Mining and Knowledge Discovery, 10 0 (1): 0 e1335, 2020
2020
-
[12]
Flamary, K
R. Flamary, K. Lounici, and A. Ferrari. Concentration bounds for linear M onge mapping estimation and optimal transport domain adaptation. 2019. URL arxiv:1905.10155
2019 arXiv
-
[13]
E. Fong, C. Holmes, and S. G. Walker. Martingale posterior distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85 0 (5): 0 1357--1391, 02 2024
2024
-
[14]
Fortini and S
S. Fortini and S. Petrone. Predictive construction of priors in B ayesian nonparametrics . Brazilian Journal of Probability and Statistics, 26 0 (4): 0 423 -- 449, 2012
2012
-
[15]
Ghosal and A
S. Ghosal and A. van der Vaart. Fundamentals of Nonparametric Bayesian Inference. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2017
2017
-
[16]
M. D. Hoffman, D. M. Blei, C. Wang, and J. Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14 0 (1): 0 1303–1347, 05 2013
2013
-
[17]
J. H. Huggins, T. Campbell, and T. Broderick. Coresets for scalable B ayesian logistic regression. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, page 4087–4095. Curran Associates Inc., 2016
2016
-
[18]
H \"u tter and P
J.-C. H \"u tter and P. Rigollet. Minimax estimation of smooth optimal transport maps . The Annals of Statistics, 49 0 (2): 0 1166 -- 1194, 2021. doi:10.1214/20-AOS1997
2021 doi
-
[19]
Johndrow, N
J. Johndrow, N. Pillai, and A. Smith. No free lunch for approximate mcmc. 10 2020. URL arxiv:2010.12514
2020 arXiv
-
[20]
J. F. C. Kingman. Uses of exchangeability. The Annals of Probability, 6 0 (2): 0 183--197, 1978
1978
-
[21]
Liu and D
Q. Liu and D. Wang. S tein variational gradient descent: A general purpose B ayesian inference algorithm. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016
2016
-
[22]
Lyddon, S
S. Lyddon, S. Walker, and C. C. Holmes. Nonparametric learning from Bayesian models with randomized objective functions. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018
2018
-
[23]
Manousakas, Z
D. Manousakas, Z. Xu, C. Mascolo, and T. Campbell. Bayesian pseudocoresets. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14950--14960. Curran Associates, Inc., 2020
2020
-
[24]
M. Meil a . Comparing clusterings—an information based distance. Journal of Multivariate Analysis, 98 0 (5): 0 873--895, 2007
2007
-
[25]
X. Nguyen. Borrowing strengh in hierarchical B ayes: Posterior concentration of the D irichlet base measure. Bernoulli, 22 0 (3): 0 1535--1571, 2016
2016
-
[26]
Quiroz, M
M. Quiroz, M. Villani, R. Kohn, and M.-N. Tran. Subsampling MCMC -- an introduction for the survey statistician. Sankhya A, 80 0 (Suppl 1): 0 33--69, 2018
2018
-
[27]
S. L. Scott, A. W. Blocker, and F. V. et al. Bayes and big data: the consensus M onte C arlo algorithm. International Journal of Management Science and Engineering Management, 11 0 (2): 0 78--88, 2016
2016
-
[28]
Sesia, S
M. Sesia, S. Bates, E. Candès, J. Marchini, and C. Sabatti. False discovery rate control in genome-wide association studies with population structure. Proceedings of the National Academy of Sciences, 118 0 (40), 2021
2021
-
[29]
C. Villani. Topics in Optimal Transportation. Graduate studies in mathematics. American Mathematical Society, 2003
2003
-
[30]
Winter, T
S. Winter, T. Campbell, L. Lin, S. Srivastava, and D. B. Dunson. Machine learning and the future of B ayesian computation, 2023. URL arxiv:2304.11251
2023 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.