Pith. sign in

REVIEW 3 major objections 5 minor 17 references

Conformal C2ST: Turning weak classifiers into strong two-sample tests

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read Conformal calibration of classifier scores turns any weak classifier into a valid two-sample test.

desk verdict The conformal C2ST paper is basically right; the stress-test worry about moment bounds not implying KS power is actually wrong, and the real issues are presentation gaps in the robustness theorem and the multiple-test null theory. read the letter →

arxiv 2507.17026 v2 pith:BR6WF6OL submitted 2025-07-22 stat.ML cs.LG

classification stat.MLcs.LG MSC 62G1062G0962F15
keywords two-sampletestingconformalpredictionclassifiertestneuralposteriorestimationvalidationdensityratioAUCfinite-samplevalidity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper answers an open question in classifier two-sample testing: whether a weak classifier can still yield a trustworthy test. It claims yes, by conformally calibrating the classifier's scores into p-values instead of thresholding them at 0.5. Two test variants are analyzed, a uniform test that resamples calibration sets for each test point and a multiple test that reuses a single calibration set to match the standard C2ST budget. The mathematical core is that the expected conformal p-value under the alternative is controlled by the score's AUC, so a classifier that merely ranks p-samples above q-samples slightly better than chance already generates detectable non-uniformity. The motivating application is validating neural posterior estimators, where a weak classifier often makes the standard C2ST's pass ambiguous.

What carries the argument

The central object is the conformal p-value $U$, the rank of a test point's score among $m$ calibration scores, with uniform random tie-breaking. The method depends only on the ordering of scores, so the oracle density ratio $r(x)=p(x)/q(x)$, and any strictly increasing transform of it, is the optimal score. The proofs connect $U$ to AUC and total variation through $\mathbb{E}[U]=1-\mathrm{AUC}(r)\le \frac12 - \frac12 TV(p,q)$, then control the error of a plug-in ratio estimate by comparing mis-ranking events $Z=r(\tilde X)-r(\tilde X')$ with their estimated counterparts, using a bounded-density low-noise assumption on $Z$ near zero and an optimized threshold to get the $O(\epsilon^{2/3})$ bound.

What would settle it

Simulate $p$ and $q$ with a density ratio that is locally flat or atomic over a region of positive $q$-mass, so that Assumption 3.4 fails, insert a classifier with measured mean-square error $\epsilon$, and run both conformal tests; if the rejection rate at $p \neq q$ remains at the nominal level for small but nonzero $\epsilon$, then the claimed power guarantee is not doing the work.

Watch

Extended reading notes

Core claim

The paper's central claim is a robustness guarantee for rank-based, conformal two-sample tests. Using any scoring function $s$ that approximates the oracle density ratio $r = p/q$ with mean-square error $\epsilon^2$ under $q$, the expected conformal p-value under the alternative stays within $O(\epsilon^{2/3})$ of the oracle p-value, provided a low-noise condition on $r$ holds; the multiple test's statistic enjoys the same relative gap. Under the null, the conformal p-values are exactly uniform for any $s$ and any finite calibration size, so type-I error is decoupled from classifier quality. Under the alternative, $\mathbb{E}[U] = 1-\mathrm{AUC}(r) \le \frac12 - \frac12 TV(p,q)$, so weak ranking signal translates directly into power. The conclusion is that a near-Bayes-optimal classifier is not required: a weak but informative ranking suffices.

Load-bearing premise

The load-bearing premise is that the true density ratio is not locally constant, in the sense that pairwise differences of $r$ under $q$ have a bounded density near zero, and that the learned score approximates the oracle ratio in mean square with error $\epsilon^2$; if either fails, most dramatically when the classifier is random, the $O(\epsilon^{2/3})$ power guarantee becomes vacuous and the test can sit at nominal power even though $p$ and $q$ differ.

Editorial extensions

If this is right

  • Posterior validation can be run with off-the-shelf or poorly calibrated classifiers, as long as their score ordering carries any signal.
  • The uniform test controls type-I error exactly at any finite calibration size, so it is a valid diagnostic even when training data are scarce.
  • The multiple test matches the standard C2ST's sample budget while keeping a relative power gap of $O(\epsilon^{2/3})$, making it a drop-in replacement for C2ST.
  • The tests detect discrepancies that affect the joint posterior without changing one-dimensional marginals, a regime where rank-based simulation-based calibration is insensitive.
  • On Gaussian perturbations, image corruptions, and a gravitational-lensing posterior benchmark, the conformal tests detect smaller misspecifications and retain more power under classifier degradation than the compared baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the test depends only on ranks, any monotone transform of the score yields the same p-values; this suggests one can freeze a feature extractor and use a lightweight head, and the method will still be valid.
  • The $O(\epsilon^{2/3})$ bound is worst-case; a sharper local expansion might show the effective cost of misestimation is often closer to $\epsilon^2$ when the score is near the oracle.
  • The same conformal calibration should extend to conditional two-sample tests with unequal marginals, provided a weak but informative ratio estimate is available.
  • A testable corollary is that power of the uniform test should saturate once $m$ is large relative to the reciprocal of the AUC deviation; the paper's ablations suggest $m=50$ is near that point.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes two conformal variants of the classifier two-sample test (C2ST): a uniform test that draws a fresh calibration set for every test point and a multiple test that shares one calibration set and matches the C2ST simulation budget. The authors show that under the null, conformal p-values are exactly uniform for any deterministic scoring function, and under the alternative the expected conformal p-value is 1 - AUC(r), which is at most 1/2 - TV(p,q)/2. They further prove an O(epsilon^{2/3}) bound on the gap between the expected p-values obtained from an oracle density ratio and from an estimated score with L2 error epsilon^2, under a low-noise condition on the density ratio. Experiments on perturbed Gaussian posteriors, CIFAR-10 corruptions, and a gravitational-lensing posterior benchmark demonstrate higher power and greater robustness to classifier degradation than standard C2ST and several other baselines.

Significance. If the theoretical claims are fully established, the paper makes a valuable practical point: exact finite-sample Type-I error control can be decoupled from classifier accuracy, and power degrades gently with the quality of the score function rather than collapsing when the classifier is far from Bayes optimal. This is directly relevant to simulation-based inference, where neural posterior estimators must be validated against the true posterior. The paper ships code, and the experiments cover a useful range of low-dimensional, high-dimensional, and real-data problems. The finite-sample validity lemma and the AUC-Total-Variation identity are clean and correct. The main caveat is that the central power claim for the uniform test, as stated, requires an additional elementary step connecting the first-moment bound to the KS sup-norm statistic; this step is missing from the manuscript. Theorems 3.5 and 4.2 are also stated more strongly than their proofs currently support. These issues are local and fixable rather than fundamental.

major comments (3)
  1. [§3.1, Lemma 3.3 and the KS statistic] The power argument for the uniform test jumps from the first-moment bound E[U] <= 1/2 - TV(p,q)/2 to the claim that the KS statistic sup_u |G(u)-u| has non-trivial power. This implication is true but is not stated or proved in the manuscript. If F(u)=P(U<=u), then integration by parts gives E[U]=1-∫_0^1 F(u)du, so |E[U]-1/2| <= sup_u |F(u)-u|. Therefore a mean gap of at least TV(p,q)/2 forces a population KS deviation of at least TV(p,q)/2, and the empirical KS statistic converges to this deviation by Glivenko-Cantelli. Please add this as an explicit lemma and use it to state the uniform test's power guarantee. As written, the central 'weak classifier suffices' claim for the uniform test is not certified by the displayed results.
  2. [Theorem 3.5 and Appendix A.2] Theorem 3.5 is stated as a finite-sample statement ('for all m > M'), but the proof only establishes the bound after passing to the limit m→∞ via the strong law of large numbers and dominated convergence. No argument is given for uniformity of the convergence in the error epsilon, and Lemma A.3's O(1/m) variance bound is not used to control the finite-m gap. Please restate Theorem 3.5 as an asymptotic result (m→∞, followed by n_q→∞), or prove a finite-m version with an explicit rate in m. This matters because Theorem 4.2 invokes Theorem 3.5 as a black box.
  3. [Theorem 4.2 and Appendix A.4.3] The proof of the relative-gap claim is too compressed in two places. First, the statement that the fluctuation term is 'O_p(n_q^{-1/2})' is only true after dividing the O_p(1) fluctuation gap by the O_p(√n_q Δ_r) denominator; the intermediate step should be shown explicitly. Second, the bound |AUC(hat r)-AUC(r)| <= C epsilon^{2/3} used for the drift gap follows from Theorem 3.5 only in the m→∞ limit and under tie-free scores; these conditions should be recorded in the statement of Theorem 4.2 or in its proof.
minor comments (5)
  1. [§5.1, Evaluation setup] The text says rejection rates are computed across 200 independent trials and later says 'All experiments use three random seeds'; please clarify whether the 200 trials are per seed or in total.
  2. [§5.1, Table 1 and Figure 2] Conformal Uniform is compared at a larger sampling budget from p (mN versus N for C2ST). The paper acknowledges this trade-off, but the main-text power comparison should state explicitly which curves use which value of m.
  3. [§3.1, paragraph after Lemma 3.3] The sentence 'conformal p-values become stochastically smaller than uniform' is stronger than what is proved; the proved statement is that their expectation is less than 1/2. Please rephrase to avoid claiming a stochastic ordering that does not follow from the displayed results.
  4. [Appendix A.2] The notation E_ξ[U_hat] is used before it is defined; please define it as expectation over the tie-breaking variable ξ conditional on the calibration and test data.
  5. [Abstract and Section 3.2] Minor wording issues include 'despite simplicity of the C2ST' in the abstract and the sentence 'We defer the proof to Appendix A.2', which appears mid-paragraph immediately after Theorem 3.5; both should be rephrased.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the conformal C2ST derivation is self-contained; validity follows from conformal exchangeability and power bounds are analytic identities, not fitted or citation-dependent conclusions.

full rationale

I walked the paper's derivation chain and found no step in which a claimed prediction reduces by construction to an input, a fitted parameter, or a load-bearing self-citation. Lemma 3.1 (exact finite-sample uniformity under H0) is a standard conformal property quoted from Vovk et al. (2005) and Lei & G'Sell (2018); it holds for any deterministic scoring function and any finite calibration size, and it does not depend on the paper's own experiments or fitted values. Lemma 3.2 and Corollary A.1 are proved by an explicit antisymmetry argument showing AUC(s) <= AUC(r), with equality iff s is a strictly increasing transform of the oracle density ratio; this is a mathematical statement about the definitions of AUC and r, not a fitted relation. Lemma 3.3 computes E[U] = 1 - AUC(r) by taking the m->infinity limit of the rank-based conformal p-value; this is an algebraic identity between two quantities defined from the same ranking, and it is not used to define either quantity. Theorem 3.5 is an analytic robustness bound: the proof decomposes E[U_hat] - E[U] into a term involving Z = r(X~)-r(X~'), uses the bounded-density condition (Assumption 3.4) and the assumed L2 error epsilon^2, and optimizes a threshold to obtain O(epsilon^{2/3}). No constant is tuned to data, and the bound is not derived from the conclusion it purports to support. Theorem 4.2 similarly follows from the U-statistic CLT, the tie-noise assumption, and the previous bound; even if its power implication is debatable (a moment bound may not control the KS statistic), that is a correctness or strength concern, not circularity. The only self-citations are the NPTBench benchmark suite (Chen et al. 2024) and the authors' own code link, used for experimental perturbations and reproducibility; neither is load-bearing for the theoretical claims. There is no uniqueness theorem imported from the authors' prior work, no ansatz smuggled in via self-citation, and no renaming of a known result as a new one; the paper explicitly credits Hu and Lei (2024) for the conformal framework and positions its contribution as robustness theory for weak plug-in classifiers. Under the stated review rules, the honest finding is 0: no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The theory relies on two regularity assumptions stated in the paper (Assumption 3.4 and 4.1), one unstated set of conditions inherited from Hu and Lei (2024), and two standard modeling assumptions for the NPE setting. No fitted constants appear in the theoretical claims; experimental hyperparameters are procedural choices.

assumptions (5)
  • domain assumption Assumption 3.4: Z = r(X~)-r(X~') has a density bounded in a neighborhood of zero under q.
    Used in Theorem 3.5 to truncate the small-|Z| region; the O(epsilon^{2/3}) rate depends on this regularity.
  • domain assumption Assumption 4.1: ties in the score r-hat occur at rate O_p(1).
    Required for the multiple test's tie-noise term to vanish; fails for discretized or clipped classifier outputs.
  • standard math Hu and Lei (2024, Theorem 2): asymptotic null normality and divergence under the alternative for the shared-calibration U-statistic.
    The multiple test's validity and power are inherited from this theorem; its regularity conditions are not restated in the paper.
  • domain assumption Shared marginal assumption: q(theta,y) = pi(y) q(theta|y) with the same y-marginal as p.
    Section 2 reduces posterior equality to joint two-sample testing; without this, the NPE framing does not hold.
  • domain assumption Absolute continuity of p(theta|y) and q(theta|y) with respect to a common base measure.
    Section 2; needed to define density ratios and classifier log-odds like p/q.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Conformal C2ST: Turning weak classifiers into strong two-sample tests." pith.science (2026). https://pith.science/paper/BR6WF6OL

@misc{pith2026250717026,
  author       = {Pith},
  title        = {Pith review of: Conformal C2ST: Turning weak classifiers into strong two-sample tests},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BR6WF6OL}},
  note         = {Machine review of arXiv:2507.17026}
}
abstract

The two-sample testing problem, a fundamental task in statistics and machine learning, seeks to determine whether two sets of samples, drawn from underlying distributions $p$ and $q$, are in fact identically distributed (i.e. whether $p=q$). A popular and intuitive approach is the classifier two-sample test (C2ST), where a classifier is trained to distinguish between samples from $p$ and $q$. Yet despite simplicity of the C2ST, its reliability hinges on access to a near-Bayes-optimal classifier, a requirement that is rarely met and difficult to verify. This raises a major open question: can a weak classifier still be useful for two-sample testing? We show that the answer is a definitive yes. Building on the work of Hu and Lei (2024), we analyze two conformal variants of the C2ST that convert the scores from any trained classifier -- even if weak, biased, or overfit -- into exact, finite-sample p-values. We establish two key theoretical properties of the conformal C2ST: (i) finite-sample Type-I error control, and (ii) non-trivial power that degrades gently in tandem with the error of the trained classifier. The upshot is that even poorly performing classifiers can yield powerful and reliable two-sample tests. This general framework finds a powerful application in Bayesian inference, particularly for validating Neural Posterior Estimation (NPE) models, where the task of comparing a learned posterior approximation $q(\theta \mid y)$ to the true posterior $p(\theta \mid y)$ can be framed as a two-sample test. Empirically, the Conformal C2ST outperforms classical discriminative tests across a wide range of benchmarks for this task. Our results establish the conformal C2ST as a practical, theoretically grounded diagnostic tool.

Figures

Figures reproduced from arXiv: 2507.17026 by the authors.

Figure 1
Figure 1. Power of the C2ST and conformal C2ST under shift and rotate perturbations of the optimal decision boundary. The conformal test is much more robust to a weak or misspecified classifier. This evaluation challenge is particularly acute in Neural Posterior Estimation (NPE), an increasingly popular and practical tool for Bayesian inference. NPE methods use sim￾ulations to train a deep generative model q(θ | y) to approx￾… view at source ↗
Figure 2
Figure 2. Statistical power of C2ST and conformal variants across benchmark perturbations. Panel (a) evaluates sensitivity to posterior mismatch; panel (b) evaluates robustness to classifier degradation. ground truth, we generate a series of flawed approximations q(θ | y), by systematically applying a controlled perturba￾tion of p. The magnitude of the perturbation is controlled by a scalar γ, which acts as a “difficulty dial… view at source ↗
Figure 3
Figure 3. Conformal C2ST consistently outperforms standard C2ST across different tasks on CIFAR-10. Solid lines correspond to well-trained classifiers (β = 0.0), dashed lines to weaker classifiers (β = 0.95). Representative examples for each task are shown below. α = 0.05) across 200 independent trials. All methods share a common base of N = 1000 observations but differ in how many additional draws they require, and from whic… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: illustrates samples from the true and approximate posteriors for various values of γ, projected onto the first two principal components using Principal Component Analysis (PCA) [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]
Figure 5
Figure 5. Figure 5: Power analysis under mean perturbation of the base distribution of the normalizing flow model. D A Practitioner’s Guide We collect here practical recommendations for applying the Conformal C2ST, distilling the theoretical and empirical findings of the main text into co…
Figure 6
Figure 6. Figure 6: Power analysis under Mean Shift 25 [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]
Figure 7
Figure 7. Figure 7: Power analysis under Covariance Scaling. (a) Statistical power as a function of γ (b) Statistical power under Conformal Uniform with varying m. (c) Statistical Power degradation under classifier degeneration [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]
Figure 8
Figure 8. Figure 8: Power analysis under Anisotropic Covariance Perturbation. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]
Figure 9
Figure 9. Figure 9: Power analysis under Heavy-Tailed Perturbation. (a) Statistical power as a function of γ (b) Statistical power under Conformal Uniform with varying m (c) Statistical Power degradation under classifier degeneration [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]
Figure 10
Figure 10. Figure 10: Power analysis under Additional Mode. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: Power analysis under Mode Collapse. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 8 canonical work pages

  1. [1]

    A two-sample conditional distribution test using conformal prediction and weighted rank sum

    Xiaoyu Hu and Jing Lei. A two-sample conditional distribution test using conformal prediction and weighted rank sum. Journal of the American Statistical Association, 119 0 (546): 0 1136--1154, 2024. doi:10.1080/01621459.2023.2177165. URL https://doi.org/10.1080/01621459.2023.2177165

  2. [2]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020

  3. [3]

    Compositional score modeling for simulation-based inference

    Tomas Geffner, George Papamakarios, and Andriy Mnih. Compositional score modeling for simulation-based inference. In International Conference on Machine Learning, pages 11098--11116. PMLR, 2023

  4. [4]

    Conditional diffusions for amortized neural posterior estimation

    Tianyu Chen, Vansh Bansal, and James G Scott. Conditional diffusions for amortized neural posterior estimation. In International Conference on Artificial Intelligence and Statistics, pages 2377--2385. PMLR, 2025

  5. [5]

    All-in-one simulation-based inference

    Manuel Gloeckler, Michael Deistler, Christian Weilbach, Frank Wood, and Jakob H Macke. All-in-one simulation-based inference. arXiv preprint arXiv:2404.09636, 2024

  6. [6]

    Normalizing flows for probabilistic modeling and inference

    George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22 0 (57): 0 1--64, 2021

  7. [7]

    Flow matching for scalable simulation-based inference

    Jonas Wildberger, Maximilian Dax, Simon Buchholz, Stephen Green, Jakob H Macke, and Bernhard Sch \"o lkopf. Flow matching for scalable simulation-based inference. Advances in Neural Information Processing Systems, 36: 0 16837--16864, 2023

  8. [8]

    Auto-encoding variational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013

Show all 17 references
  1. [9]

    Validating bayesian inference algorithms with simulation-based calibration

    Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Validating bayesian inference algorithms with simulation-based calibration. arXiv preprint arXiv:1804.06788, 2018

  2. [10]

    Sampling-based accuracy testing of posterior estimators for general inference

    Pablo Lemos, Adam Coogan, Yashar Hezaveh, and Laurence Perreault-Levasseur. Sampling-based accuracy testing of posterior estimators for general inference. In International Conference on Machine Learning, pages 19256--19273. PMLR, 2023

  3. [11]

    Revisiting classifier two-sample tests

    David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. arXiv preprint arXiv:1610.06545, 2016

  4. [12]

    Algorithmic Learning in a Random World

    Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005

  5. [13]

    Distribution-free predictive inference for regression

    Jing Lei and Max G’Sell. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113 0 (523): 0 1094--1111, 2018

  6. [14]

    E. L. Lehmann and Joseph P. Romano. Testing Statistical Hypotheses. Springer Texts in Statistics. Springer, New York, 3rd edition, 2005. ISBN 0-387-98864-5

  7. [15]

    An introduction to roc analysis

    Tom Fawcett. An introduction to roc analysis. Pattern Recognition Letters, 27 0 (8): 0 861--874, 2006

  8. [16]

    NPTBench : A benchmark suite for neural posterior testing

    Tianyu Chen, Vansh Bansal, and James Scott. NPTBench : A benchmark suite for neural posterior testing. Technical report, 2024. URL https://github.com/TianyuCodings/NPTBench

  9. [17]

    Density estimation using real nvp

    Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.