The reviewed record of science sign in
Pith

arxiv: 2606.22336 · v1 · pith:UK6DSCMR · submitted 2026-06-21 · stat.ML · cs.LG

Null-Calibrated Conformal Selection via Target-Membership Scores

Seungjin Choi This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-26 10:00 UTCgrok-4.3pith:UK6DSCMRrecord.jsonopen to challenge →

classification stat.ML cs.LG
keywords conformal selectiontarget membership scorefalse discovery ratefinite-sample validitynull exchangeabilitynonconformity scorecalibration
0
0 comments X

The pith

Target-membership scores produce finite-sample valid null p-values for conformal selection under null exchangeability.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Conformal selection identifies which test points have responses inside a chosen target region while keeping the false discovery rate controlled. Standard conformal methods borrow scores from prediction tasks such as residuals, yet these scores can rank candidates poorly when the target is an interval, a variance band, or a multimodal set rather than a simple mean. The paper instead uses the estimated probability that a point belongs to the target as its score; any monotone transform of this probability matches the Neyman-Pearson optimal ranking. Null-Calibrated Conformal Selection ranks each test score only against confirmed non-target calibration examples, yielding p-values that remain valid for any finite sample when the null exchangeability condition holds. These p-values can then be fed into the Benjamini-Yekutieli or Benjamini-Hochberg procedures to control FDR.

Core claim

Ranking test instances by their target-membership probabilities and calibrating those scores against confirmed non-target examples produces finite-sample valid null p-values under null exchangeability; the resulting p-values support FDR control via BY under arbitrary dependence or via BH under positive dependence, and the membership scores recover the same ranking as residual scores on mean-monotone targets while improving power on interval, variance-driven, or multimodal targets.

What carries the argument

Null-Calibrated Conformal Selection (NCCS), which generates null p-values by ranking each test target-membership score exclusively against the scores of confirmed non-target calibration examples.

If this is right

  • Membership scores match residual-based scores on mean-monotone targets.
  • Membership scores improve selection power on interval-valued, variance-driven, multimodal, or multi-condition targets.
  • NCCS p-values are finite-sample valid under null exchangeability.
  • The p-values can be plugged into BY for arbitrary dependence or BH for positive dependence to control FDR.
  • Direct empirical-FDP thresholding without NCCS calibration can be anti-conservative when targets are rare.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same membership-score principle could be applied to other binary conformal tasks such as outlier detection or region-based classification.
  • When exact exchangeability is hard to verify, one could monitor the empirical uniformity of the p-values on held-out null data as a diagnostic.
  • In sequential data streams the calibration set would need to be updated without breaking the exchangeability assumption used for validity.

Load-bearing premise

Test points must be exchangeable with the confirmed non-target calibration examples.

What would settle it

Generate data in which test points and non-target calibration examples come from visibly different distributions and check whether the NCCS p-values under the null deviate from uniformity.

Figures

Figures reproduced from arXiv: 2606.22336 by Seungjin Choi.

Figure 1
Figure 1. Figure 1: The target-membership principle (rows) and the calibration mechanism (columns) place the [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Schematic illustration of null-calibrated conformal selection (NCCS). The labeled data are split into an optional training subset Dtr, a score-learning subset Dsc, and a calibration subset Dcal. A target-membership score gˆ(x) ≈ P(Y ∈ I | X = x) is learned from membership labels Zi = 1{Yi ∈ I}, null calibration uses only points with Yi ∈ I / , and test candidates are selected by applying the Benjamini– Hoc… view at source ↗
read the original abstract

Conformal selection aims to identify test candidates whose unknown responses fall in a target region while controlling the false discovery rate. Existing methods often inherit prediction-oriented nonconformity scores, such as residual or clipped residual scores, from conformal prediction. We argue that the natural score for selection is instead the target-membership probability. This score directly addresses the binary event being selected, and any monotone transform of it gives the Neyman--Pearson oracle ranking at a fixed null selection level. This distinction is irrelevant for mean-monotone targets, where conventional scores induce essentially the same ranking, but becomes important for interval-valued, variance-driven, multimodal, or multi-condition targets, where prediction-oriented scores can be misaligned with selection power. We study membership-score-based conformal selection and isolate one conformal calibration route, Null-Calibrated Conformal Selection (NCCS), which ranks test scores against confirmed non-target calibration examples. Under null exchangeability, NCCS yields finite-sample valid null p-values, which can be combined with BY under arbitrary dependence or with BH under standard positive-dependence conditions. Experiments support the score principle: membership scores match conventional scores on mean-monotone targets, substantially improve over mean-score selection on variance-driven targets, and, when calibrated by NCCS, trade power for finite-sample null validity in rare-target regimes where direct empirical-FDP thresholding can be anti-conservative.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The manuscript proposes target-membership probability scores as the natural nonconformity measure for conformal selection tasks, arguing they align directly with the binary selection event and recover the Neyman-Pearson ranking for fixed null levels. It isolates Null-Calibrated Conformal Selection (NCCS), which ranks test membership scores against confirmed non-target calibration examples to produce finite-sample valid null p-values under exchangeability; these p-values are then combined with the BY procedure (arbitrary dependence) or BH (under PRDS) for FDR control. Experiments illustrate that membership scores match conventional residual scores on mean-monotone targets but improve power on variance-driven targets, while NCCS trades some power for validity in rare-target regimes.

Significance. If the central claims hold, the work supplies a score construction that is better matched to the selection objective than inherited prediction scores, with explicit finite-sample validity and standard multiple-testing compatibility. The parameter-free nature of the validity argument under the stated exchangeability condition and the reproducible experimental comparisons on non-mean-monotone targets are clear strengths.

minor comments (3)
  1. The precise definition and estimation procedure for the target-membership probability score should be stated in a dedicated subsection early in the methods, including any modeling assumptions required to obtain the probability estimate itself.
  2. In the experimental section, report the exact number of Monte Carlo repetitions and the precise construction of the variance-driven target regions so that the power gains can be reproduced.
  3. Clarify whether the confirmed non-target calibration examples are obtained via an initial screening step or assumed given; if the former, discuss any effect on the exchangeability condition.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive summary, significance assessment, and recommendation of minor revision. No major comments were provided in the report.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper's central validity claim for NCCS p-values follows directly from the classical rank-uniformity property of exchangeable scores (a standard result in conformal prediction), conditioned explicitly on the stated null exchangeability assumption. No step reduces a prediction or uniqueness claim to a fitted parameter, self-citation chain, or definitional tautology within the paper; the membership-score motivation is an independent modeling choice aligned with the binary selection task and does not alter the validity derivation. The subsequent FDR control steps invoke external multiple-testing theorems (BY, BH under PRDS) that are independent of the paper's construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Based on abstract only; full details on any additional parameters or assumptions not available.

axioms (1)
  • domain assumption null exchangeability
    Required for the finite-sample validity of the null p-values in NCCS.
invented entities (1)
  • target-membership probability score no independent evidence
    purpose: Directly measures the probability of belonging to the target region for selection purposes
    Introduced as the natural score for the binary selection event.

pith-pipeline@v0.9.1-grok · 5763 in / 1197 out tokens · 24321 ms · 2026-06-26T10:00:01.182025+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 2 canonical work pages

  1. [1]

    A. N. Angelopoulos and S. Bates. Conformal prediction: A gentle introduction. Foundations and Trends ^ in Machine Learning, 16 0 (4): 0 494--591, 2023

    2023

  2. [2]

    Bai and Y

    T. Bai and Y. Jin. Optimized conformal selection: Powerful selective inference after conformity score optimization, 2024. Preprint arXiv:2411.17983

    work page arXiv 2024

  3. [3]

    T. Bai, Y. Zhao, X. Yu, and A. Y. Yang. Multivariate conformal selection. In Proceedings of the International Conference on Machine Learning (ICML), 2025

    2025

  4. [4]

    R. F. Barber and E. J. Cand \`e . Controlling the false discovery rate via knockoffs. The Annals of Statistics, 43 0 (5): 0 2055--2085, 2015

    2055

  5. [5]

    Bates, E

    S. Bates, E. Cand \`e s, L. Lei, Y. Romano, and M. Sesia. Testing for outliers with conformal p -values. The Annals of Statistics, 51 0 (1): 0 149--178, 2023

    2023

  6. [6]

    Benjamini and Y

    Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple hypothesis testing. Journal of the Royal Statistical Society Series B, 57 0 (1): 0 289--300, 1995

    1995

  7. [7]

    Benjamini and D

    Y. Benjamini and D. Yekutieli. The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29 0 (4): 0 1165--1188, 2001

    2001

  8. [8]

    Cand \`e s, Y

    E. Cand \`e s, Y. Fan, L. Janson, and J. Lv. Planning for gold: 'model-x' knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society Series B, 80: 0 551--577, 2018

    2018

  9. [9]

    S. Choi. Conformal Bayes under label shift: Post-hoc calibration vs. in-training adaptation. In The 2nd Workshop on Epistemic Intelligence in Machine Learning, 2026

    2026

  10. [10]

    T. Ding, A. N. Angelopoulos, S. Bates, M. I. Jordan, and R. J. Tibshirani. Class-conditional conformal prediction with many classes. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  11. [11]

    G \'o mez-Bombarelli, J

    R. G \'o mez-Bombarelli, J. N. Wei, B. S \'a nchez-Lengeling, D. Sheberla, D. Duvenaud, J. M. Hern \'a ndez-Lobato, J. Aguilera-Iparraguirre, T. D. Hirzel, R. P. Adams, and A. Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science, 4: 0 268--276, 2018

    2018

  12. [12]

    Q. Hao, W. Liao, B. Jing, and H. Wei. Multi-condition conformal selection. In Proceedings of the International Conference on Learning Representations (ICLR), 2026

    2026

  13. [13]

    Jin and E

    Y. Jin and E. J. Cand \`e s. Selection by prediction with conformal p-values. Journal of Machine Learning Research, 24: 0 1--41, 2023

    2023

  14. [14]

    Jin and E

    Y. Jin and E. J. Cand \`e s. Model-free selective inference under covariate shift via weighted conformal p-values. Biometrika, 113 0 (1), 2026

    2026

  15. [15]

    S. R. K \"u nzel, J. S. Sekhon, P. J. Bickel, and B. Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, USA, 116 0 (10): 0 4156--4165, 2019

    2019

  16. [16]

    J. Lei, M. G’Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113 0 (523): 0 1094--1111, 2018

    2018

  17. [17]

    J. Qin, Y. Liu, M. Li, and C.-Y. Huang. Revamping conformal selection with optimal power: A Neyman–Pearson perspective, 2025. Preprint arXiv:2502.16513

    work page arXiv 2025

  18. [18]

    C. J. Stone. Optimal global rates of convergence for nonparametric regression. The Annals of Statistics, 10 0 (4): 0 1040--1053, 1982

    1982

  19. [19]

    J. D. Storey. A direct approach to false discovery rates. Journal of the Royal Statistical Society Series B, 64: 0 479--498, 2002

    2002

  20. [20]

    Sugiyama, T

    M. Sugiyama, T. Suzuki, S. Nakajima, H. Kashima, P. von B \"u nau, and M. Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60: 0 699--746, 2008

    2008

  21. [21]

    Sugiyama, T

    M. Sugiyama, T. Suzuki, and T. Kanamori. Density Ratio Estimation in Machine Learning. Cambridge University Press, 2012

    2012

  22. [22]

    A. B. Tsybakov. Introduction to Nonparametric Estimation. Springer, 2009

    2009

  23. [23]

    V. Vovk. Conditional validity of inductive conformal predictors. In Proceedings of the Asian Conference on Machine Learning (ACML), 2012

    2012

  24. [24]

    V. Vovk, A. Gammerman, and G. Shafer. Algorithmic Learning in a Random World. Springer, 2005

    2005

  25. [25]

    Wager and S

    S. Wager and S. Athey. Heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113 0 (523): 0 1228--1242, 2018

    2018

  26. [26]

    Zadrozny and C

    B. Zadrozny and C. Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the International Conference on Machine Learning (ICML), Williams Town, MA, 2001

    2001