REVIEW 2 major objections 3 minor 2 cited by
Robust Conformal Outlier Detection under Contaminated Reference Data
T0 review · 2 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Even a contaminated reference set can keep conformal outlier detection conservative, and a small labeling budget can recover most of the lost power.
desk verdict Solid practical paper, but the main proof of Theorem 3.1 has a genuine gap under its stated budget condition; the method still looks good and the theorem is probably repairable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument rests on the quantile-threshold view of the conformal p-value: rejection $\hat{p}_{n+1} \le \alpha$ is equivalent to the test score exceeding the empirical quantile $\hat{Q}^{\mathrm{cal}}_{1-\alpha}$ of the calibration scores. Conditioning on the unordered score multiset and using exchangeability of the inlier scores with the null test score gives the penalty term in Lemma 2.2, which counts how many contaminated outliers have scores below that quantile. For Label-Trim, the proof introduces a mirror procedure that labels the test point as well, preserving exchangeability, and shows that the quantile of the mirror trimmed set is no larger than the quantile used by Label-Trim; this transfers the clean-set bound to the selectively trimmed set.
What would settle it
Construct a contamination scenario where outliers are deliberately placed in the bulk of the inlier score distribution, for example by drawing them from a shifted cluster that the scoring model cannot distinguish on nonconformity scores, with fixed $n_0$ and $n_1$. If the empirical type-I error of the standard conformal test is then consistently above $\alpha$, Assumption 2.3 is violated and the conservativeness claim fails exactly in the regime the paper excludes; for Label-Trim, repeating the same experiment with $m > \alpha(n+1)$ tests whether the theorem's budget condition or the separation assumption is the operative gate.
Extended reading notes
Core claim
Under the fixed-composition contamination model where $n_0$ calibration inliers and $n_1$ calibration outliers are drawn from distinct distributions, the paper proves in Lemma 2.2 that, under the null hypothesis, the conformal p-value obeys $P(\hat{p}_{n+1} \le \alpha) \le \alpha - \frac{n_1}{n_0+1}\bigl(1-\alpha - E[\hat{F}_1(\hat{Q}^{\mathrm{cal}}_{1-\alpha})]\bigr)$. The correction subtracts a term that is positive exactly when the contaminated outliers rarely fall below the calibration quantile; in that common case the test is conservative rather than invalid. For its Label-Trim procedure, Theorem 3.1 gives the analogous bound $P(\hat{p}^{\mathrm{LT}}_{n+1} \le \alpha) \le \alpha + \frac{1}{n_0+1} - E\bigl[\frac{\hat{n}^{\mathrm{LT}}_1}{n_0+1}((1-\alpha)-\hat{F}^{\mathrm{LT}}_1(\hat{Q}^{\mathrm{LT}}_{1-\alpha}))\bigr]$, provided the labeling budget satisfies $m \le \alpha(n+1)$. This shows that selectively annotating the highest-scoring points and removing only the confirmed outliers yields approximately valid conformal inference, with the remaining slack controlled by the number of inliers and by how few contaminating scores survive below the refined quantile. The paper presents this as an explanation of the conservative behavior observed in practice and as a practical way to recover power without discarding inliers.
Load-bearing premise
The guarantees depend on the contaminated outliers in the calibration set rarely scoring below the calibration quantile (Assumption 2.3), which is what separates inliers from outliers; if the detection model cannot separate them, contamination can inflate the false-positive rate, and the Label-Trim bound additionally needs the labeling budget $m$ not to exceed $\alpha(n+1)$.
Editorial extensions
If this is right
- If the outlier scores are typically larger than inlier scores, standard conformal outlier detection remains valid under calibration contamination, with type-I error below $\alpha$ and decreasing as the number of contaminated points grows.
- Deleting the top-scoring calibration points without checking their labels can inflate the type-I error above the nominal level, so reliable cleaning needs annotations rather than blind trimming.
- With a labeling budget $m \le \alpha(n+1)$, Label-Trim keeps the false-positive rate near $\alpha$ on a large contaminated calibration set, and the bound improves as the number of inliers $n_0$ grows.
- Label-Trim recovers most of the oracle power at small contamination rates and small $\alpha$, where the power loss from conservativeness is otherwise largest.
- A moderate budget of tens of labels on a large contaminated set can beat a small clean reference set, whose coarse p-value granularity limits its power.
Reading between the lines
- The analysis implies a safe default for handling uncertain labels: keep an ambiguous calibration point as an inlier rather than drop it, because conservativeness is bounded while careless trimming is not.
- One testable extension is to relax the budget condition $m \le \alpha(n+1)$ under the same non-adversarial separation assumption; the paper's own experiments find validity beyond the theorem's range, suggesting a sharper constant is available.
- The same mirror-exchangeability proof strategy could apply to other selective data-cleaning rules, such as budgeted active learning that queries points with uncertain scores rather than only the largest scores.
- When the detection model is weak, active trimming of top scores becomes less useful; comparing Label-Trim against random annotation of the same budget gives a practical diagnostic for when the score model is separable enough to justify selective cleaning.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies conformal outlier detection when the calibration (reference) set contains a small fraction of outliers. It first quantifies when standard conformal p-values remain conservative under contamination, via Lemma 2.2 and Assumption 2.3. It then proposes Label-Trim, an active data-cleaning method that uses a limited labeling budget to annotate the most suspicious calibration scores and removes only the confirmed outliers. Theorem 3.1 is stated as an approximate type-I error guarantee for this procedure. The empirical section compares Standard, Oracle, Naive-Trim, Small-Clean, and Label-Trim on three tabular and six visual datasets, reporting type-I error and power with standard errors. The central formal claim is that Label-Trim approximately controls the type-I error under a budget condition and mild separation conditions.
Significance. The question is timely and practically relevant: contaminated reference sets are common in anomaly detection, and the paper gives a clean exchangeability-based explanation of the empirically observed conservativeness. Lemma 2.2 is transparent and correct, and the connection to Sesia et al. (2024) is useful. The proposed Label-Trim procedure is intuitive, the experiments are extensive (nine datasets, multiple base models, standard errors, ablations on the budget and contamination rate), and the code is released. However, the formal justification of Label-Trim in Theorem 3.1 has a load-bearing proof gap, so the main theoretical contribution is not established as written. The empirical evidence alone is suggestive but cannot replace the advertised guarantee.
major comments (2)
- [Appendix A.3.2, Eq. (12)] The inequality ihat^LT = ceil((1-alpha)(nLT+1)) <= n+1-(m+1) is asserted to follow from m <= alpha(n+1), but it is false in general. For n=3, alpha=0.5, m=2, and nLT=n=3, the left side is ceil(0.5*4)=2 while the right side is 4-3=1. The same configuration also breaks the subsequent ordering (9): with inlier scores 10 and 11, outlier score 1, and null test score 0.5, Label-Trim removes no outlier so Q^{LT,n+1} is the second-smallest score in {0.5,1,10,11}, namely 1, while the mirror procedure removes the outlier and its quantile is the second-smallest of {0.5,10,11}, namely 10. Thus the proof of Theorem 3.1 does not establish the stated bound under its stated hypothesis. A sufficient repair is to strengthen the budget condition to m <= floor(alpha(n+1))-1, but this excludes the headline experimental configuration (shuttle, n=2500, alpha=0.02, m=50).
- [Appendix A.3.3, Eq. (11)] The proof of (11) uses the equality P(Sn+1=si | Ein, Eout, tilde{D}labeled, tilde{D}inlier_labeled) = P(Sn+1=si | Ein, Eout), asserting that the mirror selection carries no information about the test score beyond the unordered score sets. This is not correct: the event that a particular set of labels is selected among the top m+1 scores depends on the relative rank of Sn+1. For a minimal example with n0=1 calibration inlier, n1=1 outlier, and mirror budget m+1=1, conditioning on the event that the calibration inlier is selected forces Sn+1 to be the smaller of the two inlier scores, so the conditional probability is 0 or 1 rather than 1/2. The mirror procedure therefore does not preserve exchangeability conditionally on the labeled set, and the derivation of the crucial upper bound (11) is unsupported.
minor comments (3)
- [Algorithm 1] The input line says 'contaminate calibration-set'; this should read 'contaminated calibration set'.
- [Figure 4 and Appendix B.2.1] The statement that 'for labeling budgets m > 50 the condition in Theorem 3.1 no longer holds' is only correct for the specific shuttle configuration; since the theorem's proof is currently invalid, the sentence should be revised to reflect the actual status of the theoretical condition.
- [Table 1 and Table 3] For Small-Clean with alpha=0.01, the reported power is 0.0 with standard error 0.0; it would be helpful to state explicitly that this occurs because the smallest achievable p-value, roughly 1/(m+1), exceeds alpha, as is done later for the tabular experiments.
Circularity Check
No circularity found: the main derivations are self-contained and the cited prior work is contextual rather than load-bearing.
full rationale
The central results are derived directly from exchangeability and standard conformal calibration facts, not from a fitted quantity or from an assertion equivalent to the target. Lemma 2.2 is proved in Appendix A.2.1 by conditioning on the unordered inlier and outlier score sets and using the exchangeability relation P(S_{n+1}=s_i | E_in, E_out)=1/(n_0+1); it does not presuppose Assumption 2.3, which is instead stated after the lemma as a sufficient condition for conservativeness. Theorem 3.1, the main justification for Label-Trim, is proved through an independent mirror-procedure argument in Appendix A.3; it does not reduce to Lemma 2.2 or to any fitted parameter. The references to Sesia et al. (2024) are presented as related results: Lemma 2.2 is said to be "related to" their Theorem 1, and Corollary A.3 is explicitly a reformulation of that theorem, but the paper proves its own version and does not use Corollary A.3 to establish the Label-Trim guarantee. No uniqueness theorem is imported, no ansatz is smuggled in through a citation, and no known empirical pattern is merely renamed as a new contribution. The empirical evaluation is against an external Oracle benchmark and real datasets, so the reported behavior is falsifiable. The reviewer-flagged gap in the proof of Theorem 3.1, concerning inequality (12) and the case analysis in Appendix A.3.2, is a correctness concern, not a circularity: even if the stated proof is invalid, that does not mean the theorem's conclusion is assumed as an input. Under the circularity criteria, the derivation chain is self-contained and no prediction reduces by construction to its own inputs.
Assumptions & free parameters
free parameters (1)
- Labeling budget m =
50 (main experiments; varied 10-100 in Figures 4, 8, 9)
assumptions (6)
- domain assumption Inlier calibration and test points are i.i.d. from P0, and outliers are i.i.d. from P1 with P1 != P0.
- domain assumption The score function s is pre-trained on a separate dataset Dtrain and is fixed before calibration.
- ad hoc to paper Assumption 2.3: E[hat F1(hat Qcal_{1-alpha})] < 1 - alpha.
- standard math Nonconformity scores are almost surely distinct, with no ties.
- ad hoc to paper Theorem 3.1 requires the labeling budget m to satisfy m <= alpha(n+1).
- domain assumption The outlier detection model is relatively accurate and the outlier distribution is non-adversarial.
Cite this review
Pith. "Pith review of Robust Conformal Outlier Detection under Contaminated Reference Data." pith.science (2026). https://pith.science/paper/76YZD5KO
@misc{pith2026250204807,
author = {Pith},
title = {Pith review of: Robust Conformal Outlier Detection under Contaminated Reference Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/76YZD5KO}},
note = {Machine review of arXiv:2502.04807}
}
read the original abstract
Conformal prediction is a flexible framework for calibrating machine learning predictions, providing distribution-free statistical guarantees. In outlier detection, this calibration relies on a reference set of labeled inlier data to control the type-I error rate. However, obtaining a perfectly labeled inlier reference set is often unrealistic, and a more practical scenario involves access to a contaminated reference set containing a small fraction of outliers. This paper analyzes the impact of such contamination on the validity of conformal methods. We prove that under realistic, non-adversarial settings, calibration on contaminated data yields conservative type-I error control, shedding light on the inherent robustness of conformal methods. This conservativeness, however, typically results in a loss of power. To alleviate this limitation, we propose a novel, active data-cleaning framework that leverages a limited labeling budget and an outlier detection model to selectively annotate data points in the contaminated reference set that are suspected as outliers. By removing only the annotated outliers in this ``suspicious'' subset, we can effectively enhance power while mitigating the risk of inflating the type-I error rate, as supported by our theoretical analysis. Experiments on real datasets validate the conservative behavior of conformal methods under contamination and show that the proposed data-cleaning strategy improves power without sacrificing validity.
Figures
Figures from the paper (17 more)
Forward citations
Cited by 2 Pith papers
-
ACS: An interactive framework for conformal selection
ACS provides finite-sample false discovery rate control for interactive, adaptive selection of promising candidates from unlabeled pools.
-
Self-Poisoning in Adaptive Out-of-Distribution Detection: A Sharp-Threshold Theory and Certified Label-Free Calibration
In adaptive OOD detection, bank impurity follows a mean-field urn law whose kernel slope acts as a reproduction number; a frozen-reserve gate removes the supercritical collapse, and a two-world theorem caps label-free...
Reference graph
Works this paper leans on
-
[1]
KDD Cup 1999 Data Set. https://www.kaggle. com/mlg-ulb/creditcardfraud. Not normal- ized, without duplicates, categorial attributes removed. Accessed: January,
work page 1999
-
[10]
and therefore, X i∈Dinlier ∪{n+1} I h si > ˆQn+1 1−α i ≤ α(n + 1)− X i∈Doutlier I h si > ˆQn+1 1−α i • (iii) Since ˆQcal 1−α ≥ ˆQn+1 1−α almost surely, increasing the threshold (i.e., using ˆQcal 1−α) results in an equal or larger value of the sum. 15 Robust Conformal Outlier Detection under Contaminated Reference Data Now, we can derive an upper bound fo...
work page 2024
-
[11]
≤ ⌈(1 − α)(nLT + 1)⌉ = ˆiLT. Using the fact that ˜iLT ≤ ˆiLT almost surely, we prove in Appendix A.3.2 that, almost surely, ˜QLT,n+1 1−α ≤ ˆQLT,n+1 1−α . (9) 17 Robust Conformal Outlier Detection under Contaminated Reference Data Since we already knew that ˆQLT 1−α ≥ ˆQLT,n+1 1−α , this implies: ˆQLT 1−α ≥ ˆQLT,n+1 1−α ≥ ˜QLT,n+1 1−α . (10) Therefore, the...
work page 2024
-
[12]
operates on feature representations extracted from a pre-trained ResNet-18 model (Zhang et al., 2024; He et al., 2016). The model applies a percentile-based threshold (set to 90%) to truncate activations, where the threshold is computed on the contaminated train set. These truncated activations then pass through the fully connected layer of the pre-traine...
work page 2024
-
[13]
operates on feature representations extracted from a pre-trained ResNet-18 model (Zhang et al., 2024; He et al., 2016). The model rescales the activations using a sample-specific factor, defined as the sum of all activations divided by the sum of activations below a certain percentile (set to 65%). Similar to ReAct, the outlier score is computed using an ...
work page 2024
-
[1995]
Breunig, M. M., Kriegel, H.-P., Ng, R. T., and Sander, J. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93–104,
work page 2000
-
[2012]
10 Robust Conformal Outlier Detection under Contaminated Reference Data Netzer, Y ., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y . Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp
work page 2011
-
[2013]
Liu, F. T., Ting, K. M., and Zhou, Z.-H. Isolation forest. In 2008 eighth IEEE international conference on data mining, pp. 413–422. IEEE,
work page 2008
Show all 13 references
-
[2019]
Pytorch cifar models.https://github.com/ chenyaofo/pytorch-cifar-models
Chen, Y . Pytorch cifar models.https://github.com/ chenyaofo/pytorch-cifar-models. Accessed: 2025-5-17. Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., , and Vedaldi, A. Describing textures in the wild. In Proceed- ings of the IEEE Conf. on Computer Vision and Pattern Recogni...
2025
-
[2021]
Estimating the conformal prediction threshold from noisy labels
Penso, C., Goldberger, J., and Fetaya, E. Estimating the conformal prediction threshold from noisy labels. arXiv preprint arXiv:2501.12749,
-
[2023]
Pre- dictive uncertainty quantification with missing covariates
Zaffran, M., Josse, J., Romano, Y ., and Dieuleveut, A. Pre- dictive uncertainty quantification with missing covariates. arXiv preprint arXiv:2405.15641,
-
[2024]
PAC prediction sets under label shift
Si, W., Park, S., Lee, I., Dobriban, E., and Bastani, O. PAC prediction sets under label shift. arXiv preprint arXiv:2310.12964,
-
[2025]
URL https: //openreview.net/forum?id=L7sQ8CW2FY
ISSN 2835-8856. URL https: //openreview.net/forum?id=L7sQ8CW2FY. Chalapathy, R. and Chawla, S. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407,
1901 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.