Pith. sign in

REVIEW 2 major objections 3 minor 2 cited by

Robust Conformal Outlier Detection under Contaminated Reference Data

T0 review · 2 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read Even a contaminated reference set can keep conformal outlier detection conservative, and a small labeling budget can recover most of the lost power.

desk verdict Solid practical paper, but the main proof of Theorem 3.1 has a genuine gap under its stated budget condition; the method still looks good and the theorem is probably repairable. read the letter →

arxiv 2502.04807 v2 pith:76YZD5KO submitted 2025-02-07 stat.ML cs.LGstat.ME

classification stat.MLcs.LGstat.ME
keywords conformalpredictionoutlierdetectioncontaminatedreferencesettype-Ierrorcontrolselectiveannotationdatacleaningp-values
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies what happens to conformal outlier detection when the calibration set is not perfectly clean but contains a small fraction of outliers. Its main claim is that, under realistic non-adversarial conditions, these outliers do not break the type-I error guarantee: standard conformal p-values become conservative, rejecting true inliers less often than the nominal level $\alpha$. Since conservativeness is lost detection power, the paper introduces Label-Trim, which uses a limited labeling budget to annotate the most suspicious calibration points and removes only the confirmed outliers. It proves approximate type-I error control for this procedure on a large contaminated calibration set, and real-data experiments show that Label-Trim approaches the power of an oracle clean reference set while the naive strategy of simply deleting high-scoring points does not control the error.

What carries the argument

The argument rests on the quantile-threshold view of the conformal p-value: rejection $\hat{p}_{n+1} \le \alpha$ is equivalent to the test score exceeding the empirical quantile $\hat{Q}^{\mathrm{cal}}_{1-\alpha}$ of the calibration scores. Conditioning on the unordered score multiset and using exchangeability of the inlier scores with the null test score gives the penalty term in Lemma 2.2, which counts how many contaminated outliers have scores below that quantile. For Label-Trim, the proof introduces a mirror procedure that labels the test point as well, preserving exchangeability, and shows that the quantile of the mirror trimmed set is no larger than the quantile used by Label-Trim; this transfers the clean-set bound to the selectively trimmed set.

What would settle it

Construct a contamination scenario where outliers are deliberately placed in the bulk of the inlier score distribution, for example by drawing them from a shifted cluster that the scoring model cannot distinguish on nonconformity scores, with fixed $n_0$ and $n_1$. If the empirical type-I error of the standard conformal test is then consistently above $\alpha$, Assumption 2.3 is violated and the conservativeness claim fails exactly in the regime the paper excludes; for Label-Trim, repeating the same experiment with $m > \alpha(n+1)$ tests whether the theorem's budget condition or the separation assumption is the operative gate.

Watch

Extended reading notes

Core claim

Under the fixed-composition contamination model where $n_0$ calibration inliers and $n_1$ calibration outliers are drawn from distinct distributions, the paper proves in Lemma 2.2 that, under the null hypothesis, the conformal p-value obeys $P(\hat{p}_{n+1} \le \alpha) \le \alpha - \frac{n_1}{n_0+1}\bigl(1-\alpha - E[\hat{F}_1(\hat{Q}^{\mathrm{cal}}_{1-\alpha})]\bigr)$. The correction subtracts a term that is positive exactly when the contaminated outliers rarely fall below the calibration quantile; in that common case the test is conservative rather than invalid. For its Label-Trim procedure, Theorem 3.1 gives the analogous bound $P(\hat{p}^{\mathrm{LT}}_{n+1} \le \alpha) \le \alpha + \frac{1}{n_0+1} - E\bigl[\frac{\hat{n}^{\mathrm{LT}}_1}{n_0+1}((1-\alpha)-\hat{F}^{\mathrm{LT}}_1(\hat{Q}^{\mathrm{LT}}_{1-\alpha}))\bigr]$, provided the labeling budget satisfies $m \le \alpha(n+1)$. This shows that selectively annotating the highest-scoring points and removing only the confirmed outliers yields approximately valid conformal inference, with the remaining slack controlled by the number of inliers and by how few contaminating scores survive below the refined quantile. The paper presents this as an explanation of the conservative behavior observed in practice and as a practical way to recover power without discarding inliers.

Load-bearing premise

The guarantees depend on the contaminated outliers in the calibration set rarely scoring below the calibration quantile (Assumption 2.3), which is what separates inliers from outliers; if the detection model cannot separate them, contamination can inflate the false-positive rate, and the Label-Trim bound additionally needs the labeling budget $m$ not to exceed $\alpha(n+1)$.

Editorial extensions

If this is right

  • If the outlier scores are typically larger than inlier scores, standard conformal outlier detection remains valid under calibration contamination, with type-I error below $\alpha$ and decreasing as the number of contaminated points grows.
  • Deleting the top-scoring calibration points without checking their labels can inflate the type-I error above the nominal level, so reliable cleaning needs annotations rather than blind trimming.
  • With a labeling budget $m \le \alpha(n+1)$, Label-Trim keeps the false-positive rate near $\alpha$ on a large contaminated calibration set, and the bound improves as the number of inliers $n_0$ grows.
  • Label-Trim recovers most of the oracle power at small contamination rates and small $\alpha$, where the power loss from conservativeness is otherwise largest.
  • A moderate budget of tens of labels on a large contaminated set can beat a small clean reference set, whose coarse p-value granularity limits its power.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The analysis implies a safe default for handling uncertain labels: keep an ambiguous calibration point as an inlier rather than drop it, because conservativeness is bounded while careless trimming is not.
  • One testable extension is to relax the budget condition $m \le \alpha(n+1)$ under the same non-adversarial separation assumption; the paper's own experiments find validity beyond the theorem's range, suggesting a sharper constant is available.
  • The same mirror-exchangeability proof strategy could apply to other selective data-cleaning rules, such as budgeted active learning that queries points with uncertain scores rather than only the largest scores.
  • When the detection model is weak, active trimming of top scores becomes less useful; comparing Label-Trim against random annotation of the same budget gives a practical diagnostic for when the score model is separable enough to justify selective cleaning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper studies conformal outlier detection when the calibration (reference) set contains a small fraction of outliers. It first quantifies when standard conformal p-values remain conservative under contamination, via Lemma 2.2 and Assumption 2.3. It then proposes Label-Trim, an active data-cleaning method that uses a limited labeling budget to annotate the most suspicious calibration scores and removes only the confirmed outliers. Theorem 3.1 is stated as an approximate type-I error guarantee for this procedure. The empirical section compares Standard, Oracle, Naive-Trim, Small-Clean, and Label-Trim on three tabular and six visual datasets, reporting type-I error and power with standard errors. The central formal claim is that Label-Trim approximately controls the type-I error under a budget condition and mild separation conditions.

Significance. The question is timely and practically relevant: contaminated reference sets are common in anomaly detection, and the paper gives a clean exchangeability-based explanation of the empirically observed conservativeness. Lemma 2.2 is transparent and correct, and the connection to Sesia et al. (2024) is useful. The proposed Label-Trim procedure is intuitive, the experiments are extensive (nine datasets, multiple base models, standard errors, ablations on the budget and contamination rate), and the code is released. However, the formal justification of Label-Trim in Theorem 3.1 has a load-bearing proof gap, so the main theoretical contribution is not established as written. The empirical evidence alone is suggestive but cannot replace the advertised guarantee.

major comments (2)
  1. [Appendix A.3.2, Eq. (12)] The inequality ihat^LT = ceil((1-alpha)(nLT+1)) <= n+1-(m+1) is asserted to follow from m <= alpha(n+1), but it is false in general. For n=3, alpha=0.5, m=2, and nLT=n=3, the left side is ceil(0.5*4)=2 while the right side is 4-3=1. The same configuration also breaks the subsequent ordering (9): with inlier scores 10 and 11, outlier score 1, and null test score 0.5, Label-Trim removes no outlier so Q^{LT,n+1} is the second-smallest score in {0.5,1,10,11}, namely 1, while the mirror procedure removes the outlier and its quantile is the second-smallest of {0.5,10,11}, namely 10. Thus the proof of Theorem 3.1 does not establish the stated bound under its stated hypothesis. A sufficient repair is to strengthen the budget condition to m <= floor(alpha(n+1))-1, but this excludes the headline experimental configuration (shuttle, n=2500, alpha=0.02, m=50).
  2. [Appendix A.3.3, Eq. (11)] The proof of (11) uses the equality P(Sn+1=si | Ein, Eout, tilde{D}labeled, tilde{D}inlier_labeled) = P(Sn+1=si | Ein, Eout), asserting that the mirror selection carries no information about the test score beyond the unordered score sets. This is not correct: the event that a particular set of labels is selected among the top m+1 scores depends on the relative rank of Sn+1. For a minimal example with n0=1 calibration inlier, n1=1 outlier, and mirror budget m+1=1, conditioning on the event that the calibration inlier is selected forces Sn+1 to be the smaller of the two inlier scores, so the conditional probability is 0 or 1 rather than 1/2. The mirror procedure therefore does not preserve exchangeability conditionally on the labeled set, and the derivation of the crucial upper bound (11) is unsupported.
minor comments (3)
  1. [Algorithm 1] The input line says 'contaminate calibration-set'; this should read 'contaminated calibration set'.
  2. [Figure 4 and Appendix B.2.1] The statement that 'for labeling budgets m > 50 the condition in Theorem 3.1 no longer holds' is only correct for the specific shuttle configuration; since the theorem's proof is currently invalid, the sentence should be revised to reflect the actual status of the theoretical condition.
  3. [Table 1 and Table 3] For Small-Clean with alpha=0.01, the reported power is 0.0 with standard error 0.0; it would be helpful to state explicitly that this occurs because the smallest achievable p-value, roughly 1/(m+1), exceeds alpha, as is done later for the tabular experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the main derivations are self-contained and the cited prior work is contextual rather than load-bearing.

full rationale

The central results are derived directly from exchangeability and standard conformal calibration facts, not from a fitted quantity or from an assertion equivalent to the target. Lemma 2.2 is proved in Appendix A.2.1 by conditioning on the unordered inlier and outlier score sets and using the exchangeability relation P(S_{n+1}=s_i | E_in, E_out)=1/(n_0+1); it does not presuppose Assumption 2.3, which is instead stated after the lemma as a sufficient condition for conservativeness. Theorem 3.1, the main justification for Label-Trim, is proved through an independent mirror-procedure argument in Appendix A.3; it does not reduce to Lemma 2.2 or to any fitted parameter. The references to Sesia et al. (2024) are presented as related results: Lemma 2.2 is said to be "related to" their Theorem 1, and Corollary A.3 is explicitly a reformulation of that theorem, but the paper proves its own version and does not use Corollary A.3 to establish the Label-Trim guarantee. No uniqueness theorem is imported, no ansatz is smuggled in through a citation, and no known empirical pattern is merely renamed as a new contribution. The empirical evaluation is against an external Oracle benchmark and real datasets, so the reported behavior is falsifiable. The reviewer-flagged gap in the proof of Theorem 3.1, concerning inequality (12) and the case analysis in Appendix A.3.2, is a correctness concern, not a circularity: even if the stated proof is invalid, that does not mean the theorem's conclusion is assumed as an input. Under the circularity criteria, the derivation chain is self-contained and no prediction reduces by construction to its own inputs.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The central results depend on exchangeability, a fixed pretrained score function, and Assumption 2.3. The labeling budget m is a user input, not a fitted constant, though the main theorem requires it to be small relative to the calibration size.

free parameters (1)
  • Labeling budget m = 50 (main experiments; varied 10-100 in Figures 4, 8, 9)
    User-specified budget; the theoretical validity condition in Theorem 3.1 is m <= alpha(n+1).
assumptions (6)
  • domain assumption Inlier calibration and test points are i.i.d. from P0, and outliers are i.i.d. from P1 with P1 != P0.
    This is the setup in equations (1) and (3); all proofs assume this exchangeable structure.
  • domain assumption The score function s is pre-trained on a separate dataset Dtrain and is fixed before calibration.
    Section 2.1; needed so inlier scores in the calibration set and the test score are exchangeable under the null.
  • ad hoc to paper Assumption 2.3: E[hat F1(hat Qcal_{1-alpha})] < 1 - alpha.
    Imposed to characterize non-adversarial settings; if it fails, contamination can inflate rather than deflate the type-I error.
  • standard math Nonconformity scores are almost surely distinct, with no ties.
    Stated in the proof of Lemma 2.2; the authors note ties can be broken by adding small random noise.
  • ad hoc to paper Theorem 3.1 requires the labeling budget m to satisfy m <= alpha(n+1).
    This condition is needed for the mirror construction in the proof; the experiments suggest the method is robust beyond it, but that is not formally covered.
  • domain assumption The outlier detection model is relatively accurate and the outlier distribution is non-adversarial.
    Stated in Section 2.3 as the practical justification for Assumption 2.3; it is plausible but not formally verified on every dataset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Robust Conformal Outlier Detection under Contaminated Reference Data." pith.science (2026). https://pith.science/paper/76YZD5KO

@misc{pith2026250204807,
  author       = {Pith},
  title        = {Pith review of: Robust Conformal Outlier Detection under Contaminated Reference Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/76YZD5KO}},
  note         = {Machine review of arXiv:2502.04807}
}
read the original abstract

Conformal prediction is a flexible framework for calibrating machine learning predictions, providing distribution-free statistical guarantees. In outlier detection, this calibration relies on a reference set of labeled inlier data to control the type-I error rate. However, obtaining a perfectly labeled inlier reference set is often unrealistic, and a more practical scenario involves access to a contaminated reference set containing a small fraction of outliers. This paper analyzes the impact of such contamination on the validity of conformal methods. We prove that under realistic, non-adversarial settings, calibration on contaminated data yields conservative type-I error control, shedding light on the inherent robustness of conformal methods. This conservativeness, however, typically results in a loss of power. To alleviate this limitation, we propose a novel, active data-cleaning framework that leverages a limited labeling budget and an outlier detection model to selectively annotate data points in the contaminated reference set that are suspected as outliers. By removing only the annotated outliers in this ``suspicious'' subset, we can effectively enhance power while mitigating the risk of inflating the type-I error rate, as supported by our theoretical analysis. Experiments on real datasets validate the conservative behavior of conformal methods under contamination and show that the proposed data-cleaning strategy improves power without sacrificing validity.

Figures

Figures reproduced from arXiv: 2502.04807 by the authors.

Figure 1
Figure 1. Histogram of nonconformity scores for inliers and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. A schematic illustration of the proposed active data-cleaning of the contaminated reference set (Algo￾rithm 1). The approach begins by computing nonconfor￾mity scores of the contaminated reference data Dcal using a pretrained outlier detection model, where blue circles denote inliers and red circles denote outliers. The scores are then sorted in increasing order, and the top m samples—those most likely to be outlier… view at source ↗
Figure 3
Figure 3. Comparison of conformal outlier detection methods on a tabular dataset (“shuttle”) as a function of the contamina [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Comparison of conformal outlier detection methods on a tabular dataset (“shuttle”) as a function of the labeling [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Comparison of conformal outlier detection methods on a tabular dataset (“shuttle”) as a function of the target [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparison of conformal outlier detection methods on real dataset “credit-card” as a function of the contamination [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Comparison of conformal outlier detection methods on real dataset “KDDCup99” as a function of the contamination [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Performance on real dataset “credit-card” as a function of the labeling budget [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Performance on real dataset “KDDCup99” as a function of the labeling budget [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Performance on real dataset “KDDCup99” as a function of the labeling budget [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Comparison of conformal outlier detection methods on real dataset “credit-card” as a function of the target type-I [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Comparison of conformal outlier detection methods on real dataset “KDDCup99” as a function of the target [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Comparison of conformal outlier detection methods on real datasets as a function of the contamination rate [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Comparison of conformal outlier detection methods on real datasets as a function of the contamination rate [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Comparison of conformal outlier detection methods on real tabular datasets as a function of the contamination [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: Histogram of nonconformity scores for inliers and outliers in a contaminated calibration subset of the “shuttle” [PITH_FULL_IMAGE:figures/full_fig_p027_16.png]
Figure 17
Figure 17. Figure 17: Performance on a real dataset “shuttle” as a function of the target type-I error rate [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]
Figure 18
Figure 18. Figure 18: Performance on a real dataset “credit-card” as a function of the target type-I error rate [PITH_FULL_IMAGE:figures/full_fig_p028_18.png]
Figure 19
Figure 19. Figure 19: Performance on a real dataset “KDDCup99” as a function of the target type-I error rate [PITH_FULL_IMAGE:figures/full_fig_p029_19.png]
Figure 20
Figure 20. Figure 20: Performance on a real dataset “shuttle” as a function of drift in the outlier test distribution. Outliers in the training [PITH_FULL_IMAGE:figures/full_fig_p030_20.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ACS: An interactive framework for conformal selection

    stat.ME 2025-07 accept novelty 8.0 of 10

    ACS provides finite-sample false discovery rate control for interactive, adaptive selection of promising candidates from unlabeled pools.

  2. Self-Poisoning in Adaptive Out-of-Distribution Detection: A Sharp-Threshold Theory and Certified Label-Free Calibration

    cs.LG 2026-07 conditional novelty 6.0 of 10

    In adaptive OOD detection, bank impurity follows a mean-field urn law whose kernel slope acts as a reproduction number; a frozen-reserve gate removes the supercritical collapse, and a two-world theorem caps label-free...

Reference graph

Works this paper leans on

13 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [1]

    https://www.kaggle

    KDD Cup 1999 Data Set. https://www.kaggle. com/mlg-ulb/creditcardfraud. Not normal- ized, without duplicates, categorial attributes removed. Accessed: January,

  2. [10]

    and therefore, X i∈Dinlier ∪{n+1} I h si > ˆQn+1 1−α i ≤ α(n + 1)− X i∈Doutlier I h si > ˆQn+1 1−α i • (iii) Since ˆQcal 1−α ≥ ˆQn+1 1−α almost surely, increasing the threshold (i.e., using ˆQcal 1−α) results in an equal or larger value of the sum. 15 Robust Conformal Outlier Detection under Contaminated Reference Data Now, we can derive an upper bound fo...

  3. [11]

    Using the fact that ˜iLT ≤ ˆiLT almost surely, we prove in Appendix A.3.2 that, almost surely, ˜QLT,n+1 1−α ≤ ˆQLT,n+1 1−α

    ≤ ⌈(1 − α)(nLT + 1)⌉ = ˆiLT. Using the fact that ˜iLT ≤ ˆiLT almost surely, we prove in Appendix A.3.2 that, almost surely, ˜QLT,n+1 1−α ≤ ˆQLT,n+1 1−α . (9) 17 Robust Conformal Outlier Detection under Contaminated Reference Data Since we already knew that ˆQLT 1−α ≥ ˆQLT,n+1 1−α , this implies: ˆQLT 1−α ≥ ˆQLT,n+1 1−α ≥ ˜QLT,n+1 1−α . (10) Therefore, the...

  4. [12]

    The model applies a percentile-based threshold (set to 90%) to truncate activations, where the threshold is computed on the contaminated train set

    operates on feature representations extracted from a pre-trained ResNet-18 model (Zhang et al., 2024; He et al., 2016). The model applies a percentile-based threshold (set to 90%) to truncate activations, where the threshold is computed on the contaminated train set. These truncated activations then pass through the fully connected layer of the pre-traine...

  5. [13]

    credit-card

    operates on feature representations extracted from a pre-trained ResNet-18 model (Zhang et al., 2024; He et al., 2016). The model rescales the activations using a sample-specific factor, defined as the sum of all activations divided by the sum of activations below a certain percentile (set to 65%). Similar to ReAct, the outlier score is computed using an ...

  6. [1995]

    M., Kriegel, H.-P., Ng, R

    Breunig, M. M., Kriegel, H.-P., Ng, R. T., and Sander, J. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93–104,

  7. [2012]

    10 Robust Conformal Outlier Detection under Contaminated Reference Data Netzer, Y ., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y . Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp

  8. [2013]

    T., Ting, K

    Liu, F. T., Ting, K. M., and Zhou, Z.-H. Isolation forest. In 2008 eighth IEEE international conference on data mining, pp. 413–422. IEEE,

Show all 13 references
  1. [2019]

    Pytorch cifar models.https://github.com/ chenyaofo/pytorch-cifar-models

    Chen, Y . Pytorch cifar models.https://github.com/ chenyaofo/pytorch-cifar-models. Accessed: 2025-5-17. Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., , and Vedaldi, A. Describing textures in the wild. In Proceed- ings of the IEEE Conf. on Computer Vision and Pattern Recogni...

  2. [2021]

    Estimating the conformal prediction threshold from noisy labels

    Penso, C., Goldberger, J., and Fetaya, E. Estimating the conformal prediction threshold from noisy labels. arXiv preprint arXiv:2501.12749,

  3. [2023]

    Pre- dictive uncertainty quantification with missing covariates

    Zaffran, M., Josse, J., Romano, Y ., and Dieuleveut, A. Pre- dictive uncertainty quantification with missing covariates. arXiv preprint arXiv:2405.15641,

  4. [2024]

    PAC prediction sets under label shift

    Si, W., Park, S., Lee, I., Dobriban, E., and Bastani, O. PAC prediction sets under label shift. arXiv preprint arXiv:2310.12964,

  5. [2025]

    URL https: //openreview.net/forum?id=L7sQ8CW2FY

    ISSN 2835-8856. URL https: //openreview.net/forum?id=L7sQ8CW2FY. Chalapathy, R. and Chawla, S. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407,

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.