pith. machine review for the scientific record. sign in

arxiv: 2107.07511 · v6 · submitted 2021-07-15 · 💻 cs.LG · cs.AI· math.ST· stat.ME· stat.ML· stat.TH

Recognition: 3 theorem links

A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification

Anastasios N. Angelopoulos, Stephen Bates

Pith reviewed 2026-05-09 01:19 UTC · model claude-opus-4-7

classification 💻 cs.LG cs.AImath.STstat.MEstat.MLstat.TH MSC 62G1562G0868T05
keywords conformal predictiondistribution-free inferenceuncertainty quantificationprediction setsexchangeabilityquantile regressioncovariate shiftrisk control
0
0 comments X

The pith

Any pre-trained model can be wrapped into prediction sets with guaranteed finite-sample coverage, regardless of how the model was built or what the data distribution is.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents conformal prediction as a single-page recipe for turning any heuristic uncertainty signal — softmax scores, quantile estimates, predicted variances, Bayesian posteriors, OOD detectors — into prediction sets with a non-asymptotic coverage guarantee at a user-chosen level. The guarantee follows from exchangeability of calibration and test points alone, so it survives arbitrarily wrong models. The authors then catalogue what changes when you change the score function: adaptive sets that grow on hard inputs, conformalized quantile regression, conformalized scalar uncertainties, and Bayes-optimal sets when a posterior is available. They extend the recipe to group- and class-conditional coverage, monotone risk control beyond miscoverage, outlier detection, known covariate shift via likelihood-ratio reweighting, and unknown distribution drift via weighted calibration. They also lay out diagnostics: the conditional-on-calibration coverage is Beta-distributed, the empirical coverage over many splits is Beta-Binomial, and adaptivity must be checked separately from marginal coverage via size-stratified or feature-stratified metrics. A companion appendix generalizes to high-probability control of arbitrary, possibly non-monotone risks via multiple testing on a parameter grid.

Core claim

The paper organizes a body of work around a single thesis: any black-box predictor, however badly trained, can be wrapped in a short post-hoc calibration step that produces prediction sets guaranteed to contain the truth with user-specified probability, in finite samples, without assumptions on the model or the data distribution. The wrapper requires only a held-out calibration set, a scalar score function s(x,y) measuring disagreement between an input and a candidate label, and an empirical quantile of those scores. The authors argue this recipe — split conformal prediction — is general enough to cover classification, quantile regression, Bayesian posteriors, outlier detection, segmentation

What carries the argument

The core object is the empirical quantile q̂ of conformal scores s(X_i, Y_i) on a held-out calibration set, taken at level ⌈(n+1)(1−α)⌉/n; the prediction set is {y : s(X_test, y) ≤ q̂}. The work is done by exchangeability: the test score is equally likely to land in any of the n+1 gaps of the sorted calibration scores, which forces marginal coverage ≥ 1−α with no further assumption. Every extension (covariate shift, drift, risk control, group balance) is a re-weighting or re-grouping of this same quantile.

If this is right

  • Any deployed predictor — including a frozen neural network whose internals are unavailable — can be retrofitted with calibrated prediction sets using only a few hundred labeled holdout points and a few lines of code.
  • Improvements in uncertainty quantification reduce, in practice, to designing better score functions for a given task rather than to proving new coverage theorems.
  • Coverage guarantees extend cleanly past miscoverage to false-negative rate, false-discovery rate, IOU, and other bounded losses, by tuning a threshold on calibration data at a slightly conservative level.
  • Under known covariate shift, reweighting calibration scores by the likelihood ratio restores exact finite-sample coverage; under unknown drift, weighted calibration with a rolling window degrades coverage only in proportion to a total-variation distance.
  • Conditional coverage — the same guarantee for every subgroup or every input — is provably unattainable in general, so practitioners must check feature- and size-stratified coverage as a routine diagnostic rather than expecting marginal coverage to imply fairness across groups.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The framework's real cost is hidden in the choice of score function: validity is free, but informativeness (small sets) inherits all the failure modes of the underlying model, so conformal prediction launders calibration but not capability.
  • Because the guarantee is marginal over the calibration draw, two practitioners running the same procedure on different held-out sets will see coverage that differs by several percentage points; reporting a single conformal interval without the Beta-distribution caveat overstates what was actually controlled.
  • The risk-control extension via multiple testing on a parameter grid effectively recasts uncertainty quantification as a hypothesis-testing problem, which suggests power-versus-conservativeness tradeoffs from multiple-comparison theory will increasingly drive practical performance.
  • The drift bound's dependence on total-variation distance, which is essentially never measurable in deployment, means the time-series guarantees are honest about being heuristic — the actual safety in production comes from short windows and fast recalibration, not from a theorem.

Load-bearing premise

The whole guarantee rests on the calibration data and the future test point being interchangeable — drawn from the same distribution in a way that does not care about order. When that fails (real distribution shift, time series, selection bias), the guarantee degrades, and the patches the paper offers require either knowing the shift or guessing its size.

What would settle it

Run the split-conformal recipe at α=0.1 on a fresh i.i.d. classification or regression task with n≈1000 calibration points, repeat over many random splits, and check that the empirical coverage histogram matches the Beta(n+1−⌊(n+1)α⌋, ⌊(n+1)α⌋) distribution centered at 1−α. A systematic shortfall below 1−α on i.i.d. data, larger than the Beta-Binomial fluctuations the paper tabulates, would falsify the central claim.

read the original abstract

Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, time-series, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run using our codebase.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 11 minor

Summary. The manuscript is a self-contained tutorial on conformal prediction (CP) and related distribution-free uncertainty quantification techniques. It presents split CP with the standard marginal coverage guarantee 1−α ≤ P(Y∈C(X)) ≤ 1−α+1/(n+1) (Theorem 1, Appendix D), walks the reader through canonical score functions (APS §2.1, CQR §2.2, scalar uncertainty §2.3, Bayes-optimal §2.4), discusses adaptivity diagnostics and finite-sample coverage fluctuations (§3, Appendix C), and surveys extensions: group/class-conditional CP (§4.1–4.2), conformal risk control (§4.3), outlier detection (§4.4), covariate shift via weighting (§4.5), distribution drift (§4.6), full/cross/CV+ CP (§6), and Learn-then-Test for general risk control (Appendix A–B). Worked examples on Imagenet, MS-COCO, tumor segmentation, weather time-series, and toxic-comment detection are accompanied by short Python snippets and Jupyter notebooks. A historical section (§7) traces the development of CP from algorithmic randomness through to current trends.

Significance. The paper is explicitly expository and does not claim new theorems; its value lies in pedagogy, breadth of coverage, accurate attribution, and reproducibility. As an introduction it succeeds: the split-CP recipe is given in ~10 lines of NumPy with a correct ⌈(n+1)(1−α)⌉/n quantile correction, the proof in Appendix D is the standard exchangeability argument and is correctly stated, and the limits of the framework (marginal vs. conditional coverage, the impossibility result of [87], the continuity requirement for the upper bound, the unmeasurable TV distances in Theorem 4) are honestly disclosed at the relevant points (§3.1, §3.2, footnote 1, §5.3). The accompanying code and notebooks support reproducibility, and the bibliography is comprehensive and current. Tutorials of this scope and accuracy are genuinely useful to the community and have been heavily cited; the manuscript meets the standard for an accepted survey/tutorial.

minor comments (11)
  1. [§1, Eq. (1) and footnote 1] The upper bound 1−α+1/(n+1) requires continuous (tie-free) scores, as later stated in Theorem D.2 and footnote 1. Because Eq. (1) is the very first display in the paper and many readers will only skim, it would help to attach a short parenthetical at Eq. (1) itself (e.g., 'upper bound assumes continuous scores; see Thm. D.2') rather than deferring this caveat to a footnote and Appendix D. The current presentation could leave readers using discrete softmax outputs with the impression that they get the two-sided bound without randomized tie-breaking.
  2. [§1, calibration recipe] When introducing ˆq as the ⌈(n+1)(1−α)⌉/n empirical quantile, it would be worth pointing out explicitly that this requires α ≥ 1/(n+1); otherwise the algorithm returns the trivial set C(X)=Y. This corner case is handled implicitly in the proof of Theorem 1 but is not flagged in the main-text recipe, and beginners running the code with very small calibration sets may be confused.
  3. [§2.1, Eq. (3)] The +1 in 'k = sup{...} + 1' that ensures non-empty sets is stated without comment. A one-line note that this corresponds to the randomized correction of [4] omitted for simplicity (and a pointer to the linked notebook for the randomized version) would help reproducibility, since the deterministic version slightly over-covers.
  4. [§3.2, Table 1] Table 1 reports n(ε) for δ=0.1, α=0.1 only. Given that the surrounding text suggests n≈1000 as a rule of thumb, a second column or remark for at least one other (α,δ) pair (say α=0.05) would make the guidance more transferable; otherwise readers may misapply the n=1000 heuristic outside the regime in which it was computed.
  5. [§4.5, weighted CP] The display defining ˆq(x) as the 1−α quantile of a reweighted distribution silently assumes the scores have been pre-sorted (the manuscript notes this 'for notational convenience'). For a tutorial, an explicit version with general (unsorted) scores, or at least a sentence stating that ties and ordering require care, would prevent implementation bugs. The accompanying code does not appear in this section.
  6. [§4.6, Theorem 4] The bound 1−α−2Σw̃_iε_i contains TV distances ε_i that the manuscript itself acknowledges are 'never known' (§5.3). This is fine as stated, but readers would benefit from a sharper sentence next to Theorem 4 that the bound is best read as a structural statement (not a deployable certificate) and that the practical justification of the fixed-window/decay weights in §5.3 is heuristic. Currently this honest caveat is somewhat buried at the end of §5.3.
  7. [§5.5, Eq. (15) and ˆR+] The selective-classification example invokes the LTT machinery and a Binomial CDF upper bound, but the symbol δ is introduced only inside ˆR+(λ) without the user-facing reminder that the guarantee is now (1−δ) over the calibration draw rather than marginal. Given that earlier sections emphasized α as the only knob, a sentence flagging this shift from expectation-style to high-probability-style guarantees would aid the reader before Appendix A formalizes it.
  8. [Appendix A, Hoeffding p-value] The Hoeffding p-value in §A.1.1 assumes losses bounded in [0,1]. This is implicit in the surrounding text but not stated at the point of definition. Since the Appendix is meant to be a self-contained crash course, the boundedness assumption should be made explicit alongside the formula.
  9. [§7, Historical Notes] The historical section is engaging but in places mixes biographical anecdote with technical history in a way that is hard for a non-specialist to parse (e.g., the Bernoulli sequences/randomness deficiency thread). A short signposting sentence ('readers in a hurry can skip to Current Trends') would help, and the link from randomness deficiency to nonconformity scores deserves one more concrete sentence to make the connection clear.
  10. [Code listings (Figs. 2, 3, 5, 7, 12, 20, 23, 24)] The code samples mix np.quantile with method='higher' (Fig. 2) and interpolation='higher' (Figs. 3, 5) — the latter is the deprecated NumPy keyword. Standardizing on the current keyword and noting the NumPy version assumed would prevent silent deprecation warnings or errors for new users.
  11. [Figure 11 / Appendix C] The Beta(n+1−l, l) distribution of conditional coverage (with l=⌊(n+1)α⌋) is stated and plotted, but the exact relationship between this and the practical 'n≈1000' guideline could be tightened with one displayed inequality (e.g., a Hoeffding-style tail bound for the Beta) so the reader can compute n for their own (α, ε, δ) without running the notebook.

Simulated Author's Rebuttal

1 responses · 0 unresolved

The referee recommends acceptance and raises no major comments, judging the tutorial accurate, honestly scoped, well-attributed, and reproducible. As there are no specific revision requests, we have nothing substantive to contest or amend. We thank the referee for the careful reading and for confirming that the central guarantees, the proof in Appendix D, the discussion of conditional-coverage limits, and the bibliography are correctly and fairly presented. We will use the opportunity of the next revision only for minor typographical polishing and to refresh pointers to recent follow-up work, leaving the technical content reviewed by the referee unchanged.

read point-by-point responses

  1. Referee: The referee recommends acceptance and raises no major comments, noting that the tutorial succeeds at its pedagogical aims, that the split-CP recipe and Appendix D proof are correctly stated, that limitations (marginal vs. conditional coverage, impossibility of [87], continuity for the upper bound, unmeasurable TV distances in Theorem 4) are honestly disclosed at the relevant points, and that the accompanying code/notebooks and bibliography are comprehensive and current.

    Authors: We thank the referee for the careful and thorough reading of the manuscript and for the positive recommendation. We are grateful that the referee has verified the correctness of the central technical statements (Theorem 1 and the Appendix D proof, the ⌈(n+1)(1−α)⌉/n quantile correction in the code), the breadth and currency of the references, and the explicit disclosure of the framework's limitations at the relevant points (footnote 1 on tie-breaking, §3.1 on marginal vs. conditional coverage, §3.2 and Appendix C on finite-sample fluctuations, and §5.3 on the unmeasurability of the TV distances appearing in Theorem 4). Since the referee raised no major comments, no substantive revisions are required in response to this report. We will, however, take the opportunity of the next arXiv revision to fix any minor typographical issues that have been brought to our attention by readers since the v6 posting, and to refresh pointers to fast-moving follow-up work (e.g., online/adaptive conformal under distribution shift and conformal risk control), without altering the technical content the referee has reviewed. We thank the referee again for engaging with the manuscript in detail. revision: no

Circularity Check

0 steps flagged

No circularity: tutorial reproducing standard, externally-attributed results with a textbook proof.

full rationale

This paper is an expository introduction to conformal prediction. Its central technical claim — Theorem 1 / Theorem D.1, the split-conformal marginal coverage guarantee 1−α ≤ P(Y_test ∈ C(X_test)) ≤ 1−α+1/(n+1) — is attributed to Vovk, Gammerman, and Saunders [5] and proved in Appendix D by the standard exchangeability-of-ranks argument: under exchangeability of (s_1,...,s_n,s_test), P(s_test ≤ s_(k)) = k/(n+1), which immediately yields the bound when ˆq is set to s_⌈(n+1)(1−α)⌉. The proof's hypotheses (exchangeability, quantile definition) do not contain the conclusion; the conclusion follows from a combinatorial fact about ranks of exchangeable variables, which is independent of the present authors. Other major results are similarly attributed and proved or cited externally: CQR (Theorem implied, citing Romano et al. [8]), conformal risk control (Theorem 2, citing [17]), weighted/covariate-shift conformal (Theorem 3, citing Tibshirani et al. [25]), drift (Theorem 4, citing Barber et al. [26]), full conformal (Theorem 5, citing [1]), and Learn-then-Test (Theorem A.1, citing [18]). Self-citations exist (e.g., [4], [17], [18] include the present authors), but they are not load-bearing for the central marginal-coverage theorem, which predates the authors. No "prediction" in the paper is a fitted quantity renamed; ˆq is explicitly a calibration-set quantile and the coverage statement is a probabilistic statement about a held-out test point. The paper is honest about scope limitations the skeptic raised (marginal vs. conditional coverage in §3.1 citing impossibility result [87]; tie-breaking for the upper bound in footnote 1 and §6.1; unmeasurable TV distances in Theorem 4 acknowledged in §5.3). None of these are circular reductions; they are disclosed assumptions. The derivation chain is self-contained against external mathematical facts (exchangeability, order statistics, Hoeffding/Bentkus concentration, Bonferroni/sequential testing), and the cited results are mathematically standard and verifiable. Score: 0.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Model omitted the axiom ledger; defaulted for pipeline continuity.

pith-pipeline@v0.9.0 · 9711 in / 4596 out tokens · 79385 ms · 2026-05-09T01:19:54.790353+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 35 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. An Optimal Sauer Lemma Over $k$-ary Alphabets

    cs.LG 2026-04 unverdicted novelty 8.0

    A sharp Sauer inequality for multiclass and list prediction is established in terms of the DS dimension, tight for every alphabet size k, list size ℓ, and dimension value.

  2. Adaptive Stopping for Multi-Turn LLM Reasoning

    cs.CL 2026-04 unverdicted novelty 8.0

    MiCP is the first conformal prediction method for multi-turn LLM pipelines that allocates per-turn error budgets to enable adaptive stopping with an overall coverage guarantee, shown to reduce turns and cost on RAG an...

  3. GRAPHLCP: Structure-Aware Localized Conformal Prediction on Graphs

    cs.LG 2026-05 unverdicted novelty 7.0

    GRAPHLCP improves localized conformal prediction on graphs by using feature-aware densification and Personalized PageRank kernels to incorporate topology for better coverage and efficiency.

  4. TRACE: Transport Alignment Conformal Prediction via Diffusion and Flow Matching Models

    stat.ML 2026-05 unverdicted novelty 7.0

    TRACE creates valid conformal prediction sets for complex generative models by scoring outputs via averaged denoising or velocity errors along stochastic transport paths instead of likelihoods.

  5. When Does Trimming Help Conformal Prediction? A Retained-Law Diagnostic under Calibration Contamination

    stat.ML 2026-05 unverdicted novelty 7.0

    Trimming helps conformal prediction under contamination precisely when the anomaly score separates retention probabilities without biasing clean scores, otherwise the retained mixture coefficient prevents substantial ...

  6. In-Context Positive-Unlabeled Learning

    stat.ML 2026-05 unverdicted novelty 7.0

    PUICL is a transformer pretrained on synthetic PU data from structural causal models that solves positive-unlabeled classification via in-context learning without gradient updates or fitting.

  7. Delving into Non-Exchangeability for Conformal Prediction in Graph-Structured Multivariate Time Series

    cs.LG 2026-05 unverdicted novelty 7.0

    SCALE uses Spectral Graph Conditional Exchangeability (SGCE) and graph wavelets to achieve valid coverage and improved efficiency in conformal prediction for non-exchangeable graph time series by conformalizing high-f...

  8. SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation

    cs.CL 2026-05 unverdicted novelty 7.0

    SURE-RAG aggregates pair-level claim-evidence relations into interpretable signals for selective RAG answering, reaching 0.9075 Macro-F1 on HotpotQA-RAG v3 while providing auditability and reducing unsafe answers by 3...

  9. Intrinsic effective sample size for manifold-valued Markov chain Monte Carlo via kernel discrepancy

    stat.ML 2026-05 unverdicted novelty 7.0

    An intrinsic effective sample size for manifold MCMC is defined via kernel discrepancy as the number of independent draws yielding equivalent expected squared discrepancy to the target.

  10. Profile Likelihood Inference for Anisotropic Hyperbolic Wrapped Normal Models on Hyperbolic Space

    math.ST 2026-05 unverdicted novelty 7.0

    The profile maximum likelihood estimator for the location in anisotropic hyperbolic wrapped normal models is strongly consistent, asymptotically normal, and attains the Hájek-Le Cam minimax lower bound under squared g...

  11. Query-Efficient Quantum Approximate Optimization via Graph-Conditioned Trust Regions

    cs.LG 2026-04 unverdicted novelty 7.0

    A GNN predicts Gaussians over QAOA parameters to create graph-conditioned trust regions that reduce circuit evaluations for MaxCut from 85-343 down to 45 while keeping approximation ratios within 3 points of heuristics.

  12. Adaptive Conformal Anomaly Detection with Time Series Foundation Models for Signal Monitoring

    cs.LG 2026-04 unverdicted novelty 7.0

    A model-agnostic adaptive conformal anomaly detection approach uses weighted quantile bounds learned from past foundation model predictions to deliver interpretable p-value scores with stable calibration under shifts ...

  13. Causal inference for social network formation

    econ.EM 2026-04 conditional novelty 7.0

    Random team assignments in a professional firm reveal that indirect ties strongly increase new direct tie formation, while effects of degree and local density are smaller and less robust.

  14. Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

    cs.AI 2026-04 unverdicted novelty 7.0

    LLM judges display per-document transitivity violations in 33-67% of cases despite low aggregate rates, while conformal prediction set widths serve as reliable indicators of document-level difficulty with cross-judge ...

  15. Conformal Margin Risk Minimization: An Envelope Framework for Robust Learning under Label Noise

    cs.LG 2026-04 unverdicted novelty 7.0

    CMRM adds a conformal quantile regularization on prediction margins to any loss, improving noisy-label classification accuracy up to 3.39% across methods and benchmarks while preserving performance at zero noise.

  16. Conformal Risk Control under Non-Monotone Losses: Theory and Finite-Sample Guarantees

    stat.ML 2026-04 unverdicted novelty 7.0

    Conformal risk control for bounded non-monotone losses over a grid of size m achieves excess risk of order sqrt(log m / n) with n calibration samples, which is minimax optimal.

  17. Multi-Fidelity Quantile Regression

    stat.ME 2026-05 unverdicted novelty 6.0

    A model-agnostic two-stage estimator links high-fidelity quantiles to low-fidelity ones via a covariate-dependent level function for faster convergence and better accuracy with limited high-fidelity data.

  18. CONTRA: Conformal Prediction Region via Normalizing Flow Transformation

    stat.ML 2026-05 unverdicted novelty 6.0

    CONTRA generates sharp multi-dimensional conformal prediction regions by defining nonconformity scores as distances from the center in the latent space of a normalizing flow.

  19. Scale selection for geometric medians on product manifolds

    math.ST 2026-05 unverdicted novelty 6.0

    Joint location-scale minimization for geometric medians on product manifolds degenerates to marginal medians, and three new scale-selection methods restore identifiability with asymptotic guarantees.

  20. Conformal Agent Error Attribution

    cs.LG 2026-05 unverdicted novelty 6.0

    A new filtration-based conformal prediction method attributes errors in multi-agent systems by producing contiguous sequence sets with finite-sample coverage guarantees, enabling rollback recovery.

  21. Networked Information Aggregation for Binary Classification

    cs.LG 2026-05 unverdicted novelty 6.0

    Sequential prediction passing on DAGs for logistic regression yields O(M/sqrt(D)) excess loss when M-agent windows cover all features, with Omega(k/D) lower bound identifying depth as the fundamental limit.

  22. Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation

    cs.LG 2026-04 unverdicted novelty 6.0

    Unsupervised single-generation confidence calibration for reasoning LLMs via offline self-consistency proxy distillation outperforms baselines on math and QA tasks and improves selective prediction.

  23. DAG-STL: A Hierarchical Framework for Zero-Shot Trajectory Planning under Signal Temporal Logic Specifications

    cs.RO 2026-04 unverdicted novelty 6.0

    DAG-STL decomposes long-horizon STL planning into decomposition, timed waypoint allocation, and diffusion-based trajectory generation to enable zero-shot planning under unknown dynamics.

  24. Answer Only as Precisely as Justified: Calibrated Claim-Level Specificity Control for Agentic Systems

    cs.CL 2026-04 unverdicted novelty 6.0

    Compositional selective specificity (CSS) improves overcommitment-aware utility from 0.846 to 0.913 on LongFact while retaining 0.938 specificity by calibrating claim-level backoffs in agentic AI responses.

  25. Blind-Spot Mass: A Good-Turing Framework for Quantifying Deployment Coverage Risk in Machine Learning Systems

    cs.LG 2026-04 unverdicted novelty 6.0

    Blind-spot mass uses Good-Turing unseen-species estimation to measure the total probability of states with low empirical support, showing that 95% of operational mass lies in blind spots at tau=5 across wearable activ...

  26. Adaptive Conformal Prediction for Reliable and Explainable Medical Image Classification

    cs.CV 2026-05 unverdicted novelty 5.0

    An adaptive lambda criterion for RAPS achieves 95.72% global coverage and at least 90% coverage across all difficulty strata on medical image datasets while keeping average prediction set size at 1.09.

  27. Quantile-Free Uncertainty Quantification in Graph Neural Networks

    cs.LG 2026-05 unverdicted novelty 5.0

    QpiGNN provides a quantile-free dual-head architecture for GNN uncertainty quantification that directly optimizes coverage and interval width, yielding 22% higher coverage and 50% narrower intervals than baselines on ...

  28. Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Prediction

    cs.IR 2026-05 unverdicted novelty 5.0

    A conformal prediction filter for retrieval chunks plus an attention-based factuality classifier can raise RAG answer quality by up to 6% and detect inconsistent generations up to 77% of the time.

  29. An empirical evaluation of the risks of AI model updates using clinical data: stability, arbitrariness, and fairness

    cs.AI 2026-04 unverdicted novelty 5.0

    Updating clinical AI models can cause prediction flips, arbitrariness, and unfair error rates across groups, requiring dedicated monitoring dimensions.

  30. ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Control

    cs.RO 2026-04 unverdicted novelty 5.0

    ReconVLA enhances pretrained vision-language-action robotic policies with conformal prediction for uncertainty estimation and failure detection without retraining.

  31. Uncertainty-Aware Transformers: Conformal Prediction for Language Models

    cs.LG 2026-04 unverdicted novelty 5.0

    CONFIDE applies conformal prediction to transformer embeddings for valid prediction sets, improving accuracy up to 4.09% and efficiency over baselines on models like BERT-tiny.

  32. Probably Approximately Correct (PAC) Guarantees for Data-Driven Reachability Analysis: A Theoretical and Empirical Comparison

    eess.SY 2026-04 conditional novelty 5.0

    Formal connections between PAC bounds for three data-driven reachability methods are established, with empirical results showing they are not interchangeable despite similarities.

  33. Neural posterior estimation for scalable and accurate inverse parameter inference in Li-ion batteries

    physics.data-an 2026-04 unverdicted novelty 5.0

    NPE delivers millisecond-scale parameter inference for Li-ion batteries that matches or exceeds Bayesian calibration accuracy while adding local sensitivity interpretability, though with higher voltage prediction errors.

  34. AIVV: Neuro-Symbolic LLM Agent-Integrated Verification and Validation for Trustworthy Autonomous Systems

    cs.AI 2026-04 unverdicted novelty 5.0

    AIVV deploys LLM agents in a council to semantically validate anomalies in time-series data against natural-language requirements, automating human-in-the-loop verification for autonomous systems.

  35. Uncertainty in Physics and AI: Taxonomy, Quantification, and Validation

    stat.ML 2026-05 accept novelty 4.0

    A unified taxonomy of uncertainty in ML for physics is introduced together with validation tools such as coverage, calibration, and proper scoring rules, illustrated on regression and classification tasks.

Reference graph

Works this paper leans on

137 extracted references · 137 canonical work pages · cited by 35 Pith papers

  1. [1]

    V. Vovk, A. Gammerman, and G. Shafer, Algorithmic Learning in a Random World . Springer, 2005

    work page 2005

  2. [2]

    Inductive confidence machines for regression,

    H. Papadopoulos, K. Proedrou, V. Vovk, and A. Gammerman, “Inductive confidence machines for regression,” in Machine Learning: European Conference on Machine Learning , 2002, pp. 345–356

    work page 2002

  3. [3]

    Distribution-free prediction bands for non-parametric regression,

    J. Lei and L. Wasserman, “Distribution-free prediction bands for non-parametric regression,” Journal of the Royal Statistical Society: Series B: Statistical Methodology , pp. 71–96, 2014

    work page 2014

  4. [4]

    Uncertainty sets for image classifiers using conformal prediction,

    A. N. Angelopoulos, S. Bates, J. Malik, and M. I. Jordan, “Uncertainty sets for image classifiers using conformal prediction,” in International Conference on Learning Representations, 2021

    work page 2021

  5. [5]

    Machine-learning applications of algorithmic random- ness,

    V. Vovk, A. Gammerman, and C. Saunders, “Machine-learning applications of algorithmic random- ness,” in International Conference on Machine Learning , 1999, pp. 444–453

    work page 1999

  6. [6]

    Least ambiguous set-valued classifiers with bounded error levels,

    M. Sadinle, J. Lei, and L. Wasserman, “Least ambiguous set-valued classifiers with bounded error levels,” Journal of the American Statistical Association , vol. 114, pp. 223–234, 2019

    work page 2019

  7. [7]

    Mauricio Sadinle, Jing Lei, and Larry Wasserman

    Y. Romano, M. Sesia, and E. J. Cand` es, “Classification with valid and adaptive coverage,”arXiv:2006.02544, 2020

    work page arXiv 2006

  8. [8]

    Conformalized quantile regression,

    Y. Romano, E. Patterson, and E. Cand` es, “Conformalized quantile regression,” inAdvances in Neural Information Processing Systems, vol. 32, 2019, pp. 3543–3553

    work page 2019

  9. [9]

    Regression quantiles,

    R. Koenker and G. Bassett Jr, “Regression quantiles,” Econometrica: Journal of the Econometric Society, vol. 46, no. 1, pp. 33–50, 1978

    work page 1978

  10. [10]

    Image-to-image regression with distribution-free uncertainty quantification and applications in imaging,

    A. N. Angelopoulos, A. P. Kohli, S. Bates, M. I. Jordan, J. Malik, T. Alshaabi, S. Upadhyayula, and Y. Romano, “Image-to-image regression with distribution-free uncertainty quantification and applications in imaging,” arXiv preprint arXiv:2202.05265 , 2022. 32

    work page arXiv 2022

  11. [11]

    Bayes-optimal prediction with frequentist coverage control,

    P. Hoff, “Bayes-optimal prediction with frequentist coverage control,” arXiv:2105.14045, 2021

    work page arXiv 2021

  12. [12]

    Frasian inference,

    L. Wasserman, “Frasian inference,” Statistical Science, vol. 26, no. 3, pp. 322–325, 2011

    work page 2011

  13. [13]

    Comparing the bayes and typicalness frame- works,

    T. Melluish, C. Saunders, I. Nouretdinov, and V. Vovk, “Comparing the bayes and typicalness frame- works,” in European Conference on Machine Learning, Springer, 2001, pp. 360–371

    work page 2001

  14. [14]

    Conditional validity of inductive conformal predictors,

    V. Vovk, “Conditional validity of inductive conformal predictors,” in Proceedings of the Asian Con- ference on Machine Learning, vol. 25, 2012, pp. 475–490

    work page 2012

  15. [15]

    Knowing what you know: Valid and validated confidence sets in multiclass and multilabel prediction,

    M. Cauchois, S. Gupta, and J. Duchi, “Knowing what you know: Valid and validated confidence sets in multiclass and multilabel prediction,” arXiv:2004.10181, 2020

    work page arXiv 2004

  16. [16]

    Improving conditional coverage via orthogonal quantile regression,

    S. Feldman, S. Bates, and Y. Romano, “Improving conditional coverage via orthogonal quantile regression,” in Advances in Neural Information Processing Systems , 2021

    work page 2021

  17. [17]

    arXiv preprint arXiv:2208.02814 , year=

    A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster, “Conformal risk control,” arXiv preprint arXiv:2208.02814, 2022

    work page arXiv 2022

  18. [18]

    and Bates, Stephen and Cand

    A. N. Angelopoulos, S. Bates, E. J. Cand` es, M. I. Jordan, and L. Lei, “Learn then test: Calibrating predictive algorithms to achieve risk control,” arXiv:2110.01052, 2021

    work page arXiv 2021

  19. [19]

    A review of novelty detection,

    M. A. Pimentel, D. A. Clifton, L. Clifton, and L. Tarassenko, “A review of novelty detection,” Signal Processing, vol. 99, pp. 215–249, 2014

    work page 2014

  20. [20]

    Design of experiments,

    R. A. Fisher, “Design of experiments,” British Medical Journal, vol. 1, no. 3923, p. 554, 1936

    work page 1936

  21. [21]

    Significance tests which may be applied to samples from any populations,

    E. J. Pitman, “Significance tests which may be applied to samples from any populations,” Supplement to the Journal of the Royal Statistical Society , vol. 4, no. 1, pp. 119–130, 1937

    work page 1937

  22. [22]

    Testing exchangeability on-line,

    V. Vovk, I. Nouretdinov, and A. Gammerman, “Testing exchangeability on-line,” in Proceedings of the 20th International Conference on Machine Learning (ICML-03) , 2003, pp. 768–775

    work page 2003

  23. [23]

    Prediction and outlier detection in classification problems,

    L. Guan and R. Tibshirani, “Prediction and outlier detection in classification problems,” arXiv:1905.04396, 2019

    work page arXiv 1905

  24. [24]

    Testing for outliers with conformal p-values,

    S. Bates, E. Cand` es, L. Lei, Y. Romano, and M. Sesia, “Testing for outliers with conformal p-values,” arXiv:2104.08279, 2021

    work page arXiv 2021

  25. [25]

    Conformal prediction under covariate shift,

    R. J. Tibshirani, R. Foygel Barber, E. Candes, and A. Ramdas, “Conformal prediction under covariate shift,” in Advances in Neural Information Processing Systems 32 , 2019, pp. 2530–2540

    work page 2019

  26. [26]

    Candes, Aaditya Ramdas, and Ryan J

    R. F. Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani, “Conformal prediction beyond ex- changeability,” arXiv:2202.13415, 2022

    work page arXiv 2022

  27. [27]

    Conformal prediction with localization.arXiv preprint arXiv:1908.08558,

    L. Guan, “Conformal prediction with localization,” arXiv:1908.08558, 2020

    work page arXiv 1908

  28. [28]

    Microsoft coco: Common objects in context,

    T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision, Springer, 2014, pp. 740–755

    work page 2014

  29. [29]

    Shifts: A dataset of real distributional shift across multiple large-scale tasks.arXiv preprint arXiv:2107.07455, 2021

    A. Malinin, N. Band, G. Chesnokov, Y. Gal, M. J. Gales, A. Noskov, A. Ploskonosov, L. Prokhorenkova, I. Provilkov, V. Raina, et al., “Shifts: A dataset of real distributional shift across multiple large-scale tasks,” arXiv preprint arXiv:2107.07455 , 2021

    work page arXiv 2021

  30. [30]

    CatBoost: gradient boosting with categorical features support

    A. V. Dorogush, V. Ershov, and A. Gulin, “Catboost: Gradient boosting with categorical features support,” arXiv preprint arXiv:1810.11363 , 2018

    work page Pith review arXiv 2018

  31. [31]

    Adaptive Conformal Inference Under Distribution Shift , year =

    I. Gibbs and E. Cand` es, “Adaptive conformal inference under distribution shift,” arXiv:2106.00170, 2021

    work page arXiv 2021

  32. [32]

    Adaptive conformal predictions for time series,

    M. Zaffran, O. F´ eron, Y. Goude, J. Josse, and A. Dieuleveut, “Adaptive conformal predictions for time series,” in International Conference on Machine Learning , PMLR, 2022, pp. 25 834–25 866

    work page 2022

  33. [33]

    Conformal inference for online prediction with arbitrary distribution shifts, 2023

    I. Gibbs and E. Cand` es, “Conformal inference for online prediction with arbitrary distribution shifts,” arXiv preprint arXiv:2208.08401 , 2022

    work page arXiv 2022

  34. [34]

    Conformal prediction interval for dynamic time-series,

    C. Xu and Y. Xie, “Conformal prediction interval for dynamic time-series,” in International Confer- ence on Machine Learning , PMLR, 2021, pp. 11 559–11 569. 33

    work page 2021

  35. [35]

    Hanu and Unitary team, Detoxify, Github

    L. Hanu and Unitary team, Detoxify, Github. https://github.com/unitaryai/detoxify, 2020

    work page 2020

  36. [36]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional trans- formers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018

    work page Pith review arXiv 2018

  37. [37]

    Wilds: A benchmark of in-the-wild distribution shifts,

    P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, et al., “Wilds: A benchmark of in-the-wild distribution shifts,” in International Conference on Machine Learning, PMLR, 2021, pp. 5637–5664

    work page 2021

  38. [38]

    A tutorial on conformal prediction,

    G. Shafer and V. Vovk, “A tutorial on conformal prediction,” Journal of Machine Learning Research, vol. 9, no. Mar, pp. 371–421, 2008

    work page 2008

  39. [39]

    Computing full conformal prediction set with approximate homotopy,

    E. Ndiaye and I. Takeuchi, “Computing full conformal prediction set with approximate homotopy,” in Advances in Neural Information Processing Systems , 2019

    work page 2019

  40. [40]

    Root-finding approaches for computing conformal prediction set,

    E. Ndiaye and I. Takeuchi, “Root-finding approaches for computing conformal prediction set,” Ma- chine Learning, 2022

    work page 2022

  41. [41]

    Cross-conformal predictors,

    V. Vovk, “Cross-conformal predictors,” Annals of Mathematics and Artificial Intelligence , vol. 74, no. 1-2, pp. 9–28, 2015

    work page 2015

  42. [42]

    Predictive inference with the jack- knife+,

    R. F. Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani, “Predictive inference with the jack- knife+,” The Annals of Statistics , vol. 49, no. 1, pp. 486–507, 2021

    work page 2021

  43. [43]

    Exact and asymptotically robust permutation tests,

    E. Chung and J. P. Romano, “Exact and asymptotically robust permutation tests,” The Annals of Statistics, vol. 41, no. 2, pp. 484–507, 2013

    work page 2013

  44. [44]

    On a test of whether one of two random variables is stochastically larger than the other,

    H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,” The Annals of Mathematical Statistics , pp. 50–60, 1947

    work page 1947

  45. [45]

    The power of rank tests,

    E. L. Lehmann, “The power of rank tests,” The Annals of Mathematical Statistics , pp. 23–43, 1953

    work page 1953

  46. [46]

    Sidak, P

    Z. Sidak, P. K. Sen, and J. Hajek, Theory of rank tests . Elsevier, 1999

    work page 1999

  47. [47]

    Efron and R

    B. Efron and R. J. Tibshirani, An introduction to the bootstrap. CRC press, 1994

    work page 1994

  48. [48]

    Distribution-free cumulative sum control charts using bootstrap-based control limits,

    S. Chatterjee and P. Qiu, “Distribution-free cumulative sum control charts using bootstrap-based control limits,” The Annals of Applied Statistics , vol. 3, no. 1, pp. 349–369, 2009

    work page 2009

  49. [49]

    G. T. Fechner, Kollektivmasslehre. Engelmann, 1897

    work page

  50. [50]

    Grundlagen der wahrscheinlichkeitsrechnung,

    R. von Mises, “Grundlagen der wahrscheinlichkeitsrechnung,” Mathematische Zeitschrift, vol. 5, no. 1, pp. 52–99, 1919

    work page 1919

  51. [51]

    Die widerspruchfreiheit des kollectivbegriffes der wahrscheinlichkeitsrechnung,

    A. Wald, “Die widerspruchfreiheit des kollectivbegriffes der wahrscheinlichkeitsrechnung,” Ergebnisse Eines Mathematischen Kolloquiums , vol. 8, no. 38-72, p. 37, 1937

    work page 1937

  52. [52]

    On the concept of a random sequence,

    A. Church, “On the concept of a random sequence,” Bulletin of the American Mathematical Society , vol. 46, no. 2, pp. 130–135, 1940

    work page 1940

  53. [53]

    Etude critique de la notion de collectif,

    J. Ville, “Etude critique de la notion de collectif,” Bull. Amer. Math. Soc , vol. 45, no. 11, p. 824, 1939

    work page 1939

  54. [54]

    The sources of Kolmogorov’s Grundbegriffe,

    G. Shafer and V. Vovk, “The sources of Kolmogorov’s Grundbegriffe,” Statistical Science, vol. 21, no. 1, pp. 70–98, 2006

    work page 2006

  55. [55]

    Kolmogorov’s complexity conception of probability,

    V. Vovk, “Kolmogorov’s complexity conception of probability,” Synthese Library, pp. 51–70, 2001

    work page 2001

  56. [56]

    Kolmogorov on the role of randomness in probability theory,

    C. P. Porter, “Kolmogorov on the role of randomness in probability theory,” Mathematical Structures in Computer Science , vol. 24, no. 3, 2014

    work page 2014

  57. [57]

    Three approaches to the quantitative definition of information,

    A. N. Kolmogorov, “Three approaches to the quantitative definition of information,” Problems of Information Transmission, vol. 1, no. 1, pp. 1–7, 1965

    work page 1965

  58. [58]

    Logical basis for information theory and probability theory,

    A. Kolmogorov, “Logical basis for information theory and probability theory,” IEEE Transactions on Information Theory , vol. 14, no. 5, pp. 662–664, 1968

    work page 1968

  59. [59]

    Combinatorial foundations of information theory and the calculus of probabili- ties,

    A. N. Kolmogorov, “Combinatorial foundations of information theory and the calculus of probabili- ties,” Russian Mathematical Surveys , vol. 38, no. 4, pp. 29–40, 1983. 34

    work page 1983

  60. [60]

    On the concept of the Bernoulli property,

    V. G. Vovk, “On the concept of the Bernoulli property,” Russian Mathematical Surveys, vol. 41, no. 1, p. 247, 1986

    work page 1986

  61. [61]

    Testing randomness online,

    V. Vovk, “Testing randomness online,” Statistical Science, vol. 36, no. 4, pp. 595–611, 2021

    work page 2021

  62. [62]

    Sophistication as randomness deficiency,

    F. Mota, S. Aaronson, L. Antunes, and A. Souto, “Sophistication as randomness deficiency,” in International Workshop on Descriptional Complexity of Formal Systems , Springer, 2013, pp. 172– 181

    work page 2013

  63. [63]

    Determination of sample sizes for setting tolerance limits,

    S. S. Wilks, “Determination of sample sizes for setting tolerance limits,” Annals of Mathematical Statistics, vol. 12, no. 1, pp. 91–96, 1941

    work page 1941

  64. [64]

    Statistical prediction with special reference to the problem of tolerance limits,

    ——, “Statistical prediction with special reference to the problem of tolerance limits,” Annals of Mathematical Statistics, vol. 13, no. 4, pp. 400–409, 1942

    work page 1942

  65. [65]

    An extension of Wilks’ method for setting tolerance limits,

    A. Wald, “An extension of Wilks’ method for setting tolerance limits,” Annals of Mathematical Statis- tics, vol. 14, no. 1, pp. 45–55, 1943

    work page 1943

  66. [66]

    Non-parametric estimation II. Statistically equivalent blocks and tolerance regions–the continuous case,

    J. W. Tukey, “Non-parametric estimation II. Statistically equivalent blocks and tolerance regions–the continuous case,” Annals of Mathematical Statistics , vol. 18, no. 4, pp. 529–539, 1947

    work page 1947

  67. [67]

    Finite exchangeable sequences,

    P. Diaconis and D. Freedman, “Finite exchangeable sequences,” The Annals of Probability , pp. 745– 764, 1980

    work page 1980

  68. [68]

    Exchangeability and related topics,

    D. J. Aldous, “Exchangeability and related topics,” in ´Ecole d’ ´Et´ e de Probabilit´ es de Saint-Flour XIII—1983, 1985, pp. 1–198

    work page 1983

  69. [69]

    Funzione caratteristica di un fenomeno aleatorio,

    B. De Finetti, “Funzione caratteristica di un fenomeno aleatorio,” in Atti del Congresso Internazionale dei Matematici: Bologna del 3 al 10 de Settembre di 1928 , 1929, pp. 179–190

    work page 1928

  70. [70]

    Bernard Friedman’s urn,

    D. A. Freedman, “Bernard Friedman’s urn,” The Annals of Mathematical Statistics , pp. 956–970, 1965

    work page 1965

  71. [71]

    Symmetric measures on Cartesian products,

    E. Hewitt and L. J. Savage, “Symmetric measures on Cartesian products,” Transactions of the Amer- ican Mathematical Society, vol. 80, no. 2, pp. 470–501, 1955

    work page 1955

  72. [72]

    Uses of exchangeability,

    J. F. Kingman, “Uses of exchangeability,” The Annals of Probability, vol. 6, no. 2, pp. 183–197, 1978

    work page 1978

  73. [73]

    Dobriban, Topics in Modern Statistical Learning (STAT 991, UPenn, 2022 Spring) , Dec

    E. Dobriban, Topics in Modern Statistical Learning (STAT 991, UPenn, 2022 Spring) , Dec. 2022

    work page 2022

  74. [74]

    Learning by transduction,

    A. Gammerman, V. Vovk, and V. Vapnik, “Learning by transduction,” Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence , vol. 14, pp. 148–155, 1998

    work page 1998

  75. [75]

    Transduction with confidence and credibility,

    C. Saunders, A. Gammerman, and V. Vovk, “Transduction with confidence and credibility,” 1999

    work page 1999

  76. [76]

    On-line confidence machines are well-calibrated,

    V. Vovk, “On-line confidence machines are well-calibrated,” in The 43rd Annual IEEE Symposium on Foundations of Computer Science , IEEE, 2002, pp. 187–196

    work page 2002

  77. [77]

    Self-calibrating probability forecasting.,

    V. Vovk, G. Shafer, and I. Nouretdinov, “Self-calibrating probability forecasting.,” in Neural Infor- mation Processing Systems, 2003, pp. 1133–1140

    work page 2003

  78. [78]

    Venn-abers predictors.arXiv preprint arXiv:1211.0025,

    V. Vovk and I. Petej, “Venn-Abers predictors,” arXiv:1211.0025, 2012

    work page arXiv 2012

  79. [79]

    Nonparametric predictive distributions based on conformal prediction,

    V. Vovk, J. Shen, V. Manokhin, and M.-g. Xie, “Nonparametric predictive distributions based on conformal prediction,” Machine Learning, pp. 1–30, 2017

    work page 2017

  80. [80]

    Efficient nonparametric conformal prediction regions,

    J. Lei, J. Robins, and L. Wasserman, “Efficient nonparametric conformal prediction regions,” arXiv:1111.1418, 2011

    work page arXiv 2011

Showing first 80 references.