REVIEW 1 major objections 4 minor 2 cited by
Conformal prediction with localization
T0 review · 1 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Localized conformal prediction keeps its coverage guarantee
desk verdict Genuinely new localized conformal method with a sound finite-sample proof; the main caveat is that the guarantee only covers feature-based localizers, and the abstract overstates that scope. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the localizer $H(x_1,x_2,\mathcal{X})\in[0,1]$ with $H(x,x,\mathcal{X})=1$, which determines normalized weights $p^H_{i,j}=H_{ij}/\sum_k H_{ik}$ for the weighted empirical distributions $\hat{F}_i$. The proof's engine is Lemma B.2, a permutation argument that conditions on the unordered data set and expresses the conditional coverage probability as a weighted average of indicators $\mathbb{1}\{v_i\le v_i^*\}$; condition (G2) then certifies that this average stays above $\alpha$ for the two hardest possible test scores, $V_{n+1}=0$ and $V_{n+1}=\bar{v}^*$. Algorithm 1 searches over candidate $\tilde{\alpha}$ values to satisfy this condition, then inverts the score inequality to form the interval.
What would settle it
Search is over all small configurations: pick $n=3$ or $4$, discrete features with all permutations, a binary score $V\in\{0,1\}$, and a localizer matrix $H$ satisfying $H_{ii}=1$ and depending only on the feature set, then check whether every $\tilde{\alpha}$ satisfying condition (G2) also has $P(V_{n+1}\le Q(\tilde{\alpha};\hat{F}))\ge\alpha$ exactly; the first counterexample would refute Theorem 3.2. A cheaper check is to reproduce Example 3.7's asymptotic setting with a box localizer and $\tilde{\alpha}=\alpha$, where the reported under-coverage $\alpha(1-\alpha)/(2-\alpha)$ verifies that naively setting $\tilde{\alpha}=\alpha$ is not sufficient.
Extended reading notes
Core claim
The discovery is that the exchangeability broken by a localizer can be restored by symmetrizing the quantile choice. For any fixed score function $V$ and any localizer $H$ satisfying $H(x,x)=1$ and depending only on the unordered feature set, define a weighted empirical distribution $\hat{F}$ with weights proportional to $H(X_{n+1}, X_i)$. If $\tilde{\alpha}$ is chosen so that $\bar{v}^*=Q(\tilde{\alpha};\hat{F})=\infty$ or condition (G2) holds, then Theorem 3.2 gives $P\{V_{n+1}\le Q(\tilde{\alpha};\hat{F})\}\ge\alpha$, so Algorithm 1's interval $\hat{C}(X_{n+1})=\{y:V(X_{n+1},y)\le Q(\tilde{\alpha};\hat{F})\}$ has marginal coverage at least $\alpha$ for i.i.d. data, for every $P$. A randomized version of the level choice makes the coverage exactly equal to $\alpha$. With the constant localizer $H_{i,j}=1$, the construction reduces to standard conformal prediction.
Load-bearing premise
Everything rests on the requirement that the localizer $H$ depend only on the feature values $\{X_1,\dots,X_{n+1}\}$ as an unordered set (and similarly that any data-dependent score depend only on the unordered data), never on the response values or the order of the samples; if a user builds $H$ from $Y_i$ or from the data ordering, the exchangeability argument in Lemma B.2 breaks and the finite-sample coverage guarantee no longer follows.
Editorial extensions
If this is right
- For any fixed score function and any localizer satisfying the feature-set condition, Algorithm 1 yields a distribution-free interval with coverage at least $\alpha$ for every $P$.
- With the identity localizer $H_{i,j}=1$ and $\tilde{\alpha}=\alpha$, the construction reproduces ordinary conformal prediction exactly (Proposition 3.4).
- A randomized choice of $\tilde{\alpha}$ between two grid levels makes the marginal coverage exactly $\alpha$, removing the slack from the non-randomized rule (Corollary 3.5, Theorem 5.5).
- Under covariate shift with known density ratio $w$, the weighted version (Theorem A.7 and Corollary A.8) retains the same finite-sample guarantee, and localization can prevent a few extreme-weight training samples from dominating the interval.
- For localizers with shrinking bandwidth and mild smoothness conditions, the interval achieves asymptotic conditional coverage at each feature value (Theorem 5.1).
Reading between the lines
- Inference: The condition (G2) works as a coverage certificate for any score function, so localized conformal prediction can be layered on top of modern scores such as conformalized quantile regression, not just fixed scores, likely narrowing intervals further where score heterogeneity remains.
- Inference: Because the localizer may depend on the feature set, a user could choose $H$ adaptively (e.g., data-adaptive bandwidth from the paper's Supplement C) to balance locality and interval width; testing whether such adaptive localizers preserve coverage under the same exchangeability condition would be a natural follow-up.
- Inference: In outlier-detection applications mentioned only as future work, a distance-based localizer could shrink prediction bands around dense regions and flag points whose response is unlikely given their local neighborhood; this could be tested by measuring detection rates on contaminated regression data.
- Inference: The symmetrization idea (considering localizers centered at each training point) suggests a general recipe for any weighted conformal scheme: any weighting scheme that is symmetric in the unordered data set can be turned into a coverage-guaranteed method by tuning the quantile level with the analogue of (G2).
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes localized conformal prediction (LCP), a modification of conformal prediction in which training samples are reweighted by a localizer H(x_i, x_{n+1}) that emphasizes points near the test feature. For a fixed score function V and a localizer depending only on the feature set, the paper proves finite-sample marginal coverage by replacing the nominal level α with a data-dependent level \tilde α chosen to satisfy condition (G2) (or with \tilde α leading to \bar v^* = ∞). The authors provide a practical Algorithm 1, discuss local and asymptotic conditional coverage, extend the idea to data-dependent scores and to covariate shift, and report simulations showing comparable coverage and more locally adaptive intervals than classical conformal prediction.
Significance. The localization idea is a natural and useful extension of conformal inference, and the paper is the first to provide a finite-sample coverage guarantee under local reweighting. The permutation argument in Lemma B.2 is clean for the symmetric case, and the formulation in terms of \tilde α is interesting. If the proof of Theorem A.7 can be completed, the result would be a valuable contribution to distribution-free predictive inference. The simulations support the qualitative claim that LCP adapts to heteroscedasticity, and the explicit treatment of covariate shift in the appendix extends the reach of the method.
major comments (1)
- [Supplement B.3, proof of Theorem A.7, Eq. (7)–(8)] The step from Eq. (7) to Eq. (8) is not valid. Since min_l \tilde α_l ≤ \tilde α_{σ_{n+1}}, the quantiles Q(min_l \tilde α_l; \hat F_{σ_i}) are no larger than Q(\tilde α_{σ_{n+1}}; \hat F_{σ_i}); therefore the left-hand side of Eq. (8) is bounded above, not below, by the left-hand side of Eq. (7). The assertion that Eq. (8) holds for the identity permutation, which is then used to obtain E[∑_i w_i 1_{V_i ≤ u^*_i}|T] ≥ α, is unsupported. This gap is load-bearing because Theorem A.7 underlies Theorem 3.2 and Corollary 3.3, the main coverage guarantees. Replacing 'min' by 'max' does not fix the issue, since the subsequent step requires Q(\tilde α; ·) ≤ Q(min_l \tilde α_l; ·). The authors need to either repair the symmetrization argument or explicitly restrict the theorem to a permutation-invariant choice of \tilde α.
minor comments (4)
- [Abstract and Section 3] The abstract calls the guarantee 'assumption-free,' but the proof requires the localizer H to be a fixed function of the feature set only, not of the responses or of the data ordering. This is an explicit scope restriction in Section 3, but it should be stated prominently in the abstract and introduction so that readers do not apply the method to response-dependent localizers, for which Lemma B.2 would fail.
- [Tables 1 and 2] The simulation tables report coverage for 1000 repetitions, but no Monte Carlo standard errors are given. With 1000 repetitions the standard error is approximately 0.006–0.01, so differences of 0.01 between methods are not interpretable without error bars.
- [Supplement B.3, Lemma B.3 proof] In the second bullet of the proof of Lemma B.3, the text writes 'we always have Q(α; \hat F) ≥ Q(α, ∑_{j=1}^{n+1} p^H_{n+1,j} δ_{V_j})'; the level in both quantiles should be \tilde α, not α, and the inequality direction should be checked carefully.
- [Section 5.2] The paper states that the general recipe for data-dependent score functions is too computationally expensive to carry out in practice. This limitation is important and should be noted in the abstract or introduction, not only in Section 5.2, so that readers do not over-interpret the data-dependent extension as a practical algorithm.
Circularity Check
No circularity: the coverage theorem is derived from exchangeability, not fitted to the target.
full rationale
The central coverage claim (Theorem 3.2 and Corollary 3.3) is derived from permutation/exchangeability arguments rather than from fitting or from a self-citation chain. The localizer is explicitly restricted to depend only on the feature set: the paper defines H(x1,x2,X) in [0,1] for x1,x2 in R^p as a function that may depend on the set X = {X1,...,X_{n+1}} with H(x,x,X)=1, so conditional on the unordered set T the normalized weights p^H_{i,j} are fixed functions of T. Lemma B.2 then proves the key identity v_i^*(sigma) = v^*_{sigma_i}, which converts the permutation probability into a weighted average of indicator events; condition (G2) is a sufficient condition proved from this same exchangeability argument, not an assumption that is definitionally equivalent to the desired coverage statement. In Algorithm 1, tilde_alpha is chosen from the scores and weights without using the test response y, and the bandwidth h is tuned on an independent data set D0, so no fitted parameter is renamed as a prediction. The paper contains self-citations, notably [19] by the same author, but these are used only as background or application suggestions and are not load-bearing for the proof. The only notable caveat is that the finite-sample guarantee is stated for localizers depending on features alone; a response-dependent localizer would break the exchangeability used in Lemma B.2, but this is an explicit scope condition in the paper rather than a circular use of the conclusion.
Assumptions & free parameters
free parameters (1)
- Localizer bandwidth h (distance or nearest-neighbor) =
auto h_hat via Appendix C; simulation values h1=0.1, h2=1 (distance), 40 and 500 (nearest-neighbor)
assumptions (4)
- domain assumption Z_1,...,Z_{n+1} are i.i.d. from P, or exchangeable, or covariate shift with known density ratio w in Appendix A
- domain assumption Score function V is fixed, or data-dependent through a function of the unordered set Z that preserves exchangeability
- standard math Standard quantile definition Q(alpha; F) = inf{t: P(T <= t) >= alpha} and its basic properties
- domain assumption For Theorem 5.1(b): X has density bounded away from 0 and infinity on [0,1]^d, the conditional density of V given X is Lipschitz, and a Gaussian kernel is used
Cite this review
Pith. "Pith review of Conformal prediction with localization." pith.science (2026). https://pith.science/paper/WKZ4JWIK
@misc{pith2026190808558,
author = {Pith},
title = {Pith review of: Conformal prediction with localization},
year = {2026},
howpublished = {\url{https://pith.science/paper/WKZ4JWIK}},
note = {Machine review of arXiv:1908.08558}
}
read the original abstract
We propose a new method called localized conformal prediction, where we can perform conformal inference using only a local region around a new test sample to construct its confidence interval. Localized conformal inference is a natural extension to conformal inference. It generalizes the method of conformal prediction to the case where we can break the data exchangeability, so as to give the test sample a special role. To our knowledge, this is the first work that introduces such a localization to the framework of conformal prediction. We prove that our proposal can also have assumption-free and finite sample coverage guarantees, and we compare the behaviors of localized conformal prediction and conformal prediction in simulations.
Forward citations
Cited by 2 Pith papers
-
Beyond Marginal Validity: Finite-Sample Guarantees for Localized Conformal Prediction
For randomly localized conformal prediction, high-probability bounds, uniform over a realized neighborhood, control conditional coverage error and oracle-relative length at rate h^beta + sqrt(log(1/delta)/(n h^d)) plu...
-
Robust Bayesian Optimization via Localized Online Conformal Prediction
LOCBO calibrates the GP likelihood with localized online conformal prediction and then denoises it, claiming utility lower bounds that are not actually established for the expected-improvement setting.
Reference graph
Works this paper leans on
-
[1]
Springer Science & Business Media, 2005
Vladimir V ovk, Alex Gammerman, and Glenn Shafer.Algorithmic learning in a random world. Springer Science & Business Media, 2005
work page 2005
-
[2]
A tutorial on conformal prediction
Glenn Shafer and Vladimir V ovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(Mar):371–421, 2008
work page 2008
-
[3]
On-line predictive linear regression
Vladimir V ovk, Ilia Nouretdinov, Alex Gammerman, et al. On-line predictive linear regression. The Annals of Statistics, 37(3):1566–1590, 2009
work page 2009
-
[4]
Distribution-free prediction bands for non-parametric regression
Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76(1):71–96, 2014
work page 2014
-
[5]
Distribution-free predictive inference for regression
Jing Lei, Max G?Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Associ- ation, 113(523):1094–1111, 2018
work page 2018
-
[6]
Conditional validity of inductive conformal predictors
Vladimir V ovk. Conditional validity of inductive conformal predictors. InAsian conference on machine learning, pages 475–490, 2012
work page 2012
-
[7]
The limits of distribution-free conditional predictive inference
Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-free conditional predictive inference. arXiv preprint arXiv:1903.04684, 2019
arXiv 1903
-
[8]
Conformal prediction under covariate shift
Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction under covariate shift. arXiv preprint arXiv:1904.06019, 2019
arXiv 1904
Show all 21 references
-
[9]
Conformalized quantile regression
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. In Advances in Neural Information Processing Systems, pages 3538–3548, 2019
2019
-
[10]
Adaptive, distribution-free prediction intervals for deep neural networks
Danijel Kivaranovic, Kory D Johnson, and Hannes Leeb. Adaptive, distribution-free prediction intervals for deep neural networks. arXiv preprint arXiv:1905.10634, 2019
1905 arXiv
-
[11]
A comparison of some conformal quantile regression methods
Matteo Sesia and Emmanuel J Candès. A comparison of some conformal quantile regression methods. Stat, 9(1):e261, 2020
2020
-
[12]
Distribution-free conditional predictive bands using density estimators
Rafael Izbicki, Gilson T Shimizu, and Rafael B Stern. Distribution-free conditional predictive bands using density estimators. arXiv preprint arXiv:1910.05575, 2019
1910 arXiv
-
[13]
Distributional conformal prediction
Victor Chernozhukov, Kaspar Wüthrich, and Yinchu Zhu. Distributional conformal prediction. arXiv preprint arXiv:1909.07889, 2019
1909 arXiv
-
[14]
Improving predictive inference under covariate shift by weighting the log-likelihood function
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227–244, 2000
2000
-
[15]
Input-dependent estimation of generalization error under covariate shift
Masashi Sugiyama and Klaus-Robert Müller. Input-dependent estimation of generalization error under covariate shift. Statistics & Decisions, 23(4/2005):249–279, 2005
2005
-
[16]
Covariate shift adaptation by importance weighted cross validation
Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert MÞller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research , 8(May): 985–1005, 2007
2007
-
[17]
Dataset shift in machine learning
Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. The MIT Press, 2009
2009
-
[18]
Cautious deep learning
Yotam Hechtlinger, Barnabás Póczos, and Larry Wasserman. Cautious deep learning. arXiv preprint arXiv:1805.09460, 2018
2018 arXiv
-
[19]
Prediction and outlier detection in classification problems
Leying Guan and Rob Tibshirani. Prediction and outlier detection in classification problems. arXiv preprint arXiv:1905.04396, 2019
1905 arXiv
-
[20]
A survey of outlier detection methodologies
Victoria Hodge and Jim Austin. A survey of outlier detection methodologies. Artificial intelligence review, 22(2):85–126, 2004
2004
-
[21]
Anomaly detection: A survey
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3):15, 2009. 19
2009
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.