REVIEW 4 major objections 3 minor 2 cited by
LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systems
T0 review · 4 major / 3 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read LEC claims that selective prediction and model routing can be reduced to a single linear expectation constraint, giving finite-sample control of the error rate among accepted answers using only held-out calibration data.
desk verdict The algorithm is simple and the experiments are thorough, but the central finite-sample FDR guarantee is not proved: the proof uses exchangeability at a data-dependent threshold, which fails, and the paper's headline claim should not be accepted as stated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The linear expectation constraint Z − αS, where S is the selection indicator and Z is the selected-and-erroneous indicator; for a fixed threshold λ, E[Z − αS] ≤ 0 is equivalent to FDR ≤ α whenever E[S] > 0. The paper's finite-sample surrogate is the '+1' condition Σ_accepted(err_j − α) ≤ −1 on calibration data, which defines the feasible threshold set; choosing the largest feasible λ maximizes retention while preserving the guarantee. For routing, the same decomposition is applied to system-level (S, Z) built from a deterministic policy that selects at most one model, so joint calibration inherits the single-model proof.
What would settle it
Generate an exchangeable dataset where the uncertainty score u is independent of the error indicator err, with error rate p > α (e.g., p = 0.4, α = 0.1), using n calibration points, and average test FDR over many recalibrations. If the empirical marginal FDR among accepted predictions exceeds α beyond sampling noise, the claimed finite-sample bound fails. A more direct check is to compare the Monte Carlo estimate of E[Z_{n+1} − αS_{n+1}] at the data-dependent threshold with the calibration-sum expression used in Eqs. (13)–(16).
Extended reading notes
Core claim
The central claim is that selection-conditioned risk control reduces to controlling E[Z − αS] ≤ 0, where S says a prediction is accepted and Z says it is accepted and wrong. Because this is a linear constraint on indicators, the paper proves a finite-sample sufficient condition from n exchangeable calibration points: the sorted accepted calibration errors must satisfy Σ(err_(j) − α) ≤ −1. The coverage-maximizing threshold is the largest λ for which this holds; if none holds, the system abstains. Theorem 3.1 states this yields Pr(err = 1 | accepted) ≤ α for a new test sample, and Theorem 3.2 extends the same guarantee to a two-model router with jointly calibrated thresholds. The argument is d
Load-bearing premise
The proof's load-bearing step is that the calibration-set sum evaluated at the data-dependent threshold can stand in for the test expectation—i.e., picking the threshold does not break exchangeability between calibration and test indicators; that substitution fails in exchangeable settings where uncertainty scores carry no information about correctness (Appendix A.1, Eqs. 13–16).
Editorial extensions
If this is right
- Any scalar uncertainty score, white-box or black-box, can be converted into a certified accept/reject rule with error ceiling α, as long as calibration and test data are exchangeable.
- The guarantee is marginal over calibration randomness; a practitioner conditioning on a specific calibration set may see realized FDR above α, as is typical for exchangeability-based guarantees.
- For a router, calibrating each model's threshold independently does not preserve system-level FDR; joint calibration over the system indicators is necessary, as the paper shows empirically and through the theorem's structure.
- At feasible risk levels, LEC accepts more correct samples than confidence-bound-based calibration because it constrains the expectation rather than a worst-case tail event.
- The same '+1' condition extends in principle to arbitrary-depth routing policies, so multi-model cascades can keep one system-level error ceiling.
Reading between the lines
- Because the guarantee rests on exchangeability, the method inherits the standard fragility of exchangeability-based guarantees under distribution shift: the error ceiling is not certified if test inputs come from a different distribution than the calibration set.
- The case where uncertainty scores are independent of correctness is a stress test for the theorem: the substitution of the calibration sum for the test expectation is most strained there, and a future variant might need a rank-based or data-splitting correction to remain valid in that exchangeable regime.
- The linear form of the constraint suggests extensions to other ratio-style risks—such as cost-weighted error budgets or domain-specific losses—by replacing Z with a bounded weighted loss indicator; the paper only sketches this direction.
- The routing results imply a practical cost-reliability trade-off: a cheaper model can handle most inputs and delegate only uncertain cases to an expensive model while sharing one risk budget; measuring the trade-off in cost per correct answer would be a natural deployment test.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes LEC, a calibration scheme for selective prediction and two-model routing. For a user-specified risk level α, LEC selects the largest threshold λhat such that the calibration sum Σ_{j=1}^{k}(err_(j)−α) ≤ −1 (Eq. 5), and claims a finite-sample, marginal FDR guarantee Pr(err_{n+1}=1 | u_{n+1}≤λhat) ≤ α under exchangeability (Theorem 3.1). The same construction is extended to two-model routing by jointly calibrating two thresholds with a system-level linear constraint, with a claimed FDR guarantee (Theorem 3.2). The paper reports extensive experiments on QA and VQA benchmarks, showing that LEC controls test-time FDR and improves retention over UCB-based baselines.
Significance. If the central theorem were correct, the paper would offer a simple, model-agnostic method for FDR control in selective prediction and routing, with a clean linear-expectation formulation and strong empirical support. The experimental study is broad and carefully executed, comparing multiple baselines across several LLMs/LVLMs, uncertainty estimators, alignment criteria, and split ratios. However, the paper's central contribution is a finite-sample guarantee, and that guarantee is not established. The proof in Appendix A.1 relies on an exchangeability identity that does not hold for data-dependent thresholds, and the claimed bound is false in exchangeable settings where the uncertainty score is uninformative about correctness. Because the main theoretical claim is unsupported and contradicted by a valid counterexample, the paper cannot be accepted in its current form.
major comments (4)
- [Appendix A.1, Eq. (13)] The proof asserts E[Z_{n+1}(λhat)−αS_{n+1}(λhat)] = (1/(n+1)) Σ_{i=1}^{n+1}[Z_i(λhat)−αS_i(λhat)]. This identity is invalid. For a fixed λ, exchangeability gives equality of the test expectation to the common per-sample expectation, not to the random empirical average. Once λhat is a function of the calibration error labels, the calibration terms and the test term are not exchangeable. The threshold is specifically chosen to make the calibration sum ≤ −1, so using that same sum as evidence for the test expectation is circular. This is not a technical gap but the core of the proof.
- [Theorem 3.1 / Eq. (5)] The claimed finite-sample guarantee is false. Consider exchangeable i.i.d. pairs (u_i, err_i) with u_i independent of err_i and P(err_i=1)=p>α. With positive probability (e.g., (1−p)^{ceil(1/α)}) the first ceil(1/α) calibration points in ascending u-order are all correct, making Eq. (5) feasible. The resulting λhat accepts a test point whose conditional error probability is p, not ≤α. Thus Theorem 3.1 fails in an exchangeable setting, and the distribution-free claim in the abstract and Section 3.2 is unsupported.
- [Theorem 3.2 / Appendix A.2, Eq. (22)] The routing extension inherits the same flaw. The footnote in Appendix A.2 asserts that the induced system-level indicators (S_i,Z_i) are exchangeable, but these indicators are evaluated at thresholds (λhat_a, λhat_b) that are calibrated on the same data. The system-level selection is a function of the calibration error labels, so the exchangeability of the underlying outputs does not imply exchangeability of the selected indicators. Consequently, Theorem 3.2's finite-sample FDR guarantee is also not established.
- [Empirical evaluation, Section 4] The experimental results do not address the failure mode identified above. The benchmarks use uncertainty scores that are informative about correctness, so the calibration condition may hold for thresholds with genuinely low error rates. The paper's Figure 11 shows errors are distributed across uncertainty intervals, but the method is never tested in the uninformative-uncertainty regime where the theorem provably breaks down. Empirical validity on informative uncertainty does not rescue a distribution-free claim.
minor comments (3)
- [Section 3.2, Eq. (7)] The definition of λhat as a supremum is problematic when the feasible set Λ_α is empty, since the text says the system abstains on all samples. The theorem statements should explicitly condition on Λ_α being nonempty or define the conditional guarantee for the non-abstention event.
- [Figures 2 and 3 and Table 3] The caption 'LEC provides tighter and lower FDR control' is ambiguous: a lower realized FDR is not necessarily 'tighter' if the target is α. The paper means LEC is closer to α without exceeding it; this should be stated clearly. Table 3 also contains unexplained entries (e.g., '-' and '(5)') that need cleanup.
- [References] There are duplicate reference entries (Campos et al. 2024a/b and Reimers & Gurevych 2019a/b are the same works) and some sources are cited with inconsistent labels. The manuscript would benefit from a careful reference pass.
Circularity Check
The central finite-sample FDR guarantee is obtained by treating the calibration inequality that defines the threshold as the test expectation, making the 'prediction' equivalent to the calibration constraint by construction.
-
fitted input called prediction
[Appendix A.1, Eqs. (13)-(18) (proof of Theorem 3.1); threshold defined in Eq. (7)]
"Since n calibration data points and the given test sample are exchangeable ... we have E[Z^{(a)}_{n+1}(\hat{\lambda}^{(a)})−αS^{(a)}_{n+1}(\hat{\lambda}^{(a)})] = 1/(n+1) Σ_{i=1}^{n+1}(Z_i(\hat{\lambda}^{(a)})−αS_i(\hat{\lambda}^{(a)})). (13) ... By the definition of the calibrated threshold \hat{\lambda}^{(a)}, the empirical constraint Σ_{j=1}^{k^{(a)}(\hat{\lambda}^{(a)})}(err^{(a)}_{(j)}−α) ≤ −1 holds ... Plugging this into Eq. (16) yields E[...] ≤ 0. (18)"
The threshold \hat{\lambda} is not an independent quantity: Eq. (7) defines it as any threshold whose calibration sum Σ(Z_i−αS_i) is ≤ −1. The proof then uses that same defining inequality as the sole evidence that the test expectation E[Z_{n+1}−αS_{n+1}] is ≤ 0. The exchangeability step in Eq. (13) is the whole argument, but it is false at a data-dependent threshold: conditioning on \hat{\lambda}, a function of the first n samples, breaks the symmetric role of the (n+1)-th term, so the calibration sum cannot be swapped for the test expectation. Thus the claimed guarantee reduces to the constraint that constructed the threshold; no rank/uniformity argument supplies independent content. In an i.i.d. exchangeable case with u independent of err and P(err)=p>α, the calibration condition can be
-
fitted input called prediction
[Appendix A.2, Eqs. (22)-(24) (proof of Theorem 3.2); thresholds defined in Eqs. (10)-(11)]
"By the exchangeability among calibration and test samples ... E[Z_{n+1}(\hat{\lambda}^{(a)},\hat{\lambda}^{(b)})−αS_{n+1}(\hat{\lambda}^{(a)},\hat{\lambda}^{(b)})] = 1/(n+1) Σ_{i=1}^{n+1}(Z_i(\hat{\lambda}^{(a)},\hat{\lambda}^{(b)})−αS_i(\hat{\lambda}^{(a)},\hat{\lambda}^{(b)})). (22) ... By the definition of the feasible region Λ^{(a,b)}_α in Eq. (10), the calibrated pair satisfies Σ_{i=1}^n(Z_i−αS_i) ≤ −1 ... Substituting this inequality into Eq. (22) yields E[...] ≤ 0. (24)"
The routing thresholds are defined by Eq. (11) as the coverage-maximizing pair inside the region Λ^{(a,b)}_α, which itself is defined by the same empirical inequality Σ_{i=1}^n(Z_i−αS_i) ≤ −1. The proof again converts this defining calibration inequality into the test-time FDR bound via the same invalid data-dependent exchangeability step. The system-level guarantee therefore inherits exactly the same circular structure: the property being proved is the property used to select the thresholds, with no independent argument that selection preserves exchangeability of the test indicator.
full rationale
There is no load-bearing self-citation circularity here: citations to Angelopoulos et al. and to the authors' own prior work are used for standard exchangeability notions, and the empirical benchmarks are run on genuine held-out splits. The circularity is in the proof of the central distribution-free claim. The paper's fundamental move is to define the calibrated threshold as the largest threshold satisfying a calibration inequality, then to 'prove' the test guarantee by invoking exchangeability to identify the expectation of the test term with that same calibration inequality. Because the threshold is a function of the calibration points, conditioning on it changes the joint distribution of the calibration indicators, so Eq. (13) and Eq. (22) are not consequences of exchangeability. The conclusion is thus not independently derived; it is the construction input restated as a population guarantee. This affects both Theorem 3.1 and Theorem 3.2, i.e., the entire claimed finite-sample FDR-control result. The empirical demonstrations are self-contained and may show good performance on the tested datasets, but they cannot supply the missing distribution-free derivation. Accordingly, the score reflects one central 'prediction' that reduces by construction to its calibration input, without accusing the authors of any intent to be circular.
Assumptions & free parameters
assumptions (3)
- domain assumption Calibration and test samples are exchangeable at the level of (u, err) pairs.
- domain assumption The routing policy is deterministic and selects at most one model per input.
- ad hoc to paper Threshold selection using the calibration error labels does not break the exchangeability argument.
Cite this review
Pith. "Pith review of LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systems." pith.science (2026). https://pith.science/paper/2PSSMFE3
@misc{pith2026251201556,
author = {Pith},
title = {Pith review of: LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/2PSSMFE3}},
note = {Machine review of arXiv:2512.01556}
}
read the original abstract
Foundation models often generate unreliable answers, while heuristic uncertainty estimators fail to fully distinguish correct from incorrect outputs, causing users to accept erroneous answers without any statistical guarantee. We address this problem through selection-conditioned risk control, aiming to ensure that an accepted prediction has an error probability no larger than a user-specified risk level. To this end, we propose LEC, a principled framework that reframes selective prediction as a decision problem governed by a linear expectation constraint over selection and error indicators. This formulation directly controls the ratio between the expected number of accepted errors and the expected number of accepted predictions, which corresponds to the marginal error probability conditioned on selection. Under exchangeability, we derive a finite-sample sufficient condition that relies only on a held-out calibration set, enabling the computation of a risk-constrained, retention-maximizing threshold. Furthermore, we extend LEC to two-model routing systems: if the primary model's uncertainty exceeds its calibrated threshold, the input is delegated to a subsequent model, while maintaining system-level selection-conditioned error control. Experiments on both closed-ended and open-ended question answering (QA) and vision question answering (VQA) demonstrate that LEC maintains the prescribed risk level in accepted predictions and substantially improves sample retention compared to baselines.
Figures
Figures from the paper (18 more)
Forward citations
Cited by 2 Pith papers
-
SCOPE: Selective Conformal Optimized Pairwise LLM Judging
A conformal calibration method (SCOPE) plus a bidirectional entropy score (BPE) lets LLM pairwise judges abstain selectively while keeping accepted-set error below a user-specified bound.
-
Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees
CIC selects the largest uncertainty threshold whose Hoeffding or Clopper–Pearson upper bound on acceptance-conditioned error stays ≤ α, guaranteeing finite-sample risk control under exchangeability.
Reference graph
Works this paper leans on
-
[1]
Angelopoulos, A. N. and Bates, S. A gentle introduction to conformal prediction and distribution-free uncertainty quantification.arXiv preprint arXiv:2107.07511,
-
[4]
Duan, J., Diffenderfer, J., Madireddy, S., Chen, T., Kailkhura, B., and Xu, K. Uprop: Investigating the uncer- tainty propagation of llms in multi-step agentic decision- making.arXiv preprint arXiv:2506.17419,
-
[6]
Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221,
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221,
-
[9]
Reimers, N. and Gurevych, I. Sentence-BERT: Sentence em- beddings using Siamese BERT-networks. InProceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), 2019a. Reimers, N. and Gurevych, I. Sentence-bert: Sentence em- beddings using s...
arXiv 2019
-
[10]
Com- monsenseQA: A question answering challenge targeting commonsense knowledge
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseQA: A question answering challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies,
2019
-
[13]
Yu, W., Yang, Z., Ren, L., Li, L., Wang, J., Lin, K., Lin, C.-C., Liu, Z., Wang, L., and Wang, X. Mm-vet v2: A challenging benchmark to evaluate large multi- modal models for integrated capabilities.arXiv preprint arXiv:2408.00765,
-
[14]
Zhang, R., Zhang, H., and Zheng, Z. Vl-uncertainty: De- tecting hallucination in large vision-language model via uncertainty estimation.arXiv preprint arXiv:2411.11919,
-
[15]
-Instruct
11 A Linear Expectation Constraint for Selective Prediction and Routing with False-Discovery Control A. Proofs A.1. Proof of Theorem 3.1 Condition on the calibrated threshold ˆλ(a) using the calibration set derived by Eq.(7). For the test sample (xn+1, y∗ n+1) with associated model prediction ˆy(a) n+1, uncertainty score u(a) n+1, error indicator err(a) n...
2021
Show all 18 references
-
[16]
as the backbone. For bi-entailment (Kuhn et al., 2023; Farquhar et al., 2024; Wang et al., 2025d), we employ DeBERTa-v37 as the Natural Language Inference (NLI) classifier, which outputs logits over three semantic relation classes: entailment, neutral, and contradiction. Two a...
2023
-
[17]
No feasible threshold for risk levelα
employed in previous research (Lin et al., 2024), the three dimensions of its output logits correspond to entailment, neutral, and contradiction, respectively. 16 A Linear Expectation Constraint for Selective Prediction and Routing with False-Discovery Control We adopt the par...
2024
-
[21]
Similar trends are observed at α= 0.10 and for the Qwen2.5-14B pairing, where routing yields both higher coverage and more correct acceptances than either individual model
At α= 0.05 , for example, routing Qwen2.5-7B with LLaMA-3.1-70B under LEC-Routing increases the total acceptance rate from 50.69% (Qwen2.5-7B alone) and 55.57% (LLaMA-3.1-70B alone) to 57.09%, resulting in a higher number of accepted correct samples. Similar trends are observe...
2016
-
[1963]
Selective labeling with false discovery rate control
Huang, H., Liao, W., Xi, H., Zeng, H., Zhao, M., and Wei, H. Selective labeling with false discovery rate control. arXiv preprint arXiv:2510.14581,
-
[2019]
Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,
-
[2021]
Inductive confidence machines for regression
Papadopoulos, H., Proedrou, K., V ovk, V ., and Gammerman, A. Inductive confidence machines for regression. InMa- chine learning: ECML 2002: 13th European conference on machine learning,
2002
-
[2022]
D., Roy, A., Matejek, B., Acharya, M., Elenius, D., Berenbeim, A
Kaur, R., Samplawski, C., Cobb, A. D., Roy, A., Matejek, B., Acharya, M., Elenius, D., Berenbeim, A. M., Pavlik, J. A., Bastian, N. D., and Jha, S. Addressing uncertainty in LLMs to enhance reliability in generative AI. InNeurips Safe Generative AI Workshop 2024,
2024
-
[2023]
Openchat: Advancing open-source language models with mixed-quality data
Wang, G., Cheng, S., Zhan, X., Li, X., Song, S., and Liu, Y . Openchat: Advancing open-source language models with mixed-quality data. InThe Twelfth International Conference on Learning Representations, 2024a. Wang, Q., Fan, Y ., and Wang, X. E. Safer: Risk-constrained sample-...
2024
-
[2024]
Atf, Z., Safavi-Naini, S. A. A., Lewis, P. R., Mahjoubfar, A., Naderi, N., Savage, T. R., and Soroush, A. The challenge of uncertainty quantification of large language models in medicine.arXiv preprint arXiv:2504.05278,
-
[2025]
Qwen technical report.arXiv preprint arXiv:2309.16609,
Bai, J., Bai, S., Chu, Y ., Cui, Z., Dang, K., Deng, X., Fan, Y ., Ge, W., Han, Y ., Huang, F., et al. Qwen technical report.arXiv preprint arXiv:2309.16609,
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.