{"id":"f6bdb990-33dd-432e-a9c7-c9b552042741","arxiv_id":"2504.14154","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Selective Conformal Uncertainty (SConU) filters test questions with a conformal outlier test before prediction, and SConU-Pro refines that test, aiming to preserve coverage in large language model question answering.","lead":"The paper introduces SConU, a method that filters out 'uncertainty outliers' before applying conformal prediction to language models, so the model only answers questions it judges as similar to its calibration data. It claims to keep error rates at a user-set level even when the test and calibration data come from different domains, but the statistical guarantee for this filtering is not proven.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix C's key inequality (10) for the refined p-value p' is asserted without proof and is false: since k is a subset count, P(k≤m) ≥ P(K≤m), not ≤, so SConU-Pro has no valid significance test.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing flaw, and I agree. The paper's headline contribution—significance tests that manage miscoverage after filtering—rests entirely on the validity of p' in Eq. (2). The proof in Appendix C is not merely missing a justification; the claimed inequality has the wrong direction because k is a subset of K. The N=2 counterexample and the 'unanswerable calibration questions' scenario show p' can have rejection probability 1 at δ>1/(N+1), which is the opposite of super-uniformity. This is an internal inconsistency, not a disagreement with consensus. The empirical sections do not repair it: EMR after filtering is reported only on the selected subset, with no selection rates and no abstention baseline, so the 'rigorous management' conclusion cannot be separated from simply discarding hard questions. I therefore keep the reader's REJECT verdict; no change is needed.","tokens_in":37499,"tokens_out":6393,"duration_ms":60263,"concrete_test":"Independently re-derive Eq. (10) from the definition of k and K: because k≤K, P(k≤m) ≥ P(K≤m); the printed ≤ is backwards. Then verify by simulation under an exchangeable null: draw U_1,...,U_{N+1} iid, set R_i=1 for a random subset with, say, 20% zeros, compute p'=(1+k)/(N+1) over many trials, and plot empirical P(p'≤δ) for δ∈{0.05,0.1}; if it exceeds δ, the p-value is invalid. Also rerun Table 2/Figure 3b reporting the selection rate and a random-abstention baseline at the same selection rate; if EMR coverage is matched, the apparent guarantee is abstention, not conformal validity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that both conformal p-values in Eqs. (1)-(2) satisfy the statistical definition of p-values, and that filtering on them gives 'rigorous management of miscoverage rates' (Abstract, §3.2). The proof for the unrefined p-value in Eq. (8) is standard. The proof for p'_{N+1} in Appendix C fails at the asserted inequality P(k≤m) ≤ (m+1)/(N+1) (Eq. 10), where k counts calibration points with u_i≥u_{N+1} and y*_i ∈ E(x_i,D_cal,α). Let K count all calibration points with u_i≥u_{N+1}; then k≤K, so {K≤m} ⊆ {k≤m}, giving P(k≤m) ≥ P(K≤m) = (m+1)/(N+1). The printed inequality has the wrong direction. A concrete counterexample to (10): take N=2, exchangeable iid uncertainties, R_1=1, R_2=0. Then k=1{U_1≥U_3}, so P(k≤0)=1/2 > 1/3=(m+1)/(N+1) for m=0. In the paper's own setting, if many calibration questions have no correct answer in their sampled candidate sets—explicitly allowed in §3.2—then R_i=0 for those points, p'=1/(N+1) regardless of the test uncertainty, and P(p'≤δ)=1 for δ>1/(N+1). Thus p' is not super-uniform and SConU-Pro's outlier test is not a valid hypothesis test. The claimed finite-sample coverage guarantee for the selected test set therefore has no support; the empirical EMR gains could be pure abstention. Separately, no theorem shows that conditioning on p>δ preserves exchangeability with the calibration set, so even the SConU filter lacks a coverage proof on the remaining samples.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces SConU and SConU-Pro, two procedures that compute conformal p-values from an LLM's uncertainty scores and filter out test samples deemed to violate exchangeability with the calibration set; the remaining samples are then fed to a standard conformal uncertainty framework. The paper also derives a 'minimum risk level' to justify keeping calibration points whose sampled candidate sets do not contain a correct answer. It claims rigorous management of miscoverage rates in single-domain and cross-domain QA tasks and reports experiments on MMLU, MMLU-Pro, MedMCQA, TriviaQA, and CoQA, together with ablations over risk level, sampling size, and uncertainty measures.","tokens_in":37896,"tokens_out":6426,"duration_ms":58945,"significance":"If the guarantees were valid, SConU would offer a practical way to combine outlier detection with conformal prediction for LLM question answering. The paper's strengths are its clear problem statement, the minimum-risk-level analysis in Section 3.2, the breadth of the empirical study, and the public code release. However, the central statistical claim rests on a p-value validity proof that is incorrect, and no theorem establishes coverage after filtering; the empirical EMR results are computed on the filtered set, so they do not substitute for a guarantee. As a statistical contribution the central claim is not established.","major_comments":[{"comment":"Equation (10) states P(k≤m) ≤ (m+1)/(N+1), where k counts calibration points satisfying both u_i ≥ u_{N+1} and y*_i ∈ E(x_i, D_cal, α). Let K be the count of all calibration points with u_i ≥ u_{N+1}; since k ≤ K, the event {K ≤ m} implies {k ≤ m}, so P(k≤m) ≥ P(K≤m) = (m+1)/(N+1). The printed inequality has the wrong direction. A concrete counterexample with N=2, exchangeable iid uncertainties, and R_1=1, R_2=0 gives k = 1{U_1 ≥ U_3}, so P(k≤0)=1/2 > 1/3. Moreover, in the paper's own setting (Section 3.2 keeps calibration points even when no sampled response matches y*_i), if all calibration points have y*_i ∉ E then R_i=0 for all i, so p'_{N+1}=1/(N+1) regardless of the test uncertainty and P(p'≤δ)=1 for any δ>1/(N+1). Thus p'_{N+1} is not super-uniform and SConU-Pro's outlier test is not a valid significance test; the claimed finite-sample control for SConU-Pro is unsupported.","section":"Appendix C, Eq. (10)"},{"comment":"The paper asserts that after filtering out low-p-value test samples, \"we conduct conformal procedures for samples within the remaining test set with finite-sample guarantees of correctness coverage\" (Section 3.2), and claims \"rigorous management of the miscoverage rates\" (Abstract, Section 4.2). No theorem establishes that the selected test samples remain exchangeable with the calibration set, or that the standard conformal quantile provides coverage conditional on p_{N+1} > δ. Because the p-value is a function of the test uncertainty and the whole calibration set, conditioning on survival changes the distribution of the nonconformity scores used for the subsequent ConU step. This affects SConU, not only SConU-Pro, and is therefore load-bearing for the paper's central claim.","section":"Section 3.2, Workflow of SConU; Section 4.2, Marginal Coverage"},{"comment":"The EMR is computed only on test samples that pass the outlier filter, so the reported miscoverage rates measure performance on the surviving subset rather than on the full test stream. Since the filter is designed to remove high-uncertainty points, which are exactly the points most likely to violate coverage, the reported improvements are partly an artifact of abstention. The paper should report the full-test EMR with rejected points counted as errors, or provide a formal bound on the full-test miscoverage rate that accounts for the selection step. As it stands, the empirical results do not demonstrate the claimed guarantee.","section":"Tables 2–4 and Figures 3, 6, 8, 9"},{"comment":"In the cross-domain experiments, the risk level α is not fixed by the user ex ante but is set as the maximum of the minimum manageable risk levels computed from the calibration set via Eq. (5). This data-dependent choice is legitimate for describing what risk is achievable, but it is not a user-specified level, and controlling EMR at a level chosen after seeing the calibration data is not the same as the guarantee advertised in the abstract. This should at least be stated explicitly in the main text and in the captions.","section":"Figure 1b and Appendix E"}],"minor_comments":[{"comment":"The text says \"if M < 0\" but should say \"if m < 0\"; the variable m is also introduced after it is first used.","section":"Appendix C, after Eq. (9)"},{"comment":"The notation E(x_i, D_cal, α) is used in the definition of p' before the prediction-set construction is specified; define it formally in the main text.","section":"Section 3.2, Eq. (2)"},{"comment":"The phrase \"qIn order\" is a typo for \"In order\".","section":"Appendix B.2"},{"comment":"The diagonal entries are set manually to α or 1; stating this only in the captions is insufficient, and the main text should clarify that these are not empirical measurements.","section":"Figures 1, 3, 8, and 9"},{"comment":"Several entries have standard deviations exceeding the mean (e.g., 0.1056±0.1860 at size 3); reporting only mean±std for these highly skewed quantities is uninformative, and the authors should consider median or trimmed statistics.","section":"Table 3"}],"recommendation":"reject","confidential_remarks":"The core statistical proof in Appendix C is incorrect, and the selective-coverage guarantee that the paper advertises is not proven; the empirical evaluation on filtered sets cannot compensate for this. I recommend rejection. If the authors can supply a valid selective-coverage theorem and adjust the evaluation to account for abstention, a resubmission could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The quick take: the headline claim of rigorous miscoverage control after filtering is not supported. Appendix C's proof that the refined p-value p' is valid contains a false inequality. The authors assert P(k ≤ m) ≤ (m+1)/(N+1), but since k is a subset of the usual rank-based count K, the inequality runs the other way: P(k ≤ m) ≥ P(K ≤ m) = (m+1)/(N+1). The stress-test's counterexample is correct. So SConU-Pro's outlier test is not a valid hypothesis test, and the claimed finite-sample guarantee collapses.\n\nWhat's good: the paper identifies a real problem—exchangeability violations in cross-domain LLM QA—and runs a wide set of experiments across multiple models and datasets. The basic conformal p-value (Eq. 1) is correctly cited to Bates et al. and Jin & Candès, and the minimum risk level α_l = N L_N(1)/(N+1) is a simple but useful bound. The practical point about not manually pruning calibration samples is worth keeping.\n\nThe soft spots beyond the broken proof: even for the basic SConU filter, no theorem shows that conditioning on p > δ preserves any coverage guarantee with the calibration set; selection breaks exchangeability. The empirical section never reports the fraction of test points filtered, and there is no abstention baseline. If you remove the hardest questions, EMR on the survivors trivially looks good. The risk level α in cross-domain experiments is chosen from calibration data, which is another source of optimism not covered by the theory.\n\nOverall, the empirical study is substantial, but the central methodological promise is not there. The paper could be reframed as an empirical study of uncertainty-based abstention heuristics, without claiming rigorous significance tests or coverage bounds. As is, I would not accept it. A careful referee would catch the Appendix C error quickly, but the breadth of experiments still merits a serious read. I'd send it to a serious referee rather than desk reject, but with the expectation of heavy revision or rejection unless the theory is repaired.","headline":"The central claim of rigorous post-filter coverage is undone by an invalid proof of the refined conformal p-value; the broad empirical work does not rescue the missing theory.","tokens_in":38474,"tokens_out":3609,"would_cite":false,"duration_ms":31981,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A conformal p-value lets LLMs abstain when a question is an uncertainty outlier.","keywords":["selective conformal prediction","conformal p-value","LLM question answering","exchangeability","uncertainty outlier detection","miscoverage control","calibration set","predictive entropy"],"falsifier":"Construct a calibration set in which no sampled candidate set contains the true answer; then in Eq. (2) the refined count $k'$ is always 0, so $p'_{N+1} = 1/(N+1)$ for every test point. At any risk level $\\delta > 1/(N+1)$, the procedure rejects every exchangeable test point, contradicting the claimed validity and the statement that the remaining samples meet user-specified marginal coverage.","tokens_in":37256,"feed_emoji":"🎯","tokens_out":7554,"duration_ms":63991,"temperature":0.7,"pith_summary":"SConU attacks a specific failure mode of conformal prediction for large language models: test questions whose uncertainty profile is not exchangeable with the calibration set, which make user-specified coverage guarantees break. The paper's proposal is to convert the exchangeability precondition into a per-question significance test: a conformal p-value derived from predictive entropy decides whether the new question is statistically unlike the calibration data, and the model declines to answer when the p-value is too small. A refined version, SConU-Pro, also counts only calibration points whose own prediction sets contain the true answer, and the paper reports that after this filtering the empirical miscoverage rate stays within the user-set risk level in both single-domain and cross-domain QA settings. The same framework keeps the full calibration set, derives the minimum risk level the set can control, and shrinks prediction sets to near-singleton size after semantic deduplication.","feed_headline":"Conformal p-values flag LLM outliers before coverage promises apply.","feed_subtitle":"A new QA framework abstains on out-of-distribution questions, keeping miscoverage at user-set risk levels.","key_machinery":"The central object is the conformal p-value. In SConU, Eq. (1) computes $p_{N+1} = (1 + \\sum_{i=1}^N \\mathbf{1}\\{u_i \\ge u_{N+1}\\})/(N+1)$, with $u_i$ the predictive entropy of the model on question $i$; small p-values reject the null hypothesis that the calibration set can provide coverage for the new point. SConU-Pro uses Eq. (2), $p'_{N+1} = (1 + \\sum_{i=1}^N \\mathbf{1}\\{u_i \\ge u_{N+1},\\, y_i^* \\in \\mathcal{E}(x_i,\\mathcal{D}_{\\rm cal},\\alpha)\\})/(N+1)$, which only counts calibration points whose own conformal prediction set contains the true answer. The p-value carries the argument because it turns the exchangeability precondition into a testable per-sample statement, and the calibration-set prediction status is the mechanism that gives SConU-Pro its tighter control.","core_discovery":"The paper's central claim is that the miscoverage anomalies observed in existing conformal uncertainty frameworks are caused by uncertainty data outliers — test points whose uncertainty scores sit outside the calibration distribution — and that significance tests can remove those outliers before the conformal quantile step. Eq. (1) defines the base conformal p-value; Eq. (2) defines the refined version. After filtering, the paper argues, the remaining test samples are expected to satisfy user-specified marginal coverage even when the original exchangeability condition is violated, and the experiments on MMLU-Pro show empirical miscoverage rates at or below the target risk level across calibration/test subject pairs. The framework also introduces the minimum manageable risk level, quantifies when a calibration set can support a given error rate, and keeps calibration samples that earlier methods would discard.","pith_inferences":["If the refined p-value validity holds, the same filtering logic could be ported to semantic-entropy or word-sequence-entropy uncertainty scores, potentially catching hallucinations that predictive entropy misses.","The framework's abstention rule is equivalent to a data-dependent refusal set; a natural diagnostic is whether the rejected questions are the same ones a human expert would flag as out-of-domain.","When the refined p-value is not a valid p-value (e.g., if the asserted inequality in Appendix C fails), the observed cross-domain improvement may come from selective abstention rather than conformal control; comparing coverage on filtered-in versus filtered-out test points would separate the two.","The minimum-risk-level analysis suggests a practical deployment rule: measure the minimum manageable risk level after model deployment, and refuse any task whose required risk level falls below it."],"forward_implications":["A deployed QA system can abstain on out-of-distribution questions instead of emitting empty or unreliable prediction sets.","The minimum risk level tells a practitioner the lowest error rate a given calibration set can support with finite sampling.","Because the filter is score-agnostic, it layers onto logit-based, frequency-based, and black-box conformal uncertainty methods.","The experiments indicate that refined outlier filtering reduces cross-domain empirical miscoverage rates to within the target risk level and trims average prediction set size toward 1.","Semantic deduplication cuts average prediction set size from roughly 8–9 to about 1 in open-domain QA, making prediction sets actionable."],"supporting_citations":[{"why":"Introduced split conformal prediction, the base procedure SConU builds on.","marker":"Papadopoulos et al., 2002"},{"why":"Provides the conformal prediction framework, notation, and coverage quantile arguments used throughout.","marker":"Angelopoulos and Bates, 2021"},{"why":"Supplies the conformal p-value construction for outlier detection that Eq. (1) adapts.","marker":"Bates et al., 2023"},{"why":"Gives selection-by-prediction conformal p-values and the multiple-testing perspective used for significance testing.","marker":"Jin and Candès, 2023"},{"why":"Shows how conformal p-values decide when to trust a foundation model, a direct template for SConU.","marker":"Gui et al., 2024"},{"why":"Provides the baseline conformal uncertainty procedure for multiple-choice QA that SConU filters.","marker":"Kumar et al., 2023"},{"why":"The ConU framework for black-box LLM generation that SConU retains and improves.","marker":"Wang et al., 2024c"},{"why":"Conformal language modeling framework serving as a risk-control baseline.","marker":"Quach et al., 2024"}],"fun_headline_variants":["SConU filters outliers to keep LLM coverage promises","Conformal p-values catch LLM outliers before coverage","SConU: significance tests for reliable LLM coverage","Detect outliers first, then guarantee LLM coverage","SConU keeps miscoverage low via conformal p-values"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole guarantee rests on the claim that the refined conformal p-value in Eq. (2) is a true p-value under the null hypothesis; the proof in Appendix C uses an inequality, $P(k' \\le m) \\le (m+1)/(N+1)$, that is asserted without proof and can fail, and if it fails the miscoverage guarantee collapses.","fun_headline_variants_meta":{"raw":{"variants":["SConU filters outliers to keep LLM coverage promises","Conformal p-values catch LLM outliers before coverage","SConU: significance tests for reliable LLM coverage","Detect outliers first, then guarantee LLM coverage","SConU keeps miscoverage low via conformal p-values"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000379,"raw_usage":{"total_tokens":1981,"prompt_tokens":880,"completion_tokens":1101,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":1020}},"tokens_in":496,"tokens_out":1101,"duration_ms":7966,"temperature":1.0,"reasoning_tokens":1020,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:55:38.432487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a calibration set in which no sampled candidate set contains the true answer; then in Eq. (2) the refined count $k'$ is always 0, so $p'_{N+1} = 1/(N+1)$ for every test point. At any risk level $\\delta > 1/(N+1)$, the procedure rejects every exchangeable test point, contradicting the claimed validity and the statement that the remaining samples meet user-specified marginal coverage.","supporting_citations":[],"review_version":1}