{"id":"73c5599b-7951-4c2d-933c-08723594d131","arxiv_id":"2608.12008","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A-CRC-QA calibrates a reliability threshold via a monotonized empirical linear risk, yielding asymptotic control of the accepted-answer error rate while retaining more answers than confidence-bound baselines.","lead":"A-CRC-QA is a post-hoc method that chooses an uncertainty cutoff so a language model's accepted answers stay near a user-set error rate on average, and it is tested on CoQA and MedMCQA. It is a lightweight, model-agnostic combination of linear constraint and conformal risk control ideas with an asymptotic guarantee.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's proof sketch leaves the infimum-attainment step unjustified, so the central asymptotic guarantee is not fully established.","rationale":"The reader's conditional verdict correctly identifies that Theorem 1 is only sketched and that distribution shift is outside the guarantee. My stress-test goes further and locates a specific unjustified step in the proof sketch: the move from the infimum definition to the claim that the selected threshold satisfies the empirical constraint. This is not a fatal flaw; the theorem is likely fixable via a limiting argument using right-continuity, and the empirical evaluation is consistent with the asymptotic claim. However, because the central claim depends on this bridge, the manuscript should either supply the missing argument or explicitly restrict the theorem to the discrete implementation. This does not change the reader's conditional recommendation: the paper should be accepted only after the proof is tightened and the requested artifacts and baseline checks are provided.","tokens_in":14349,"tokens_out":27664,"duration_ms":255898,"concrete_test":"Independently re-derive Theorem 1 using the actual definition S(\\lambda)=1{r\\ge\\lambda} from Eq. (1) without assuming the infimum is attained. Check whether \\hat{g}^\\uparrow_n(\\hat\\lambda_n) \\le -\\gamma_n holds for the infimum; if not, construct a calibration set with a single point at r=0.5, E=1, no other scores, \\Lambda=[0,1], and \\gamma_n=0, and test whether the selected threshold violates the bound. Also verify whether the tie-block procedure in Section 3.6 corresponds to a right-continuous loss; if it does not, the theorem's assumptions do not match the algorithm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Theorem 1 (Section 3.7), whose proof is only a sketch. The critical step is: 'By construction, \\hat{g}^\\uparrow_n(\\hat\\lambda_n) \\le -\\gamma_n'. But \\hat\\lambda_n is defined as an infimum in Eq. (11). If the feasible set {\\lambda : \\hat{g}^\\uparrow_n(\\lambda)+\\gamma_n \\le 0} is open at its left endpoint, the infimum need not lie in the set, and the inequality can fail. The loss in Eq. (4) uses S(\\lambda)=1{r\\ge\\lambda} (Eq. 1), which is left-continuous, not right-continuous, unless a special tie convention is imposed; the proof does not explain how the stated right-continuity or the tie convention ensures the inequality holds at the infimum. The algorithm in Section 3.6 sidesteps this by searching only over observed scores and processing ties as blocks, but the theorem is stated for a general threshold set \\Lambda. This omitted justification is load-bearing because it is the bridge from the empirical monotone-envelope constraint to the asymptotic expectation bound in Eq. (15).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes A-CRC-QA, a post-hoc calibration procedure for selective question answering. The method converts the selection-conditioned error rate SCER(λ)=Pr(E=1|S(λ)=1) into a linear expectation constraint via the instance-wise loss L(λ)=S(λ)(E−α), monotonizes the empirical risk by taking a supremum over more conservative thresholds, adds a vanishing correction γ_n, and chooses the smallest threshold satisfying the monotone envelope constraint. The central theoretical claim is Theorem 1 in Section 3.7: under i.i.d. calibration and test examples and bounded right-continuous losses, the expectation of the test loss at the calibrated threshold satisfies limsup E[L_{n+1}(lambda_hat_n)]≤0, implying asymptotic SCER control when the expected selection probability is bounded away from zero. The experiments on CoQA and MedMCQA with two LLMs and several uncertainty estimators compare the method with Fixed-50, Empirical, Hoeffding and Clopper-Pearson UCB baselines, and LEC-Direct. A-CRC-QA achieves average SCER below the 0.15 target with higher acceptance rates than the UCB baselines and lower violation rates than LEC-Direct, and the calibration-size sweep shows convergence toward the target as n grows.","tokens_in":14539,"tokens_out":22638,"duration_ms":237478,"significance":"The paper addresses a real problem: heuristic uncertainty thresholds do not control the error rate among accepted LLM answers. If Theorem 1 can be made rigorous, the contribution is valuable because the selective loss is non-monotone in the threshold, which blocks the standard finite-sample CRC theorem; an asymptotic calibration rule with a simple closed-form correction is a useful addition to COIN and LEC. The empirical protocol is a strength: 100 calibration–test splits, explicit violation-rate and infeasibility-rate metrics, a calibration-size sweep, and an honest Remark 1 that the guarantee is asymptotic and marginal rather than finite-sample. The ablation separating monotonization from the finite-sample correction is informative. The main weakness is that the proof of the central theorem is only a sketch and contains a genuine attainment gap, so the paper's theoretical contribution is not yet established. The experimental results are consistent with the intended behavior and support the practical recommendation, but they do not by themselves establish Eq. (15).","major_comments":[{"comment":"The proof's key step that the monotonized empirical risk at lambda_hat_n is at most -gamma_n is not justified. Since lambda_hat_n is defined as an infimum in Eq. (11), the feasible set can be open at its left endpoint. In the notation of Section 3.6, this happens when the monotonized empirical risk jumps downward at an observed threshold: all thresholds strictly above a score satisfy the constraint, but the score itself does not, so the infimum falls at the score and the constraint fails at the selected threshold. The algorithm in Eq. (14) avoids this by searching only over observed scores, so the theorem's object is not the object whose behavior is demonstrated. Please supply an attainment lemma for Eq. (11), for example by restricting Lambda to observed thresholds, by choosing any feasible point instead of the infimum, or by proving the feasible set is closed at its left endpoint under the tie convention actually used. Alternatively, the proof can be repaired by bounding the jump at the infimum by O(1/n) and showing the resulting violation vanishes; but as written this step is missing and Eq. (15) is not established.","section":"Section 3.7, Eq. (11) and proof sketch"},{"comment":"Theorem 1 assumes the losses are right-continuous under a fixed deterministic tie convention, but the loss defined in Eq. (4) with S(λ)=1{r≥λ} is left-continuous in λ: at λ=r, S jumps from 1 to 0 as λ increases past r, so the right limit at the jump is 0 while the value is 1. Section 3.6 processes ties as blocks and accepts scores equal to the threshold, which is the left-continuous convention. The manuscript does not define the tie convention that would make the loss right-continuous. This matters because the infimum-attainment problem in the previous comment is controlled by the continuity convention: with a right-continuous loss the feasible set is closed at its left endpoint, while with the left-continuous loss used in the paper it need not be. The theorem's assumptions should be matched to the actual selection rule, or the theorem and Section 3.6 should be changed together.","section":"Section 3.7, Eq. (4) and Eq. (1) (continuity assumption)"},{"comment":"Theorem 1 does not cover the case in which the feasible set in Eq. (11) is empty, although the procedure explicitly handles that case by the all-abstention rule lambda_bottom. This is not a vacuous corner case: Table 2 reports infeasibility rates of 16% and 22% at alpha=0.05. The theorem should state a convention for lambda_hat_n when Eq. (11) is empty (e.g., lambda_hat_n=lambda_bottom, for which L=0 and the bound holds trivially), and the proof should mention this case. Without such a convention, the statement 'let lambda_hat_n be obtained from Eq. (11)' has no object in precisely the settings where the method is exercised.","section":"Sections 3.5 and 3.7 (empty feasible set)"}],"minor_comments":[{"comment":"The abstract says the guarantee holds 'under exchangeability', but Theorem 1 assumes i.i.d. calibration and test examples; please harmonize the wording.","section":"Abstract and Section 3.7"},{"comment":"Eq. (13) would be clearer if it defined C_0=0 and M_0=0 to represent the all-abstain prefix, since thresholds above the largest observed reliability score are otherwise not represented in the prefix-maximum notation.","section":"Section 3.6, Eq. (13)"},{"comment":"At alpha=0.05, the paper reports SCER and AR alongside IF=16–22%; please clarify whether SCER and AR are averaged over all splits or only over feasible splits, and how undefined SCER values (no accepted answers) are handled.","section":"Section 4.3, Table 2"},{"comment":"Please provide the exact split construction or a code artifact; the protocol is described in sufficient detail that reproduction is plausible, but the random-split seeds and any deduplication or filtering steps are not specified.","section":"Section 4.1"},{"comment":"The author affiliation line lists 'Zhangjiang University' in 'Hangzhou, Zhangjiang, China'; this geographic identification should be verified and corrected.","section":"Author affiliation"},{"comment":"The ACM template metadata in the header says 2018 while the arXiv submission is dated 2026; please update the conference and copyright information.","section":"Title page"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a useful, honest synthesis of LEC and CRC for selective QA. The specific procedure—monotonizing the empirical linear risk and adding a vanishing CRC-style correction—is new, and the paper is refreshingly clear about its asymptotic scope. It reports violation rates, infeasibility rates, and calibration-size sweeps, and the ablation study isolates the contributions of the two main components. That is more transparency than most papers in this area.\n\nThe empirical work is solid and the main claim is plausible: at a target error rate of 0.15, A-CRC-QA keeps the average SCER near the target while retaining more answers than confidence-bound baselines, at the cost of a slightly higher violation rate than the very conservative UCB methods. The method is simple, post-hoc, and model-agnostic, so it could be genuinely useful to practitioners.\n\nThat said, the theoretical guarantee is not fully established. The proof sketch of Theorem 1 skips the infimum-attainment step: Eq. (11) defines the threshold as an infimum, and the argument then assumes the inequality holds at that infimum. If the feasible set is open at its left endpoint, this can fail. The right-continuity assumption also sits awkwardly with the loss in Eq. (4), where S(lambda)=1{r>=lambda} is left-continuous. The algorithm avoids this by searching only over observed scores and blocking ties, so the theorem can likely be repaired by restricting to that finite set—but as written, the proof does not cover the stated generality. This is a fixable flaw, not a fatal one, but the authors need to address it before the paper appears.\n\nOther soft spots are minor but real. No code or data are provided, which makes the uncertainty computations (semantic entropy, WSE) hard to reproduce. The LEC-Direct baseline looks simplified relative to what LEC actually proposes. And the manuscript still contains ACM template placeholders in the references and odd affiliation text, which suggests it is an unfinished draft.\n\nOverall: a coherent and honest piece of work with a load-bearing but repairable proof gap. I would not cite it as a rigorous guarantee yet, but I would bring it to a reading group and I would send it to peer review with a request for major revision, primarily to fix the theorem and release artifacts.","headline":"A useful and honest synthesis of LEC and CRC for selective QA that deserves review once the asymptotic proof is tightened and artifacts are released.","tokens_in":15066,"tokens_out":4111,"would_cite":false,"duration_ms":43609,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A post-hoc calibration rule for selective question answering asymptotically caps the error rate among accepted LLM answers at a user-specified tolerance, without retraining the model.","keywords":["selective question answering","uncertainty quantification","risk calibration","abstention","conformal risk control","asymptotic risk control","threshold calibration","large language models"],"falsifier":"Run the calibration on a stream where the test distribution is deliberately shifted from the calibration distribution (calibrate on one medical topic and test on another) and check whether the long-run average SCER stays below $\\alpha$ as the calibration size grows; a systematic exceedance falsifies the claimed asymptotic control. Within the theorem's own assumptions, simulate bounded right-continuous losses with fixed tie-breaking and verify directly whether $\\limsup_{n\\to\\infty}\\mathbb{E}[L_{n+1}(\\hat{\\lambda}_n)]$ remains non-positive.","tokens_in":14137,"feed_emoji":"🎯","tokens_out":11922,"duration_ms":107235,"temperature":0.7,"pith_summary":"Large language models can produce fluent but wrong answers, and a hand-picked uncertainty threshold gives no statistical control over how often the answers actually returned are wrong. This paper proposes A-CRC-QA, a post-hoc calibration rule that turns a held-out calibration set and a user-specified error tolerance $\\alpha$ into one acceptance threshold; under exchangeable data, the expected error rate among accepted answers is asymptotically at most $\\alpha$. The key move is to rewrite selection-conditioned error control as a linear expectation constraint and then calibrate that constraint through a monotonized empirical risk, even though the per-example loss moves in opposite directions for correct and incorrect answers. The authors present this as an asymptotic, marginal guarantee rather than a finite-sample one, and experiments on CoQA and MedMCQA show it retains more answers than conservative confidence-bound baselines at comparable average risk.","feed_headline":"Threshold calibration caps accepted LLM answer errors","feed_subtitle":"A post-hoc, model-agnostic method keeps the average error among returned answers below a user-set target.","key_machinery":"The load-bearing object is the monotonized empirical risk $\\hat{g}^{\\uparrow}_n(\\lambda)=\\sup_{t\\ge\\lambda}\\hat{g}_n(t)$, an upper envelope of the average linear loss over all more conservative thresholds. The per-example loss $L(\\lambda)=S(\\lambda)(E-\\alpha)$ is not monotone in $\\lambda$, because rejecting a correct answer raises the loss while rejecting an incorrect answer lowers it; monotonizing the empirical curve, not each instance loss, restores a monotone population-level target. The calibrated threshold is $\\hat{\\lambda}_n=\\inf\\{\\lambda:\\hat{g}^{\\uparrow}_n(\\lambda)+\\gamma_n\\le 0\\}$ with $\\gamma_n=(1-\\alpha)/(n+1)$, and the proof uses uniform Glivenko-Cantelli convergence of bounded one-dimensional threshold functions to pass from the finite-sample constraint to a limiting expectation constraint. The equivalence $g(\\lambda)=\\mathbb{E}[S(\\lambda)(E-\\alpha)]\\le 0 \\iff \\mathrm{SCER}(\\lambda)\\le\\alpha$ (when selection probability is positive) connects the threshold choice to the user's error target.","core_discovery":"On the paper's own terms, the central claim is that the calibrated decision rule provides asymptotic control of the selection-conditioned error rate: if calibration examples and a future test example are independent and identically distributed and the per-example losses are bounded and right-continuous under a fixed tie convention, then the threshold $\\hat{\\lambda}_n$ chosen from Eq. (11) satisfies $\\limsup_{n\\to\\infty}\\mathbb{E}[L_{n+1}(\\hat{\\lambda}_n)]\\le 0$, so accepted answers error no more than $\\alpha$ in expectation as the calibration set grows. The loss being calibrated is $L(\\lambda)=S(\\lambda)(E-\\alpha)$, where $S$ is the accept indicator and $E$ is the error indicator; controlling its expectation is exactly the same as controlling $\\mathrm{SCER}(\\lambda)\\le\\alpha$ whenever selection has positive probability. Because this loss is not monotone in the threshold, the paper targets the limit rather than a finite-sample guarantee and explicitly warns that the result should not be presented as conditional or high-probability finite-sample validity.","pith_inferences":["Beyond the paper: because the theorem needs only i.i.d. calibration data and bounded right-continuous losses on a one-dimensional threshold class, the same monotonized-envelope calibration should transfer to any scalar-gated decision rule (retrieval acceptance, model routing) whose per-example loss is non-monotone; this is an extension, not a claim the paper makes.","Beyond the paper: the violation rates at small calibration sizes (for instance 26% at $n=100$ on CoQA) imply that the asymptotic guarantee does not say when a finite dataset is large enough; a practical extension would be a calibration-size diagnostic that reports the largest $\\alpha$ supportable at a given $n$.","Beyond the paper: distribution shift breaks the guarantee, so a natural testable extension is sliding-window recalibration or reweighted exchangeability schemes; the paper lists distribution shift only as future work.","Beyond the paper: the ablation that monotonizes each instance loss accepts fewer than 19% of answers, which suggests a finite-sample guarantee for this problem, if one exists, will require a different loss design than per-instance monotone envelopes."],"forward_implications":["If Theorem 1 holds, a deployment can specify a target accepted-answer error rate $\\alpha$ and, with enough calibration data from the same distribution, the long-run average error among returned answers will not exceed $\\alpha$; the residual finite-sample risk is visible in the empirical violation rate across calibration-test splits.","Because the calibration is post-hoc and needs only a scalar score, the same rule attaches a statistically meaningful operating point to any existing uncertainty estimator, so better uncertainty ranking and better calibration compound rather than compete.","Monotonizing the empirical risk cuts the violation rate roughly in half relative to direct linear-expectation calibration on the two tested benchmarks, at a cost of about 3 percentage points of acceptance rate; compared with confidence-bound calibration, it accepts roughly 7 percentage points more answers.","At small target error rates the feasible set may be empty, meaning the underlying model and uncertainty signal are not reliable enough to support that operating point; the method then abstains on everything rather than pretending the risk is achievable."],"supporting_citations":[{"why":"supplies the bounded-loss risk-control calibration device and the asymptotic argument for non-monotone losses that the proposed method adapts.","marker":"[1]"},{"why":"reformulates selection-conditioned risk control as a linear expectation constraint connecting SCER to a simple expectation bound.","marker":"[35]"},{"why":"supplies the confidence-bound selective QA calibration strategy used as a comparison baseline and the motivation for improving answer retention.","marker":"[37]"},{"why":"supplies the semantic entropy uncertainty estimator used in the open-ended CoQA experiments.","marker":"[9]"},{"why":"supplies the word-sequence entropy uncertainty estimator used in the free-form answer experiments.","marker":"[38]"},{"why":"supplies the Hoeffding inequality used to build the UCB-HFD baseline.","marker":"[13]"},{"why":"supplies the Clopper-Pearson confidence bound used in the UCB-CLP comparison baseline.","marker":"[8]"},{"why":"supplies the MedMCQA medical multiple-choice dataset used for the closed-ended QA evaluation.","marker":"[25]"},{"why":"supplies the CoQA conversational question answering dataset used for the open-ended evaluation.","marker":"[28]"}],"fun_headline_variants":["Asymptotic error cap for selected LLM answers","Calibration tames accepted-answer error rate","Post-hoc method bounds LLM answer error asymptotically","Selective QA error control via calibration","Risk-calibrated acceptance limits for LLM answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee rests entirely on calibration and test examples being exchangeable draws from the same distribution, with bounded right-continuous losses; if deployment data comes from a shifted distribution, the calibrated threshold has no stated control over the accepted-answer error rate.","fun_headline_variants_meta":{"raw":{"variants":["Asymptotic error cap for selected LLM answers","Calibration tames accepted-answer error rate","Post-hoc method bounds LLM answer error asymptotically","Selective QA error control via calibration","Risk-calibrated acceptance limits for LLM answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1550,"prompt_tokens":946,"completion_tokens":604,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":532}},"tokens_in":562,"tokens_out":604,"duration_ms":5581,"temperature":1.0,"reasoning_tokens":532,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:19:27.390598+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the calibration on a stream where the test distribution is deliberately shifted from the calibration distribution (calibrate on one medical topic and test on another) and check whether the long-run average SCER stays below $\\alpha$ as the calibration size grows; a systematic exceedance falsifies the claimed asymptotic control. Within the theorem's own assumptions, simulate bounded right-continuous losses with fixed tie-breaking and verify directly whether $\\limsup_{n\\to\\infty}\\mathbb{E}[L_{n+1}(\\hat{\\lambda}_n)]$ remains non-positive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the bounded-loss risk-control calibration device and the asymptotic argument for non-monotone losses that the proposed method adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"reformulates selection-conditioned risk control as a linear expectation constraint connecting SCER to a simple expectation bound."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the confidence-bound selective QA calibration strategy used as a comparison baseline and the motivation for improving answer retention."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the word-sequence entropy uncertainty estimator used in the free-form answer experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the CoQA conversational question answering dataset used for the open-ended evaluation."}],"review_version":1}