{"id":"426d6846-251c-4b6a-a8db-6d5c1941996a","arxiv_id":"2412.15683","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PROBAR, the estimated probability that a language model's sampled responses are adequate to the prompt, outperforms semantic entropy for selective prediction across ambiguous and open-ended prompts.","lead":"A new uncertainty score called PROBAR measures how often a language model's sampled answers pass an adequacy check, instead of treating different answers as a sign of error. On question answering and next-word prediction, it beats semantic entropy at deciding when to trust the model's output.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AmbigQA results rest on an adequacy classifier never validated against human labels, an acknowledged limitation that leaves half the QA evidence for PROBAR's advantage unsupported.","rationale":"The paper's central claim is that PROBAR, the probability of adequate responses, is a better confidence measure than semantic entropy across prompts of varying open-endedness. The evidence base is three datasets; AmbigQA is the only knowledge-based QA dataset and the only one where the adequacy classifier was never evaluated against human annotations. The authors state this in Limitations, so it is an acknowledged missing support rather than a hidden flaw. Without human labels for AmbigQA, the observed AUROC advantage could reflect the specific inductive biases of Mis22B (e.g., leniency, style preferences) that happen to align with gpt3.5-turbo's correctness judgments, both of which are prompted to use parametric knowledge. This is a concrete, testable gap: manual adequacy labels would let us compute the classifier's F1 and, more importantly, an upper-bound PROBAR(Manual) AUROC. If that upper-bound still beats Norm.SE, the AmbigQA result is genuine; if not, the claim must be restricted to RCQA and NWP. The reader's weakest assumption pointed to judge/classifier correlation on Abg-COQA; our concern is the same class of issue but located at the least-validated dataset, which the paper itself flags. We therefore partially agree with the reader. We do not propose to reject the paper; the Abg-COQA manual analysis and the Provo exact-match results provide independent support, and the released code makes replication feasible. The appropriate verdict remains CONDITIONAL: the AmbigQA claim should be conditioned on validation of the adequacy classifier.","tokens_in":28170,"tokens_out":7728,"duration_ms":67550,"concrete_test":"Apply the manual-annotation protocol of Appendix A.5 to 50 randomly selected AmbigQA prompts: have two annotators independently label the 10 sampled responses per OPT model for adequacy, then compute (i) Mis22B's F1 against these labels and (ii) the AmbigQA AUROC panel of Figure 4 replacing PROBAR(LM) with PROBAR(Manual). If PROBAR(Manual) does not outperform Norm.SE(NLI) by at least as much as PROBAR(LM) did, or if Mis22B's F1 falls below the 0.70 range observed for Abg-COQA, the reported AmbigQA advantage is not established and the claim should be restricted to RCQA and NWP.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap in the empirical case is the AmbigQA leg. In the Limitations section the authors state: 'The classifiers for the KBQA task was not explicitly evaluated for their performance, and we only \"implicitly\" validate their performance through the improved AUROC performance of PROBAR.' AmbigQA supplies roughly half of the QA evidence for the headline claim, and its correctness labels are produced by gpt3.5-turbo using a prompt that, like the Mis22B adequacy classifier, appeals to the model's parametric knowledge. Neither the adequacy classifier nor the correctness judge is checked against human labels for this dataset. If Mis22B's 'adequate' decisions are noisy or systematically misaligned with what a human would accept, the PROBAR AUROC advantage over Norm.SE in Figure 4 (right) could be an artifact of the classifier rather than a property of the PROBAR estimator. The Abg-COQA and Provo results are better supported (manual classifier evaluation, exact-match correctness for Provo), but the headline claim is stated across both QA datasets and varying open-endedness; the AmbigQA result is the least secure support and is explicitly flagged by the authors.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that semantic variability among sampled LM responses is not necessarily evidence of error, especially for ambiguous or open-ended prompts. It defines PROBAR (Eq. 3) as a Monte Carlo estimate of the probability that sampled responses are judged adequate by a classifier or LLM judge, and evaluates PROBAR as a confidence measure for selective prediction with OPT models (2.7B-30B) on Abg-COQA, AmbigQA, and Provo next-word prediction. Compared with entropy, semantic entropy, and a P(True) variant, the paper reports that PROBAR mostly achieves higher AUROC and better selective precision-coverage trade-offs, supplemented by human annotation on Abg-COQA and Provo, upper-bound analyses on 50 manually labeled contexts, and ablations on sample size and bootstrap variance.","tokens_in":28332,"tokens_out":4879,"duration_ms":43075,"significance":"If the empirical claims hold, PROBAR is a conceptually useful alternative to consistency-based uncertainty: it formalizes confidence as the rate of adequate, rather than semantically homogeneous, responses, and it is simple to compute from samples and a judge. The study is thorough in coverage (three datasets, four OPT sizes, human annotation, classifier ablations, sample-size analysis, and bootstrap variance), and the core estimator in Eq. (3) is transparent and free of fitted parameters. The paper also releases code and data, which supports reproducibility. The main reservation is that one of the two QA legs (AmbigQA) rests on an adequacy classifier that is never validated against human labels, while the correctness judge may share a model-specific bias with that classifier; this makes the headline claim stronger than the current evidence supports.","major_comments":[{"comment":"The AmbigQA leg is load-bearing for the headline claim, but its adequacy classifier (Mistral-Small-Instruct-2409, Appendix A.4) is never validated against human labels; the Limitations section explicitly concedes that the KBQA classifier was 'not explicitly evaluated' and only 'implicitly' validated through improved AUROC. Because the correctness labels for AmbigQA are produced by gpt3.5-turbo asked to judge plausibility 'given your training data' (Appendix B.2) and the adequacy classifier is prompted to judge plausibility 'with respect to your training data' (Appendix A.4), a shared model-specific bias could inflate PROBAR's AUROC without any human ground truth to detect it. This needs human validation of the adequacy classifier on AmbigQA, or at least a sensitivity analysis with an independently sourced judge, before the cross-dataset claim is supported.","section":"Section 5 (AmbigQA) and Limitations"},{"comment":"The abstract and Section 1 state that PROBAR outperforms semantic entropy 'across prompts with varying degrees of ambiguity/open-endedness,' but the AmbigQA results show OPT-13B, where PROBAR does not outperform Norm.SE and the paper itself notes 'except for OPT13B.' The headline claim should either be qualified to exclude this exception or the exception should be explained; as written, the claim is broader than the reported evidence.","section":"Section 5 (AmbigQA), Figure 4 right"},{"comment":"The main AUROC comparisons are presented as point estimates without confidence intervals or error bars. Appendix C.3 provides bootstrap distributions only for Abg-COQA (and an appendix figure for AmbigQA), not for the headline figures, and several reported advantages are small, for example in the Abg-COQA ambiguous test set in Figure 3b. Without intervals or paired significance tests, the reader cannot tell whether the reported advantage of PROBAR over Norm.SE is statistically distinguishable from noise.","section":"Figures 3 and 4"}],"minor_comments":[{"comment":"The dataset name is spelled inconsistently: 'AbgCOQA', 'Abc-COQA', and 'Abg-COQA' all appear within one caption; the rest of the paper should use a single spelling.","section":"Figure 3 caption"},{"comment":"The text refers to 'Section 2.2' to explain the NLI-based clustering algorithm, but the section numbering in the paper does not include a Section 2.2; this cross-reference should be updated.","section":"Appendix A.2"},{"comment":"There are typos in the related work paragraph, including 'diffeent methods' and the garbled phrase 'rather than repeatedly sampling, responses’ some work perturbs inputs,' which should be rewritten.","section":"Section 7 (Related Work)"},{"comment":"Section 4 says 100 contexts were sampled from Provo, while the Figure 4 caption says '200 prompts'; clarify that the count includes the 100 corrupted counterpart contexts.","section":"Section 4 and Figure 4 caption"},{"comment":"The claim that 5 samples give 'reasonably similar performance' to 10 samples in Figure 17 is not accompanied by error bars or a statistical comparison; please state whether the difference is within sampling variability.","section":"Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its main limitation, which is to its credit. My recommendation of major revision is driven by the gap between the broad claim and the unvalidated AmbigQA leg; I do not see grounds for reject, because the core idea is sound and the missing validation is a concrete, fixable empirical step rather than a flaw in the definition of PROBAR."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is the short version: PROBAR is a simple, well-motivated idea—estimate the probability mass the model puts on responses judged adequate to the prompt, rather than on responses semantically consistent with each other—and the paper gives the best evidence I've seen that this beats semantic entropy for selective prediction in open-ended QA and next-word prediction. No big theory, but a real conceptual correction, and the empirical work mostly backs it up.\n\nWhat is actually new: the framing, which is cleanly different from Kuhn et al.'s semantic entropy. The estimator is just a Monte Carlo fraction, so the contribution is the observation that adequacy, not semantic homogeneity, is the right target for reliability in ambiguous and underspecified settings. The manual upper-bound analysis on 50 Abg-COQA contexts is the strongest section: with human adequacy labels, PROBAR beats human-labeled semantic entropy by a large margin, which rules out the worry that the advantage comes only from their classifier. They also release code, prompts, generations, and annotations, and they include sample-size and bootstrap ablations. That is careful, reproducible work.\n\nSoft spots, in order of seriousness. First, the AmbigQA adequacy classifier (Mistral judging from parametric knowledge) is never checked against human labels, and the correctness judge for that dataset is also an LLM using its parametric knowledge. The authors admit this in the Limitations section. It matters because AmbigQA is half of the QA evidence; shared biases between the adequacy judge and the correctness judge could inflate the AUROC gap there. Second, the main AUROC figures have no confidence intervals; the bootstrap in the appendix is good but is not attached to the headline numbers. Third, the claim that PROBAR wins on both more and less ambiguous prompts has an exception—OPT-13B on AmbigQA—which the paper notes but does not reconcile with the abstract. The Provo classifier gets a small human evaluation (5 contexts), which is thin but acceptable for a pilot.\n\nMy overall read: the central argument holds. The AmbigQA gap is legitimate and should be fixed, but it is not load-bearing, because the Abg-COQA upper-bound result and the Provo results stand independently of it. I would send this to reviewers with a request for a human validation of the AmbigQA classifier and confidence intervals on the main plots. It deserves a serious refereeing, and I would expect a solid publication after those revisions.","headline":"A simple, well-tested idea—adequacy-based confidence (PROBAR) beats semantic entropy in open-ended settings—with one honest, fixable gap in the AmbigQA half.","tokens_in":28880,"tokens_out":3387,"would_cite":true,"duration_ms":30831,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Judge language-model responses for adequacy, not semantic variation, to know when to trust them.","keywords":["selective prediction","uncertainty quantification","semantic entropy","adequacy","open-ended language generation","PROBAR","language models"],"falsifier":"Recompute the QA AUROC comparisons with correctness labels produced by a second independent judge, such as two human annotators or an LLM from a different model family than the adequacy classifier, and check whether PROBAR still beats semantic entropy by the same margin; if the advantage collapses or reverses under the alternative judge, the central claim is not robust.","tokens_in":27933,"feed_emoji":"🎯","tokens_out":6472,"duration_ms":50684,"temperature":0.7,"pith_summary":"The paper argues that a language model's uncertainty should be measured by how often it produces adequate responses, not by how varied its responses are. Semantic variation is a poor error signal in ambiguous or open-ended settings, where a model may give many different yet valid answers. The paper introduces PROBAR, an estimate of the probability a model assigns to responses judged adequate to the prompt, computed as the fraction of sampled responses a classifier labels adequate. In selective prediction experiments with OPT models across two QA datasets and next-word prediction, PROBAR reports higher AUROC than entropy, semantic entropy, and a P(True) variant, for both ambiguous and non-ambiguous prompts. If the finding holds, the practical lesson is that confidence should be read as the rate of adequate answers, not as semantic homogeneity.","feed_headline":"Judge answers for adequacy, not variety, to tell when an LLM errs","feed_subtitle":"Count adequate answers, not semantic variety, to predict when a model is reliable.","key_machinery":"The central object is PROBAR, an MC estimate of the probability that the model, given a prompt, produces a response judged adequate: $\\mathrm{PROBAR} = \\frac{1}{N}\\sum_{n=1}^{N} [a(x, y_n) = A_1]$, where $a$ is a judge mapping prompt-response pairs to adequate or inadequate and $y_1,\\dots,y_N$ are sampled responses. The judge can be a human, a trained classifier, or a large language model; the paper implements it with an instruction-tuned language model for adequacy and uses a separate LLM judge for the correctness labels in QA. The measure works by interpreting the relative frequency of adequate samples as instance-level confidence, so that high semantic variability is not penalised when the model's varied responses are all adequate. It is this adequacy judge that carries the argument: PROBAR is only as informative as the classifier's ability to recognise plausible answers to open-ended prompts.","core_discovery":"The paper's claim is that an uncertainty score should quantify the probability that a sampled response is adequate to the prompt, rather than the variability of the response set. Semantic entropy treats dispersion over meaning clusters as evidence of error; PROBAR instead groups responses by adequacy and accumulates the model's probability mass over adequate responses. Using OPT models of four sizes and prompts of varying open-endedness, the paper reports that PROBAR achieves higher AUROC and better selective precision versus coverage than semantic entropy and other baselines, in both ambiguous and non-ambiguous settings. The authors further show that the advantage is not an artifact of automation: replacing PROBAR's adequacy classifier and semantic entropy's clustering with human judgements still leaves PROBAR ahead, and even the practical PROBAR implementation beats semantic entropy's human-labelled upper bound.","pith_inferences":["One extension the authors leave implicit is making adequacy multi-dimensional, so a response counts only if it is plausible, grammatical, and non-toxic, which would carry the method toward safety-sensitive generation.","A testable extension is to recompute the QA comparisons with correctness labels from a judge drawn from a different model family than the adequacy classifier, isolating whether shared judge biases inflate PROBAR's margin.","Because PROBAR separates what the model knows (adequacy mass) from how it hedges (semantic spread), it points toward a two-dimensional reliability profile rather than a single uncertainty score."],"forward_implications":["In selective prediction, choosing responses by PROBAR instead of semantic entropy should raise precision at fixed coverage for ambiguous and open-ended prompts.","PROBAR can flag confident errors, where a model commits strongly to a single inadequate response, a case that variation-based scores miss.","Because PROBAR relies only on a per-response adequacy judgement, the same recipe transfers across tasks, including next-word prediction over passage prefixes.","Fewer samples are sufficient for a usable PROBAR estimate, with the paper reporting that five samples behave similarly to ten, reducing the computational cost.","PROBAR can seed confidence-aware decoding: choosing an adequate sampled response instead of the greedy decoding improved precision in most settings tested."],"supporting_citations":[{"why":"Defines semantic entropy, the main baseline whose premise that variation implies error the paper disputes.","marker":"Kuhn et al. (2022b)"},{"why":"Supplies the improved semantic entropy estimator used for the SE baseline.","marker":"Aichberger et al. (2024)"},{"why":"Introduces P(True), which the paper adapts into the P(Adequate) baseline.","marker":"Kadavath et al. (2022)"},{"why":"The instruction-tuned model used as PROBAR's adequacy classifier and for NLI-based clustering in next-word prediction.","marker":"Mistral (2024)"},{"why":"Provides the LLM-as-judge protocol used to decide whether greedy responses are correct in QA datasets.","marker":"Lin et al. (2024b)"},{"why":"Supplies the OPT models whose responses are generated and scored.","marker":"Zhang et al. (2022)"},{"why":"Provides Abg-COQA, the reading-comprehension QA dataset with ambiguous and non-ambiguous prompts.","marker":"Guo et al. (2021)"},{"why":"Provides AmbigQA, the knowledge-based QA dataset used for ambiguous prompts.","marker":"Min et al. (2020)"},{"why":"Provides the Provo Corpus passage prefixes used for the next-word prediction task.","marker":"Luke and Christianson (2018)"}],"fun_headline_variants":["Semantic variety ≠ error: rely on answer adequacy for LLM confidence","For LLM reliability, count adequate answers, not semantic spread","PROBAR: better confidence by weighting adequate responses over variety","When LLMs vary, that's not always error—check adequacy instead","Adequacy beats variability as a signal of LLM error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated judge used to label responses as correct, and the human labels used for upper bounds, are accurate enough and independent of the errors made by PROBAR's adequacy classifier; if the same bias corrupts both the correctness labels and the adequacy labels, PROBAR's measured advantage over semantic entropy would be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Semantic variety ≠ error: rely on answer adequacy for LLM confidence","For LLM reliability, count adequate answers, not semantic spread","PROBAR: better confidence by weighting adequate responses over variety","When LLMs vary, that's not always error—check adequacy instead","Adequacy beats variability as a signal of LLM error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1454,"prompt_tokens":922,"completion_tokens":532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":442}},"tokens_in":538,"tokens_out":532,"duration_ms":5018,"temperature":1.0,"reasoning_tokens":442,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:10:43.597591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the QA AUROC comparisons with correctness labels produced by a second independent judge, such as two human annotators or an LLM from a different model family than the adequacy classifier, and check whether PROBAR still beats semantic entropy by the same margin; if the advantage collapses or reverses under the alternative judge, the central claim is not robust.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The instruction-tuned model used as PROBAR's adequacy classifier and for NLI-based clustering in next-word prediction."}],"review_version":1}