Evaluating and Calibrating LLM Confidence on Questions with Multiple Correct Answers

Keping Bi; Shiyu Ni; Yuanzi Li; Yuhan Wang; Zhikai Ding; Zihang Zhan

REVIEW 4 major objections 6 minor 1 cited by

Evaluating and Calibrating LLM Confidence on Questions with Multiple Correct Answers

T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Summing response probabilities fixes LLM confidence calibration

desk verdict Useful benchmark and a plausible cardinality effect, but SCA's calibration claim needs ECE and the Math domain construction is broken. read the letter →

arxiv 2602.07842 v2 pith:ZAH67EVV submitted 2026-02-08 cs.CL

Yuhan Wang , Shiyu Ni , Zhikai Ding , Zihang Zhan , Yuanzi Li , Keping Bi This is my paper · ORCID

classification cs.CL

keywords confidencecalibrationlargelanguagemodelsmultiplecorrectanswerssemanticconsistencytoken-levelprobabilitiesbenchmarkMACEaggregation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models' confidence estimates are usually calibrated on questions with exactly one right answer. This paper shows that when a question has several correct answers, almost all existing training-free calibration methods systematically underestimate confidence: the model knows the answer but gives different valid answers across samples, and consistency-based methods read that disagreement as uncertainty. The paper builds a benchmark (MACE) with 12,000 questions that have 1, 2, 4, or 6 verified correct answers, and measures 15 calibration methods across four LLM families. It reports a paradox: accuracy rises with answer count while confidence falls, so in realistic mixed settings calibration collapses. The proposed fix, SCA, sums token-level probabilities over multiple sampled responses instead of trusting the single most confident response, and the paper reports that this restores state-of-the-art calibration in mixed settings while keeping single-answer behavior.

What carries the argument

SCA (Semantic Confidence Aggregation): sample N responses, optionally group them into semantic clusters, and take the question's confidence as the sum of token-level sequence generation probabilities over all clusters (or all responses when threshold τ=0). The key move is aggregating absolute probabilities rather than normalizing or counting agreement; absolute token probabilities already assign low weight to spurious answers, so the total reflects the model's belief across all valid answers instead of punishing disagreement. The other machinery is MACE, a benchmark built from Wikidata triplets and rule-synthesized math questions with exactly 1, 2, 4, or 6 ground-truth answers, which makes t

What would settle it

Compute confidence on a new set of questions where answer cardinality is known by construction (e.g., generated from a database with explicit answer sets) and check whether the accuracy/confidence divergence and SCA's advantage persist; alternatively, re-annotate a random sample of MACE to see whether any 1a/2a/4a/6a questions actually have a different number of valid answers, and if the mismatch is large enough, the reported trends would disappear.

Watch

Extended reading notes

Core claim

The central discovery is that consistency-based confidence estimates conflate answer-space diversity with model uncertainty. On questions with multiple valid answers, a model that samples several different but equally correct answers appears to 'disagree with itself,' producing low confidence scores indistinguishable from genuine doubt. The paper shows this failure extends beyond consistency methods to token-probability and verbalized methods, and that larger models suffer sharper confidence drops because they know a wider set of correct answers and therefore alternate among more of them. The correction proposed is SCA: sum the per-token generation probabilities of all sampled responses (opt

Load-bearing premise

The MACE benchmark's ground-truth answer sets are complete and exactly 1, 2, 4, or 6 correct answers; if any answer list is missing valid answers—especially in the Math domain, whose construction is described inconsistently (uniform sampling vs interval templates)—the measured decline in confidence and the gains from SCA could be artifacts rather than properties of multi-answer questions.

Editorial extensions

If this is right

In real-world deployments where questions can have several valid answers, existing confidence scores (consistency, verbalized, token-probability) systematically under-report certainty, which can mislead hallucination detection and retrieval-augmentation triggers.
Larger models are more susceptible to this underestimation because they know more correct answers; calibration comparisons between model sizes that ignore answer cardinality will be biased.
Simple summation of token-level probabilities across samples gives strong calibration without clustering or threshold tuning, meaning the correction is cheap to compute.
SCA preserves single-answer calibration while fixing mixed-answer settings, so it can replace existing training-free estimators without sacrificing previous gains.
Confidence on multi-answer questions should be interpreted as a sum over valid alternatives, not as the probability of one specific answer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same accuracy–confidence divergence should appear in other 'multiple acceptable output' settings, such as open-ended QA, code generation, and summarization; a testable extension is whether SCA-style aggregation improves calibration for tasks with a set of acceptable outputs rather than a single reference.
The threshold-free result suggests that token-level probability mass is already concentrated on 'the set of things the model is willing to say'; summing absolute probabilities effectively computes a lower bound on the model's total belief over its answer space. One could test whether a closed-form Bayesian posterior over answer clusters reproduces SCA without sampling.
If answer sets in MACE are incomplete, the observed accuracy-increase/confidence-decrease pattern would be confounded by label noise; a stress test that re-annotates a random subset and measures robustness would clarify how much of the effect is real.
The 'larger models drop more' result implies that scaling up a model can worsen calibration on open questions even as accuracy improves; a practical consequence is that calibration targets should be conditioned on expected answer-set size, not just on question type.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

What you should know first: the paper's central observation—LLMs get more accurate but less confident as the number of valid answers grows—is clearly documented across many tables and models, and it's a point worth taking seriously. The MACE benchmark is a real contribution, even if it ships with one ugly seam.

The genuinely new thing is the systematic study: a 12k-question benchmark across six domains with 1/2/4/6 ground-truth answers, 15 training-free calibration methods, four model families. The finding that consistency-based methods over-penalize multi-answer questions, and that larger models show a sharper confidence drop (because they know more valid answers), is well supported by the knowledge-coverage analysis. SCA itself is simple—sum token-level probabilities of sampled responses—but at τ=0 it does beat the baselines on AUROC in mixed settings, and the fact that it needs no tuning is nice.

Now the soft spots, in order of importance.

First, SCA's "state-of-the-art calibration" claim rests entirely on AUROC. AUROC is a discrimination metric, not a calibration metric. Table 18 reports ECE for every baseline but SCA is absent. That's a real gap; the method may simply produce scores that rank well, and we have no evidence it is calibrated on a probability scale. This is fixable—normalize the sum and report ECE—but it is load-bearing.

Second, the Math domain construction is internally inconsistent. Appendix A.5 says queries are generated by uniformly sampling numbers from [0,1,000,000], but Table 5's template asks for "a number between lower and upper," and §3.5 claims exact answer counts are guaranteed. No rule connects the sampled number to an interval that yields exactly 1, 2, 4, or 6 primes/squares/cubes/etc. If Math cardinalities are uncontrolled, the cardinality sweep and SCA evaluation have an unmeasured label-noise problem in one of six domains. The other five domains rest on Wikidata plus manual verification (κ=0.94), so I'd expect the trend to survive, but the paper needs to reconcile the appendix or the benchmark can't be reconstructed.

Third, the conclusion says "all these methods collapse" under mixing, but the ECE table shows several baselines improving (e.g., Prob Entropy 26.2→19.4). The overstatement is minor, but it undersells a more nuanced story.

Fourth, the benchmark is not released, so the empirical claims can't be independently verified yet.

So: the paper deserves a serious referee. The central phenomenon is likely real, the benchmark is useful, and the flaws are addressable. I'd send it to review with a request for ECE on SCA, a rewritten Math-construction section, and a release commitment. For a reading group it's a solid choice for anyone working on calibration.

Referee Report

4 major / 6 minor

Summary. This paper introduces MACE, a 12,000-question benchmark with 1/2/4/6 correct answers across six factual domains, and uses it to evaluate 15 training-free confidence calibration methods on four LLM families. The main empirical claim is that as the number of valid answers increases, QA accuracy rises while estimated confidence falls, causing miscalibration on mixed answer-count questions. The paper proposes Semantic Confidence Aggregation (SCA), which sums token-level generation probabilities over sampled responses (optionally filtered by a threshold), and reports that SCA achieves the best AUROC on mixed-answer settings while remaining competitive on single-answer questions.

Significance. The paper targets a real and understudied problem: existing calibration methods are designed for single-answer QA, and multi-answer questions create a systematic mismatch between consistency-based uncertainty and correctness. The trend documented in Table 3 is nontrivial and potentially useful; the benchmark, if carefully constructed, would be a valuable resource. A particular strength is that the main SCA result at tau=0 has no fitted parameters, so the comparison is not a tuning artifact; the authors also include a wide set of baselines and model families. However, the current evidence is not yet sufficient for the paper's strongest claims: the Math domain construction is internally inconsistent, and SCA is evaluated only by AUROC, not by probability-scale calibration, so the 'state-of-the-art calibration' claim is not demonstrated.

major comments (4)

[Appendix A.5 vs. Table 5 / Section 3.4] The Math-domain construction is internally inconsistent and undermines the cardinality control that drives both main results. Section 3.4 and Table 5 promise questions of the form 'Name one {number type} number between {lower_num} and {upper_num}' with exactly 1, 2, 4, or 6 correct answers, but Appendix A.5 states that 'we do not construct natural questions explicitly' and instead 'generate queries by uniformly sampling numbers from the interval [0,1,000,000]'. No rule is given for converting a sampled number into an interval containing exactly the labeled number of primes/squares/cubes/Fibonacci/triangular numbers; a random interval in [0,10^6] will typically contain far more than 6 such numbers. As written, the Math answer-count labels are not reproducible, so the observed accuracy/confidence trends and SCA gains may be driven by label noise or uncontrolled difficulty. Please provide t
[Section 6.3, Table 4, Table 18, Eq. (1)-(2)] SCA is claimed to 'achieve state-of-the-art calibration performance', but the paper reports only AUROC for SCA. AUROC is a rank-based discrimination metric and is invariant to monotone transformations of the confidence score; it does not measure whether confidence equals accuracy on the probability scale. Moreover, Eq. (2) sums absolute token-level probabilities over sampled responses and is not normalized to [0,1]; depending on N and duplicate responses, the score can exceed 1. No ECE or reliability diagram is reported for SCA in Table 18 (which lists ECE only for the 15 baselines). Please add ECE/reliability analysis for SCA (and ideally for the other models) and specify how the raw sum is rescaled; otherwise the central contribution is a ranking heuristic, not a calibrated confidence.
[Section 6.3, Table 4, Appendix D.1] The SCA advantage over existing methods is small and is not accompanied by significance testing. On LLaMA-3.1-70B, SCA(tau=0) AUROC is 80.5 vs. 80.9 for Perplexity on [1], 81.2 vs. 80.3 on [1,2], 79.2 vs. 78.1 on [1,2,4], and 76.7 vs. 75.3 on [1,2,4,6]; on Qwen-2.5-72B the pattern is similar. Thus on the single-answer setting SCA is actually below the best baseline, and the mixed-setting gains are within 1-2 AUROC points. No confidence intervals, multiple runs, or paired significance tests are reported, and SCA is evaluated on only two of the model families used elsewhere. Please provide variability estimates and test the mixed-setting advantage against the best baseline.
[Section 4.1, Appendix B, Tables 3/8-15] The central trend that 'estimated confidence decreases' requires a common confidence scale, but the paper never defines how the entropy/perplexity quantities in Appendix B are converted to the 'Confidence' percentages in Tables 3 and 8-15. Equations (3), (4), (7), and (12) define uncertainties U, not confidences C; without the transformation, the reported 1a->6a declines for Prob Entropy, N-Prob Entropy, Sem Entropy, and Perplexity cannot be interpreted. Please state the explicit monotone mapping used (or, if raw values are reported, correct the tables), since this mapping is load-bearing for the paper's first finding.

minor comments (6)

[Section 5.2] Typo: 'exisiting' should be 'existing'.
[Figure 3] The x-axis labels '[1]', '[1,2]', etc. are not defined in the caption. Please explain that they denote mixed sets of answer counts.
[Section 1 / Dataset links] The placeholders '/githubCode' and 'Datasets' appear without URLs. Please provide the actual code and data links.
[References] Several references are malformed or incomplete (e.g., 'Joue-An Hu and 1 others', 'Hurst et al.' with truncated author lists). Use a consistent, complete reference format.
[Appendix D.1] The text says results for other models are reported in Appendix D.1, but only Qwen-2.5-72B is actually shown. Clarify which models were tested and where the remaining results are.
[Appendix C.2] Minor formatting issues: 'over70%' should read 'over 70%'; also the threshold discussion would benefit from a reference to the figure or table that shows the cluster probability distribution.

Circularity Check

2 steps flagged · score 2.0 of 10

Central claims are empirically self-contained; only minor definitional and post-hoc elements appear in the consistency-mechanism framing and the knowledge-coverage threshold.

self definitional [Introduction (para. 2); Appendix §B.1 Eq. (10); Table 3]
"Response-consistency–based approaches rely on the implicit assumption that higher agreement among generated responses indicates greater correctness. While this assumption often holds for single-answer questions, it breaks down for questions with multiple valid answers... disagreement among equally correct responses can lead to low confidence estimates, even when the model possesses the relevant knowledge. ... Consistency score measures the proportion of model samples that agree with the aggregated prediction ˜Y among M sampled outputs: C_consistency = 1/M Σ I{ ˆYi = ˜Y}"

For consistency-based methods, the reported confidence collapse on multi-answer questions (Table 3: Consis 51.3 at 1a to 35.9 at 6a) is largely a restatement of the metric's definition: consistency is defined as the fraction of samples agreeing with the aggregate answer, so whenever several answers are valid and the model alternates among them, the score must drop unless the model always emits the same answer. The paper itself names this mechanism, so the framed 'discovery' is partially forced by construction. This is not load-bearing for the full claim, since the accuracy rise and confidence decline are also shown for token-probability and verbalized methods, which are not definitionally tied to agreement.
other [§5.3 (Definition of Knowledge Coverage); Appendix §C.2]
"Specifically, we apply a token-level probability–based filter and retain only clusters whose cumulative probability exceeds a threshold of τ = 0.1. ... Motivated by the empirical distribution, we retain only clusters whose cumulative probability satisfies P(C_m) ≥ 0.1."

The τ=0.1 threshold that defines 'knowledge coverage' is selected from the same cluster-probability distribution it is then used to characterize, and the same token-level probabilities underlie both the coverage measure and the confidence estimates being 'explained.' The §5.3 explanation of the sharper confidence decline in larger models (broader knowledge coverage → wider answer diversity → lower consistency) therefore partly restates the observations from which the threshold and clusters were derived. This is an auxiliary analysis, not the central calibration claim, and the SCA results do not depend on it.

full rationale

The paper's core contributions—the MACE benchmark, the observation that accuracy rises while estimated confidence falls as ground-truth cardinality increases, and the SCA calibration method—are supported by held-out empirical evaluation rather than by derivation from their own outputs. SCA is evaluated on a test set distinct from the development set, and the paper shows τ=0 matches or nearly matches the dev-tuned optimum (Table 4), so the central result does not hinge on the fitted threshold; the tuned-τ reporting is therefore not a 'fitted input called prediction.' The self-citations (Ni et al. 2024, 2025a, 2025b) are contextual and non-load-bearing. The two mild circular elements are: (1) for consistency-based methods, low confidence under multiple valid answers is a built-in consequence of the agreement metric (Eq. 10), though the paper explicitly acknowledges this mechanism and the finding extends to non-consistency methods; and (2) the knowledge-coverage threshold τ=0.1 in §5.3 is chosen post hoc from the very distribution it summarizes, making that explanatory analysis partially restate its input. Neither element forces the central claims, which would stand or fall on benchmark quality and test-set AUROC. The internal inconsistency in MACE Math construction (Table 5 interval template vs. §A.5 uniform sampling) is a ground-truth-quality/correctness risk, not circularity, and is outside this pass's scope.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claims draw heavily on the MACE benchmark, whose validity is assumed rather than externally verified. SCA itself introduces no free constants beyond sampling count and threshold, but its probability-sum interpretation is built on the assumption that token probabilities are comparable across sequences of different lengths and answer sets.

free parameters (4)

Sampling count N = 20
All consistency-based methods and SCA sample N=20 responses (Figure 1, Appendix B); no sensitivity analysis over N is reported, and SCA's sum depends on N.
SCA threshold τ (non-zero variant) = 0 recommended; 0.05 (LLaMA-70B dev), 0.3 (Qwen-72B dev) when tuned
Tuned on the 20% MACE dev split; the τ=0 variant avoids tuning, but τ=0 is itself a hand-chosen simplification.
Knowledge-coverage cluster threshold τ = 0.1
Used in §5.3 to define high-confidence clusters and explain why larger models' confidence drops more; chosen post hoc from the cluster probability distribution (Appendix C.2).
K-means sampling cluster count k = 500
Used in benchmark construction to select 500 questions per domain/answer-count setting (§A.5); affects MACE composition and difficulty.

assumptions (6)

domain assumption Wikidata relationships are complete and correct for the chosen domains (award winners, office holders, subdivisions, rivers, languages).
MACE ground-truth answer sets are derived from Wikidata SPARQL queries (§3.2) plus heuristic filtering; if Wikidata omits valid answers, answer counts are wrong.
domain assumption Human annotators can certify exact answer cardinality (1, 2, 4, 6) via manual verification.
§3.3 reports κ=0.94, but completeness of open factual sets such as all administrative subdivisions of a country is assumed, and only small random samples are spot-checked.
domain assumption GPT-4o is a valid judge of answer correctness and semantic consistency.
Used as the primary evaluator (§4.3) for all correctness and consistency labels; human agreement on 300 instances is high (κ=0.92), but the judge is still an API model.
domain assumption Token-level generation probability p(s) is a meaningful confidence signal when summed across sampled responses.
SCA's core mechanism (§6.1) assumes absolute sequence probabilities carry the model's perceived correctness; unnormalized sums are not proven to be calibrated.
domain assumption Questions with different answer counts are comparable in difficulty except for the number of valid answers.
§3.5 claims questions differ only in constraints such as year range; if 4a/6a questions are systematically easier, the accuracy increase is partly confounded.
ad hoc to paper Math questions can be generated with exactly 1, 2, 4, or 6 correct answers.
§A.5 says Math queries are generated by uniformly sampling numbers without explaining how fixed answer counts are enforced, despite Table 5's interval template.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating and Calibrating LLM Confidence on Questions with Multiple Correct Answers." pith.science (2026). https://pith.science/paper/ZAH67EVV

@misc{pith2026260207842,
  author       = {Pith},
  title        = {Pith review of: Evaluating and Calibrating LLM Confidence on Questions with Multiple Correct Answers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZAH67EVV}},
  note         = {Machine review of arXiv:2602.07842}
}

read the original abstract

Confidence calibration is essential for making large language models (LLMs) reliable, yet existing training-free methods have been primarily studied under single-answer question answering. In this paper, we show that these methods break down in the presence of multiple valid answers, where disagreement among equally correct responses leads to systematic underestimation of confidence. To enable a systematic study of this phenomenon, we introduce MACE, a benchmark of 12,000 factual questions spanning six domains with varying numbers of correct answers. Experiments across 15 representative calibration methods and four LLM families (7B-72B) reveal that while accuracy increases with answer cardinality, estimated confidence consistently decreases, causing severe miscalibration for questions with mixed answer counts. To address this issue, we propose Semantic Confidence Aggregation (SCA), which aggregates confidence over multiple high-probability sampled responses. SCA achieves state-of-the-art calibration performance under mixed-answer settings while preserving strong calibration on single-answer questions.

Figures

Figures reproduced from arXiv: 2602.07842 by the authors.

**Figure 2.** The MACE benchmark construction pipeline, illustrated using the Honorable Award domain. ①② Valid triplet (e.g., Pritzker Prize winners) are retained. ③ Low-popularity triplet (e.g., obscure awards like the Ig Nobel Prize) are removed via Popularity Filter. ④ Noisy triplet containing invalid formats (e.g., URLs) are removed via a Validity Filter. ⑤ Factually incorrect triplet are identified and removed during the Man… view at source ↗

**Figure 3.** AUROC of LLaMA-3.1-70B-Instruct on mixed questions with varying counts of correct answers. veals a consistent downward trend in confidence across most estimation methods. This indicates that miscalibration between confidence and QA performance arises in multi-answer settings not only for response-consistency-based methods, but for nearly all existing approaches. Across different domains, the overall trend is simila… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: QA performance and confidence variation with the number of correct answers across model scales. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Knowledge coverage across ground-truth (GT) set sizes. Different color schemes denote different model families. Color intensity increases from light to dark to indicate increasing model size. 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0 1 2 3 4 5 6 7 Proportion of clus…

**Figure 6.** Figure 6: Distribution of the cluster sizes after filtering. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 8.** Figure 8: Sensitivity of average AUROC across answer [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 9.** Figure 9: Domain-specific accuracy and confidence under different ground-truth sizes. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Can LLM Rerankers Predict Their Own Ranking Performance?
cs.IR 2026-06 unverdicted novelty 7.0 of 10

LLM rerankers can internally predict ranking quality via self-consistency of sampled outputs, matching SOTA external QPP while direct confidence is overconfident; supervised token-efficient methods improve calibration.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith · cited by 1 Pith paper

[2023]

InProceedings of EMNLP 2023, pages 530–543

Selectively answering ambiguous questions. InProceedings of EMNLP 2023, pages 530–543. DeepSeek-AI. 2024. Deepseek-v3: Scaling open-source language models with dense and mixture-of-experts architecture.arXiv preprint arXiv:2412.19437. Shrey Desai and Greg Durrett. 2020. Calibra- tion of pre-trained transformers.arXiv preprint arXiv:2003.07892. Yanai Elaza...

arXiv 2023
[2024]

is officially awarded to

When do llms need retrieval augmentation? mitigating llms’ overconfidence helps retrieval aug- mentation.arXiv preprint arXiv:2402.11457. Shiyu Ni, Keping Bi, Jiafeng Guo, Minghao Tang, Jing- tong Wu, Zengxin Han, and Xueqi Cheng. 2025a. Annotation-efficient universal honesty alignment. arXiv preprint arXiv:2510.17509. Shiyu Ni, Keping Bi, Jiafeng Guo, Lu...

arXiv 2023

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2023]

InProceedings of EMNLP 2023, pages 530–543

Selectively answering ambiguous questions. InProceedings of EMNLP 2023, pages 530–543. DeepSeek-AI. 2024. Deepseek-v3: Scaling open-source language models with dense and mixture-of-experts architecture.arXiv preprint arXiv:2412.19437. Shrey Desai and Greg Durrett. 2020. Calibra- tion of pre-trained transformers.arXiv preprint arXiv:2003.07892. Yanai Elaza...

arXiv 2023

[2] [2024]

is officially awarded to

When do llms need retrieval augmentation? mitigating llms’ overconfidence helps retrieval aug- mentation.arXiv preprint arXiv:2402.11457. Shiyu Ni, Keping Bi, Jiafeng Guo, Minghao Tang, Jing- tong Wu, Zengxin Han, and Xueqi Cheng. 2025a. Annotation-efficient universal honesty alignment. arXiv preprint arXiv:2510.17509. Shiyu Ni, Keping Bi, Jiafeng Guo, Lu...

arXiv 2023