{"id":"4f01b3d7-ab18-4a71-9b37-7d0f27237ee3","arxiv_id":"2502.01126","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Relative pairwise confidence comparisons aggregated by rank aggregation produce more reliable confidence scores for language models than direct absolute confidence prompts.","lead":"This paper proposes estimating a language model's confidence by asking it to compare pairs of questions and rank which it is more confident about, then converting those preferences into scores with Elo or similar methods. Across five large models and 14 question-answering sets, the relative scores beat direct self-reported confidence by about 3.5 percentage points of selective classification AUC on average.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central claim rests on an unverified premise: pairwise confidence preferences encode correctness rather than surface cues. A direct preference-accuracy check would settle whether the method's mechanism actually works.","rationale":"The method has two stages: elicit pairwise preferences and aggregate them. The aggregation stage is mathematically standard; the novelty and the claimed advantage live entirely in the preference signal. If the preferences carry no correctness information, no rank aggregation method can produce a confidence ordering better than chance, and the AUC gains in Tables 1 and 3-6 must be due to noise, selection of the best rank-aggregation method per model, or dataset-specific surface artifacts. The paper's own ablations are suggestive but not decisive: the answer-removal experiment (Section 5) shows a 5.3% AUC drop when answers are omitted, consistent with preferences using the answer content, but it does not establish that the content is used to judge correctness rather than, say, answer verbosity or format. The CoT experiment also shows that preferences can be corrupted by hallucinated evidence, highlighting that the preference signal is not uniformly reliable. The reader's weakest assumption is exactly this preference-validity premise. My proposed test is the minimal direct check: compute the probability that a preferred question is the correctly answered one, conditional on the two answers disagreeing in correctness, and compare with chance and with a surface-feature predictor. This settles whether the mechanism exists before worrying about error bars or method selection. Since the reader already returned CONDITIONAL and my concern is the same premise, I recommend no change in verdict; the paper should be accepted only after this direct validation is reported.","tokens_in":16779,"tokens_out":7293,"duration_ms":79230,"concrete_test":"Regenerate or use the released preference data for one model (e.g., GPT-4o) on two datasets (e.g., GPQA and MedQA). For every pair (qi, qj) in which exactly one answer is correct, compute the proportion of times the model's chosen 'more confident' question is the correct one. Report this against 0.5 and against a surface-feature baseline (e.g., answer length, option position, first/second prompt position). Also measure the rate at which the model prefers the first-placed question to quantify position bias. If preference accuracy is not significantly above 0.5 or above the surface-feature baseline, the rank-aggregation improvements are not evidence for genuine confidence preferences; if it is, the core assumption holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central argument — that relative confidence estimation yields more reliable confidence scores than absolute estimation — requires that a model's pairwise confidence preference ('I am more confident in question i than j') is predictive of which answer is actually correct. This is the foundation of the data-generation stage (Section 4.1, Algorithm 1), and the paper never verifies it directly. The only evidence is the end-to-end selective-classification AUC on 250-example subsets (Tables 1, 3-6), which is an indirect aggregate: many noisy or biased preferences could still be rank-aggregated into a ranking that, by chance or due to confounds, improves AUC on this scale. Because the same model supplies both the answers and the preferences, there is no external check that preference judgments track knowledge rather than superficial features such as answer length, choice position, or prompt order; the paper does not report randomizing which question is presented first in the comparison prompt. The 'without answers' ablation (Section 5) shows preferences depend on seeing answers, but it does not show that the preferences are correct. Without a direct measurement of preference accuracy and bias, the observed AUC gains are not attributable to the proposed mechanism, so this is the load-bearing soft spot in the paper's central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes relative confidence estimation: rather than asking a language model to produce an absolute confidence score for a single answer, the model is asked to compare pairs of questions and indicate which question it is more confident in answering correctly. These pairwise preferences are converted into scalar confidence scores via rank aggregation methods (Elo, TrueSkill, Bradley-Terry). The authors evaluate the method on five LLMs (GPT-4, GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, Llama 3.1 405B) across 14 multiple-choice datasets, reporting that relative confidence estimation improves selective classification AUC by an average of 3.5% over direct confidence prompting and 1.7% over self-consistency prompting.","tokens_in":16952,"tokens_out":7493,"duration_ms":80923,"significance":"If the reported gains are reliable, this is a valuable contribution: it offers a training-free, log-probability-free way to obtain more discriminative confidence estimates, and framing confidence elicitation as relative comparison is an intuitive and potentially generalizable idea. The experimental breadth across five state-of-the-art models and 14 diverse tasks is a strength, and the method is clearly specified with prompts and hyperparameters documented. The evidence for the central claim is weakened, however, by the absence of statistical uncertainty quantification, by the selection of the best rank aggregation method per model based on test-set results, and by the lack of a direct validation that pairwise confidence preferences actually track answer correctness. The core idea is sound, but stronger empirical backing is needed to support the headline quantitative claim.","major_comments":[{"comment":"The central quantitative claim is not accompanied by any measure of uncertainty. Each dataset is evaluated on only 250 examples, and the reported differences between methods are often 0.1 to 2 percentage points in selective classification AUC, which is within the sampling noise expected for this sample size. The authors should report bootstrap confidence intervals or run multiple random 250-example subsets, and assess the statistical significance of the average gains before claiming consistent improvements.","section":"Section 5; Tables 1, 3-6"},{"comment":"The headline gains are computed by choosing, for each model, the rank aggregation method with the highest test-set average AUC (e.g., TrueSkill for most models, Bradley-Terry for Gemini 1.5 Pro, Elo for GPT-4o). This selection is made after seeing the test results and is not accounted for in the reported numbers. The authors should either select the aggregation method on a heldout set and report the corresponding gains, or fix a single method (e.g., TrueSkill) and report its performance; the abstract should not present test-selected best-case numbers without qualification.","section":"Section 5 ('report gains over the best relative confidence approach for each model')"},{"comment":"The method assumes that a model's pairwise confidence preference between two questions is informative about which answer is actually correct. This premise is never directly tested. The paper should report preference-level accuracy, e.g., the fraction of comparisons where the preferred question's answer is correct (and how this varies when both answers are correct, both wrong, or one is correct), and should check for prompt-order bias by randomizing which question appears first in the comparison prompt. The 'without answers' ablation in Section 5 changes the prompt from confidence comparison to difficulty comparison, so it does not isolate the effect of showing the model its own answers.","section":"Section 4.1; Algorithm 1"},{"comment":"The abstract states that relative confidence estimation 'consistently' provides more reliable confidence scores, but for Claude 3.5 Sonnet the method underperforms self-consistency by 0.1% (Table 4). The word 'consistently' is only accurate for the comparison to direct prompting; the 1.7% gain over self-consistency is an average across models that includes a negative result for one model. The claim should be rephrased to reflect average, not universal, improvement.","section":"Abstract; Section 5"}],"minor_comments":[{"comment":"The entry '2 .2%' in the Llama 3.1 column appears to be a typo for '2.2%'.","section":"Table 2"},{"comment":"It would be helpful to state explicitly which datasets were tuned on a heldout set and which used the fixed hyperparameters, rather than saying 'when available' without specifying.","section":"Appendix A.3"},{"comment":"The 'without answers' analysis is conducted only on GPT-4o; the paper should acknowledge that this is a single-model observation and discuss whether the conclusion may differ for other models.","section":"Section 5 ('How important are answers?')"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a clear and promising idea, but the abstract overstates the consistency of the results and the evaluation lacks statistical rigor. The authors should be encouraged to add uncertainty quantification, move the rank-aggregation method selection to a validation set, and directly validate that the pairwise preferences carry correctness signal. The work fits the journal's scope and, with these revisions, could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Sarah, this one is worth a look. The core idea—replace absolute confidence prompts with pairwise comparisons across questions and then feed the preferences into Elo/TrueSkill/Bradley-Terry—is genuinely new to me, and it works. Across five model families and fourteen datasets, relative confidence beats direct prompting on every model and beats self-consistency on four of five, with the only miss being Claude 3.5 Sonnet by a hair. The dataset-level tables are there, and the gains are not tiny: 3-6 points on average for Llama and GPT-4o. The paper is also well written and the method is specified precisely enough to reproduce.\n\nThe soft spots are real but not disqualifying. First, there are no error bars or seed variation anywhere. Each dataset is a 250-example subset, so a 1-2 point difference could be noise. The fact that the pattern is consistent across many datasets helps, but the authors should report variance. Second, they pick the best of the three rank aggregation methods per model after looking at the test-set averages. That inflates the headline numbers a bit. A proper heldout selection rule, like the one they use for hyperparameters, would fix it. Third, the abstract says 'consistently' but Claude is a counterexample at 0.1%, and self-consistency still wins on several commonsense datasets. The claim should be softened.\n\nThe stress test asks whether the pairwise preferences actually track correctness or just surface cues. The paper doesn't verify this directly, and it doesn't counterbalance question order in the prompt. But the end-to-end AUC is the real question for selective classification, and the gains there are consistent. A direct preference-accuracy analysis would make the paper stronger, but its absence doesn't sink the method.\n\nBottom line: this is a solid, novel empirical contribution. It deserves a serious referee, not a desk reject. The revisions I'd ask for: variance reporting, heldout selection of the aggregation method, and a toned-down abstract. Send it out.","headline":"A genuinely new and mostly convincing method for eliciting confidence from LLMs via pairwise preferences and rank aggregation, with some overstated claims that need tightening.","tokens_in":17536,"tokens_out":3020,"would_cite":true,"duration_ms":30545,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pairwise confidence comparisons outperform direct confidence scores for language models on multiple-choice tasks.","keywords":["relative confidence estimation","confidence preferences","pairwise comparison","rank aggregation","Elo rating","Bradley-Terry","selective classification","language models"],"falsifier":"Run the same relative-confidence pipeline on a dataset where the recorded preferences are deliberately inverted or randomized for a held-out subset of pairs; if selective classification AUC does not drop accordingly, the end-to-end gains are not caused by the preference signal. A more direct test is to measure, on pairs where exactly one of the model's two answers is correct, how often the model prefers the question with the correct answer; if this pair-level accuracy is at or below chance, the ranking mechanism has no correctness signal to aggregate.","tokens_in":16493,"feed_emoji":"🎯","tokens_out":3076,"duration_ms":32214,"temperature":0.7,"pith_summary":"The paper proposes that language models give more reliable confidence estimates when asked to compare two questions—'Which are you more confident in answering correctly?'—than when asked to rate confidence on a 0 to 1 scale. These pairwise preferences are aggregated with Elo, TrueSkill, or Bradley-Terry ranking into a confidence score per question. Across five large models and fourteen question-answering datasets, the relative method improves selective classification AUC by an average of 3.5% over direct absolute prompting and 1.7% over self-consistency prompting. The method needs no model internals, only the model's own answers and natural-language comparisons, so it applies to closed API models. A reader should care because better confidence estimates let users know when to trust a model's answer and when to defer to a human.","feed_headline":"Relative confidence beats absolute scoring in LLMs by 3.5% AUC","feed_subtitle":"Asking a model which of two questions it is surer about, then aggregating with Elo-style ratings, improves abstention accuracy.","key_machinery":"The central mechanism is the conversion of pairwise confidence preferences into scalar scores by rank aggregation, treating each question as a player in a tournament. The paper uses three algorithms: Elo rating, an iterative logistic update; TrueSkill, a Bayesian extension that tracks a skill distribution per question; and Bradley-Terry, a likelihood-based paired-comparison model. All three convert the model's 'I am more confident in question i than j' statements into scores that can be sorted to decide which answers to keep when abstaining on low-confidence items.","core_discovery":"Relative confidence estimation treats each question as a 'player' and the model's pairwise confidence judgments as match outcomes, then uses rank aggregation to assign scores. The paper reports that this consistently produces more discriminative confidence scores than absolute linguistic confidence estimation: across GPT-4, GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, and Llama 3.1 405B on 14 STEM, social science, and commonsense multiple-choice datasets, the best relative method beats direct prompting on selective classification AUC in all five models, with average gains of 3.5% over direct and 1.7% over self-consistency. The gains are largest for Llama 3.1 405B (6.1% over direct, 4.9% over self-consistency) and smallest for Claude 3.5 Sonnet, whose relative scores slightly underperform self-consistency. The paper also reports that scaling the number of comparisons per question from 5 to 15 increases the gains, and that removing the model's own answers from the comparison prompt drops AUC by 5.3%, showing the answers are an important part of the signal.","pith_inferences":["If the pairwise preference signal is genuinely tracking correctness, the same relative-comparison design could improve calibration of models' self-evaluations in generation tasks where token-level log probabilities are uninformative, an extension the paper mentions but does not test.","The paper's difficulty-without-answers experiment suggests that relative judgments partly reflect intrinsic question difficulty; that difficulty signal could be reused for curriculum learning or dataset filtering, though the paper only notes this as future work.","Because the preference data come from the same model being scored, the method's success depends on a model's introspective honesty; a model that is confidently wrong on the same pairs it ranks highly could undermine the gains, a boundary case the paper does not explore.","A direct pair-level test—does the model prefer the question it actually answers correctly more often than chance?—would separate the ranking signal from the aggregation machinery; the paper reports only the end-to-end AUC, not this intermediate diagnostic."],"forward_implications":["Users of closed API models can obtain better confidence signals without accessing log probabilities, needing only natural-language comparisons and answers.","Increasing the number of pairwise comparisons per question from 5 to 15 raises selective classification AUC, suggesting a compute-reliability trade-off that favors relative over self-consistency prompting at matched call counts.","The method extends naturally to rank aggregation settings beyond multiple-choice, such as scoring longform generations on different attributes, as discussed in the paper's future-work section.","Because preferences are relative, the approach sidesteps the coarse-grained score compression that plagues direct prompting, where models overuse a small set of values like 0.9.","The best rank aggregation method varies by model, but TrueSkill is recommended as a default since it matches or exceeds the other two for most models."],"supporting_citations":[{"why":"Supplies the direct confidence prompting baseline and the observation that absolute linguistic confidences are coarse-grained, which motivates the relative approach.","marker":"Shrivastava et al. (2023)"},{"why":"Supplies the self-consistency confidence prompting baseline and the post-processing aggregation procedure used for comparison.","marker":"Xiong et al. (2023)"},{"why":"Provides the Elo rating algorithm that converts pairwise preference outcomes into iterative score updates.","marker":"Elo (1978)"},{"why":"Provides the Bradley-Terry paired-comparison model used for likelihood-based rank aggregation.","marker":"Bradley & Terry (1952)"},{"why":"Provides TrueSkill, the Bayesian skill-rating algorithm that tracks uncertainty in each question's score.","marker":"Herbrich et al. (2006)"},{"why":"Motivates the relative-judgment design by showing language models are better at comparing multiple outputs than at verifying a single generation.","marker":"Kadavath et al. (2022)"}],"fun_headline_variants":["Relative confidence via pairwise judgments beats direct LLM scoring","Pairwise confidence beats direct scoring in LLMs by 3.5% AUC","Asking LLMs to compare questions improves confidence scores","Relative confidence ranking boosts LLM abstention accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model's pairwise preference between two questions—'which are you more confident in answering correctly?'—actually tracks which answer is more likely to be correct, rather than being a superficial or biased judgment.","fun_headline_variants_meta":{"raw":{"variants":["Relative confidence via pairwise judgments beats direct LLM scoring","Pairwise confidence beats direct scoring in LLMs by 3.5% AUC","Asking LLMs to compare questions improves confidence scores","Relative confidence ranking boosts LLM abstention accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000645,"raw_usage":{"total_tokens":3034,"prompt_tokens":1085,"completion_tokens":1949,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":701,"completion_tokens_details":{"reasoning_tokens":1881}},"tokens_in":701,"tokens_out":1949,"duration_ms":14402,"temperature":1.0,"reasoning_tokens":1881,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T16:29:15.593288+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same relative-confidence pipeline on a dataset where the recorded preferences are deliberately inverted or randomized for a held-out subset of pairs; if selective classification AUC does not drop accordingly, the end-to-end gains are not caused by the preference signal. A more direct test is to measure, on pairs where exactly one of the model's two answers is correct, how often the model prefers the question with the correct answer; if this pair-level accuracy is at or below chance, the ranking mechanism has no correctness signal to aggregate.","supporting_citations":[{"cited_title":"Minka, and Thore Graepel","cited_arxiv_id":null,"evidence_quote":"Provides TrueSkill, the Bayesian skill-rating algorithm that tracks uncertainty in each question's score."}],"review_version":1}