Pith. sign in

REVIEW 4 major objections 3 minor 3 cited by

Language Models Prefer What They Know: Relative Confidence Estimation via Confidence Preferences

T0 review · 4 major / 3 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pairwise confidence comparisons outperform direct confidence scores for language models on multiple-choice tasks.

desk verdict A genuinely new and mostly convincing method for eliciting confidence from LLMs via pairwise preferences and rank aggregation, with some overstated claims that need tightening. read the letter →

arxiv 2502.01126 v1 pith:6OXAXF5W submitted 2025-02-03 cs.CL

Vaishnavi Shrivastava , Ananya Kumar , Percy Liang This is my paper · ORCID
classification cs.CL
keywords relativeconfidenceestimationpreferencespairwisecomparisonrankaggregationEloratingBradley-Terryselectiveclassificationlanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that language models give more reliable confidence estimates when asked to compare two questions—'Which are you more confident in answering correctly?'—than when asked to rate confidence on a 0 to 1 scale. These pairwise preferences are aggregated with Elo, TrueSkill, or Bradley-Terry ranking into a confidence score per question. Across five large models and fourteen question-answering datasets, the relative method improves selective classification AUC by an average of 3.5% over direct absolute prompting and 1.7% over self-consistency prompting. The method needs no model internals, only the model's own answers and natural-language comparisons, so it applies to closed API models. A reader should care because better confidence estimates let users know when to trust a model's answer and when to defer to a human.

What carries the argument

The central mechanism is the conversion of pairwise confidence preferences into scalar scores by rank aggregation, treating each question as a player in a tournament. The paper uses three algorithms: Elo rating, an iterative logistic update; TrueSkill, a Bayesian extension that tracks a skill distribution per question; and Bradley-Terry, a likelihood-based paired-comparison model. All three convert the model's 'I am more confident in question i than j' statements into scores that can be sorted to decide which answers to keep when abstaining on low-confidence items.

What would settle it

Run the same relative-confidence pipeline on a dataset where the recorded preferences are deliberately inverted or randomized for a held-out subset of pairs; if selective classification AUC does not drop accordingly, the end-to-end gains are not caused by the preference signal. A more direct test is to measure, on pairs where exactly one of the model's two answers is correct, how often the model prefers the question with the correct answer; if this pair-level accuracy is at or below chance, the ranking mechanism has no correctness signal to aggregate.

Watch

Extended reading notes

Core claim

Relative confidence estimation treats each question as a 'player' and the model's pairwise confidence judgments as match outcomes, then uses rank aggregation to assign scores. The paper reports that this consistently produces more discriminative confidence scores than absolute linguistic confidence estimation: across GPT-4, GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, and Llama 3.1 405B on 14 STEM, social science, and commonsense multiple-choice datasets, the best relative method beats direct prompting on selective classification AUC in all five models, with average gains of 3.5% over direct and 1.7% over self-consistency. The gains are largest for Llama 3.1 405B (6.1% over direct, 4.9% over self-consistency) and smallest for Claude 3.5 Sonnet, whose relative scores slightly underperform self-consistency. The paper also reports that scaling the number of comparisons per question from 5 to 15 increases the gains, and that removing the model's own answers from the comparison prompt drops AUC by 5.3%, showing the answers are an important part of the signal.

Load-bearing premise

The load-bearing premise is that a model's pairwise preference between two questions—'which are you more confident in answering correctly?'—actually tracks which answer is more likely to be correct, rather than being a superficial or biased judgment.

Editorial extensions

If this is right

  • Users of closed API models can obtain better confidence signals without accessing log probabilities, needing only natural-language comparisons and answers.
  • Increasing the number of pairwise comparisons per question from 5 to 15 raises selective classification AUC, suggesting a compute-reliability trade-off that favors relative over self-consistency prompting at matched call counts.
  • The method extends naturally to rank aggregation settings beyond multiple-choice, such as scoring longform generations on different attributes, as discussed in the paper's future-work section.
  • Because preferences are relative, the approach sidesteps the coarse-grained score compression that plagues direct prompting, where models overuse a small set of values like 0.9.
  • The best rank aggregation method varies by model, but TrueSkill is recommended as a default since it matches or exceeds the other two for most models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pairwise preference signal is genuinely tracking correctness, the same relative-comparison design could improve calibration of models' self-evaluations in generation tasks where token-level log probabilities are uninformative, an extension the paper mentions but does not test.
  • The paper's difficulty-without-answers experiment suggests that relative judgments partly reflect intrinsic question difficulty; that difficulty signal could be reused for curriculum learning or dataset filtering, though the paper only notes this as future work.
  • Because the preference data come from the same model being scored, the method's success depends on a model's introspective honesty; a model that is confidently wrong on the same pairs it ranks highly could undermine the gains, a boundary case the paper does not explore.
  • A direct pair-level test—does the model prefer the question it actually answers correctly more often than chance?—would separate the ranking signal from the aggregation machinery; the paper reports only the end-to-end AUC, not this intermediate diagnostic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes relative confidence estimation: rather than asking a language model to produce an absolute confidence score for a single answer, the model is asked to compare pairs of questions and indicate which question it is more confident in answering correctly. These pairwise preferences are converted into scalar confidence scores via rank aggregation methods (Elo, TrueSkill, Bradley-Terry). The authors evaluate the method on five LLMs (GPT-4, GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, Llama 3.1 405B) across 14 multiple-choice datasets, reporting that relative confidence estimation improves selective classification AUC by an average of 3.5% over direct confidence prompting and 1.7% over self-consistency prompting.

Significance. If the reported gains are reliable, this is a valuable contribution: it offers a training-free, log-probability-free way to obtain more discriminative confidence estimates, and framing confidence elicitation as relative comparison is an intuitive and potentially generalizable idea. The experimental breadth across five state-of-the-art models and 14 diverse tasks is a strength, and the method is clearly specified with prompts and hyperparameters documented. The evidence for the central claim is weakened, however, by the absence of statistical uncertainty quantification, by the selection of the best rank aggregation method per model based on test-set results, and by the lack of a direct validation that pairwise confidence preferences actually track answer correctness. The core idea is sound, but stronger empirical backing is needed to support the headline quantitative claim.

major comments (4)
  1. [Section 5; Tables 1, 3-6] The central quantitative claim is not accompanied by any measure of uncertainty. Each dataset is evaluated on only 250 examples, and the reported differences between methods are often 0.1 to 2 percentage points in selective classification AUC, which is within the sampling noise expected for this sample size. The authors should report bootstrap confidence intervals or run multiple random 250-example subsets, and assess the statistical significance of the average gains before claiming consistent improvements.
  2. [Section 5 ('report gains over the best relative confidence approach for each model')] The headline gains are computed by choosing, for each model, the rank aggregation method with the highest test-set average AUC (e.g., TrueSkill for most models, Bradley-Terry for Gemini 1.5 Pro, Elo for GPT-4o). This selection is made after seeing the test results and is not accounted for in the reported numbers. The authors should either select the aggregation method on a heldout set and report the corresponding gains, or fix a single method (e.g., TrueSkill) and report its performance; the abstract should not present test-selected best-case numbers without qualification.
  3. [Section 4.1; Algorithm 1] The method assumes that a model's pairwise confidence preference between two questions is informative about which answer is actually correct. This premise is never directly tested. The paper should report preference-level accuracy, e.g., the fraction of comparisons where the preferred question's answer is correct (and how this varies when both answers are correct, both wrong, or one is correct), and should check for prompt-order bias by randomizing which question appears first in the comparison prompt. The 'without answers' ablation in Section 5 changes the prompt from confidence comparison to difficulty comparison, so it does not isolate the effect of showing the model its own answers.
  4. [Abstract; Section 5] The abstract states that relative confidence estimation 'consistently' provides more reliable confidence scores, but for Claude 3.5 Sonnet the method underperforms self-consistency by 0.1% (Table 4). The word 'consistently' is only accurate for the comparison to direct prompting; the 1.7% gain over self-consistency is an average across models that includes a negative result for one model. The claim should be rephrased to reflect average, not universal, improvement.
minor comments (3)
  1. [Table 2] The entry '2 .2%' in the Llama 3.1 column appears to be a typo for '2.2%'.
  2. [Appendix A.3] It would be helpful to state explicitly which datasets were tuned on a heldout set and which used the fixed hyperparameters, rather than saying 'when available' without specifying.
  3. [Section 5 ('How important are answers?')] The 'without answers' analysis is conducted only on GPT-4o; the paper should acknowledge that this is a single-model observation and discuss whether the conclusion may differ for other models.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: confidence preferences are aggregated into scores and evaluated against external correctness labels.

full rationale

The paper's relative confidence scores are constructed from pairwise preference judgments (Algorithm 1) via rank aggregation (Elo, TrueSkill, Bradley-Terry, Section 4.2). Correctness labels enter only in evaluation (AUC/AUROC), not in score construction; there is no equation in which the predicted confidence score is defined in terms of the correctness label or vice versa. The claim that relative preferences beat absolute estimates is an empirical comparison on held-out test examples, not a consequence of a fitted parameter. Self-citations to Shrivastava et al. (2023) motivate the direct-prompt baseline and supply the prompt, but the baseline is re-run and compared fairly; the central result does not reduce to that citation. The per-model choice of best aggregation method is a selection-bias concern, not a circularity, because the chosen method's scores are still evaluated against external labels. The unverified premise that pairwise confidence preferences carry correctness signal is a validity limitation (the paper tests it only indirectly via AUC), but it is not circularity: even if the premise were false, the derivation chain would remain non-circular, merely unsupported.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method has several hand-set or tuned numeric hyperparameters (Elo K and iterations, TrueSkill priors, Bradley-Terry lambda and iterations, number of comparisons per question) and relies on domain assumptions about the informativeness of LM self-preference, the validity of rank aggregation, and the representativeness of the evaluation sample. It introduces no new entities. The main non-parameter choice is selecting a different rank aggregation algorithm per model after seeing test results, which is closer to selection bias than to a fitted parameter.

free parameters (6)
  • Elo K-factor = 400
    Controls sensitivity of Elo rating updates; fixed by hand, not tuned (Appendix A.3).
  • Elo number of iterations = 1 default; tuned 1-20 on heldout set
    Number of passes over preference data; tuned per task to maximize heldout AUC.
  • TrueSkill priors (mu, sigma, beta, tau) = mu=25; sigma=mu/3; beta=mu/6; tau=mu/300; sigma/beta/tau tuned
    Hand-set scale and tuned relative parameters for the Bayesian skill model.
  • Bradley-Terry L2 regularization lambda = 0.01
    Regularization strength for maximum likelihood estimation; fixed by hand.
  • Bradley-Terry max iterations = 5 default; tuned 1-20
    Optimization iterations for the MLE objective; tuned per task on heldout AUC.
  • Number of pairwise comparisons per question (n) = 15 in main results; 5, 10, 15 in scaling study
    Determines preference data size and compute; larger n improves gains in Table 2.
assumptions (4)
  • domain assumption A model's pairwise preference between two questions is an informative signal about which of its answers is more likely correct.
    Central premise of the method; validated only indirectly through downstream AUC, with no error bars or external preference-quality checks (Section 4.1, Figure 3).
  • domain assumption Rank aggregation methods (Elo, TrueSkill, Bradley-Terry) produce a meaningful total ordering from noisy, inconsistent, incomplete pairwise preferences.
    The paper relies on this standard property of these methods but does not independently verify it in the LM preference setting (Section 4.2).
  • domain assumption The 250-example subsets of the 14 tasks are representative enough to support the reported average AUC differences.
    No sampling error or seed variability is reported, so the representativeness of the test subsets is assumed (Section 2, Tables 1-6).
  • domain assumption AUC computed with Gaussian tie-breaking is a valid and sufficient metric for comparing confidence methods.
    Standard ranking-quality metric, but the paper deliberately sets aside calibration, so the claim is only about ranking quality (Section 2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Language Models Prefer What They Know: Relative Confidence Estimation via Confidence Preferences." pith.science (2026). https://pith.science/paper/6OXAXF5W

@misc{pith2026250201126,
  author       = {Pith},
  title        = {Pith review of: Language Models Prefer What They Know: Relative Confidence Estimation via Confidence Preferences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6OXAXF5W}},
  note         = {Machine review of arXiv:2502.01126}
}
read the original abstract

Language models (LMs) should provide reliable confidence estimates to help users detect mistakes in their outputs and defer to human experts when necessary. Asking a language model to assess its confidence ("Score your confidence from 0-1.") is a natural way of evaluating its uncertainty. However, models struggle to provide absolute assessments of confidence (i.e. judging confidence in answering a question independent of other questions) and the coarse-grained scores they produce are not useful for evaluating the correctness of their answers. We propose relative confidence estimation, where we match up questions against each other and ask the model to make relative judgments of confidence ("Which question are you more confident in answering correctly?"). Treating each question as a "player" in a series of matchups against other questions and the model's preferences as match outcomes, we can use rank aggregation methods like Elo rating and Bradley-Terry to translate the model's confidence preferences into confidence scores. We evaluate relative confidence estimation against absolute confidence estimation and self-consistency confidence methods on five state-of-the-art LMs -- GPT-4, GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, and Llama 3.1 405B -- across 14 challenging STEM, social science, and commonsense reasoning question answering tasks. Our results demonstrate that relative confidence estimation consistently provides more reliable confidence scores than absolute confidence estimation, with average gains of 3.5% in selective classification AUC over direct absolute confidence estimation methods and 1.7% over self-consistency approaches across all models and datasets.

Figures

Figures reproduced from arXiv: 2502.01126 by the authors.

Figure 1
Figure 1. Relative Confidence Estimation. We first prompt models to elicit their answers to different questions. For each question qi , we match qi with n other questions qj and generate confidence preference data. We ask the model to compare its level of confidence in the pair of questions and decide which question it is more confident in answering correctly. We treat the questions and answers as “players” in these matchups … view at source ↗
Figure 2
Figure 2. Direct Confidence Prompt Instruction. Asks the model to directly score its confidence in its answer to a question. C(x, yˆ) ∈ [0, 1] is the model’s confidence in yˆ(x) be￾ing the correct output for x. Our goal is to derive reli￾able confidence estimates from language models—i.e. higher C(x, yˆ) where R(y, y ˆ ) is 1 and lower C(x, yˆ) where R(y, y ˆ ) is 0. Reliable confidence estimates can help prioritize high-conf… view at source ↗
Figure 3
Figure 3. Relative Confidence Prompt. Asks model to compare its confidence in two questions. Additionally, we also report the AUROC (Hendrycks & Gimpel, 2017; Xiong et al., 2023), area under the receiver operating characteristic curve, in Ap￾pendix A.2. AUROC is a standard classification met￾ric used to measure how well a model can separate correct and incorrect examples at different thresh￾olds. In our setting, we use the ou… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Selective Classification AUC Across Models. For each model, we plot the selective classification AUC averaged across the 14 tasks for each confidence estimation method. The absolute confidence estimation baselines—direct prompting (Direct) and self-consistency (Hybrid …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MARS-RA: Rank Aggregation for Credit Assignment via Multimodal Comparisons in Embodied Multi-Agent Cooperation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Credit assignment via LMM pairwise comparisons plus Bradley–Terry rank aggregation and potential-based shaping improves cooperative MARL under sparse rewards and dynamic agent counts.

  2. Generalization of Fine-Tuned Uncertainty Communication and Metacognition in Large Language Models

    cs.CL 2025-09 conditional novelty 6.0 of 10

    Fine-tuning LLMs to verbalize consistency-based confidence improves calibration and discrimination on held-out and new domains, but single-task gains do not transfer between confidence-rating and pairwise comparison t...

  3. Reasoning Strategies in Large Language Models: Can They Follow, Prefer, and Optimize?

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Prompting LLMs with distinct reasoning strategies and ensembling their outputs improves accuracy on logical deduction tasks, though not as consistently as the paper claims.

Reference graph

Works this paper leans on

55 extracted references · 24 canonical work pages · cited by 3 Pith papers

  1. [1]

    Claude 3.5 Sonnet

    Anthropic . Claude 3.5 Sonnet . 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet

    work page 2024

  2. [2]

    Kenneth J. Arrow. Social Choice and Individual Values. Yale University Press, 1951. URL https://yalebooks.yale.edu/book/9780300179316/social-choice-and-individual-values/

    arXiv 1951

  3. [3]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  4. [4]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39: 0 324, 1952. URL https://api.semanticscholar.org/CorpusID:125209808

    1952

  5. [5]

    Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017. URL https://api.semanticscholar.org/CorpusID:4787508

    arXiv 2017

  6. [6]

    DeGroot and Stephen E

    Morris H. DeGroot and Stephen E. Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society. Series D (The Statistician), 32: 0 12--22, 1983

    work page 1983

  7. [7]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  8. [8]

    Sivakumar

    Cynthia Dwork, Ravi Kumar, Moni Naor, and D. Sivakumar. Rank aggregation methods for the web. In The Web Conference, 2001. URL https://api.semanticscholar.org/CorpusID:8393813

    work page 2001

Show all 55 references
  1. [9]

    On the foundations of noise-free selective classification

    Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research (JMLR), 11, 2010

    2010

  2. [10]

    The Rating of Chessplayers, Past and Present

    Arpad E Elo. The Rating of Chessplayers, Past and Present. Arco Pub., New York, 1978. URL https://www.amazon.com/Rating-Chess-Players-Past-Present/dp/0668047216

    1978

  3. [11]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. ArXiv, abs/2402.01306, 2024. URL https://api.semanticscholar.org/CorpusID:267406810

    2024 arXiv

  4. [12]

    Selective prediction-set models with coverage guarantees

    Jean Feng, Arjun Sondhi, Jessica Perry, and Noah Simon. Selective prediction-set models with coverage guarantees. arXiv preprint arXiv:1906.05473, 2019

    1906 arXiv

  5. [13]

    Gptscore: Evaluate as you desire

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. In North American Chapter of the Association for Computational Linguistics, 2023. URL https://api.semanticscholar.org/CorpusID:256662188

    2023

  6. [14]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. ArXiv, abs/1706.04599, 2017 a . URL https://api.semanticscholar.org/CorpusID:28671436

    2017 arXiv

  7. [15]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), pp.\ 1321--1330, 2017 b

    2017

  8. [16]

    A baseline for detecting misclassified and out-of-distribution examples in neural networks

    Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations (ICLR), 2017

    2017

  9. [17]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021 a

    2021

  10. [18]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv, abs/2103.03874, 2021 b . URL https://api.semanticscholar.org/CorpusID:232134851

    2021 arXiv

  11. [19]

    Minka, and Thore Graepel

    Ralf Herbrich, Thomas P. Minka, and Thore Graepel. Trueskilltm: A bayesian skill rating system. In Neural Information Processing Systems, 2006. URL https://api.semanticscholar.org/CorpusID:9744799

    2006

  12. [20]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. In arXiv preprint arXiv:2009.13081, 2021

    2009 arXiv

  13. [21]

    Selective classification can magnify disparities across groups

    Erik Jones, Shiori Sagawa, Pang Wei Koh, Ananya Kumar, and Percy Liang. Selective classification can magnify disparities across groups. In International Conference on Learning Representations (ICLR), 2021

    2021

  14. [22]

    Language models (mostly) know what they know, 2022

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav For...

    2022

  15. [23]

    Kemeny and J

    John G. Kemeny and J. Laurie Snell. Mathematical Models in the Social Sciences. The MIT Press, 1978. URL https://mitpress.mit.edu/9780262610308/mathematical-models-in-the-social-sciences/

    1978

  16. [24]

    Unanimous prediction for 100\ In Association for Computational Linguistics (ACL), 2016

    Fereshte Khani, Martin Rinard, and Percy Liang. Unanimous prediction for 100\ In Association for Computational Linguistics (ACL), 2016

    2016

  17. [25]

    Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, D. Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Dr...

    2022 arXiv

  18. [26]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021

    2021 arXiv

  19. [27]

    Teaching models to express their uncertainty in words

    Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=8s8K2UZGTZ

    2022

  20. [28]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with s...

    2023 arXiv

  21. [29]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Empirical Methods in Natural Language Processing (EMNLP), 2018

    2018

  22. [30]

    Murphy and Robert L

    Allan H. Murphy and Robert L. Winkler. Reliability of subjective probability forecasts of precipitation and temperature. Journal of the Royal Statistical Society. Series C (Applied Statistics), 26: 0 41--47, 1977

    1977

  23. [31]

    Cooper, and Milos Hauskrecht

    Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Binary classifier calibration: Non-parametric approach. arXiv, 2014

    2014

  24. [32]

    Cooper, and Milos Hauskrecht

    Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence., 2015: 0 2901--2907, 2015. URL https://api.semanticscholar.org/CorpusID:6292807

    2015

  25. [33]

    Negahban, Sewoong Oh, and Devavrat Shah

    Sahand N. Negahban, Sewoong Oh, and Devavrat Shah. Rank centrality: Ranking from pairwise comparisons. Oper. Res., 65: 0 266--287, 2012. URL https://api.semanticscholar.org/CorpusID:3602049

    2012

  26. [34]

    Introducing chatgpt

    OpenAI . Introducing chatgpt. 2022. URL https://openai.com/index/chatgpt/

    2022

  27. [35]

    OpenAI . GPT-4 . 2023. URL https://openai.com/index/gpt-4-research/

    2023

  28. [36]

    OpenAI . GPT-4o . 2024. URL https://openai.com/index/hello-gpt-4o/

    2024

  29. [37]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan...

    2022 arXiv

  30. [38]

    The pagerank citation ranking : Bringing order to the web

    Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking : Bringing order to the web. In The Web Conference, 1999. URL https://api.semanticscholar.org/CorpusID:1508503

    1999

  31. [39]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. ArXiv, abs/2305.18290, 2023. URL https://api.semanticscholar.org/CorpusID:258959321

    2023 arXiv

  32. [40]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. ArXiv, abs/2311.12022, 2023. URL https://api.semanticscholar.org/CorpusID:265295009

    2023 arXiv

  33. [41]

    Socialiqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  34. [42]

    Llamas know what gpts don't show: Surrogate models for confidence estimation

    Vaishnavi Shrivastava, Percy Liang, and Ananya Kumar. Llamas know what gpts don't show: Surrogate models for confidence estimation. ArXiv, abs/2311.08877, 2023. URL https://api.semanticscholar.org/CorpusID:265213392

    2023 arXiv

  35. [43]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In North American Association for Computational Linguistics (NAACL), 2019

    2019

  36. [44]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  37. [45]

    Manning, and Chelsea Finn

    Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn. Fine-tuning language models for factuality. ArXiv, abs/2311.08401, 2023. URL https://api.semanticscholar.org/CorpusID:265158181

    2023 arXiv

  38. [46]

    Nicolaus Tideman

    T. Nicolaus Tideman. Independence of clones as a criterion for voting rules. Social Choice and Welfare, 4: 0 185--206, 1987. URL https://api.semanticscholar.org/CorpusID:122758840

    1987

  39. [47]

    Minimum weighted feedback arc sets for ranking from pairwise comparisons

    Soroush Vahidi and Ioannis Koutis. Minimum weighted feedback arc sets for ranking from pairwise comparisons. ArXiv, abs/2412.16181, 2024. URL https://api.semanticscholar.org/CorpusID:274982240

    2024

  40. [48]

    Chi, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ArXiv, abs/2203.11171, 2022. URL https://api.semanticscholar.org/CorpusID:247595263

    2022 arXiv

  41. [49]

    Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063, 2023. URL https://arxiv.org/pdf/2306.13063.pdf

    2023 arXiv

  42. [50]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv, abs/2305.10601, 2023. URL https://api.semanticscholar.org/CorpusID:258762525

    2023 arXiv

  43. [51]

    Self-rewarding language models

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. ArXiv, abs/2401.10020, 2024. URL https://api.semanticscholar.org/CorpusID:267035293

    2024 arXiv

  44. [52]

    Xing, Haotong Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv, abs/2306.05685, 2023. URL http...

    2023 arXiv

  45. [53]

    Navigating the grey area: Expressions of overconfidence and uncertainty in language models, 2023

    Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: Expressions of overconfidence and uncertainty in language models, 2023

    2023

  46. [54]

    Ziegler, Nisan Stiennon, Jeff Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, abs/1909.08593, 2019. URL https://api.semanticscholar.org/CorpusID:202660943

    1909 arXiv

  47. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.