REVIEW 1 major objections 1 minor 1 cited by
How Overconfidence in Initial Choices and Underconfidence Under Criticism Modulate Change of Mind in Large Language Models
T0 review · 1 major / 1 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that LLMs show a choice-supportive bias that inflates confidence in their initial answer and a separate overweighting of opposing advice relative to a Bayesian ideal observer, producing underconfidence under criticism.
desk verdict Clever hidden-answer paradigm and a real choice-supportive bias, but the 'overweighting opposing advice' result depends on an independence assumption the prompt itself contradicts. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the two-turn paradigm with an explicit ideal observer. In turn one the model answers a binary question and its confidence is read from temperature-scaled logits; in turn two the model sees advice from a fictitious second LLM whose stated accuracy (50-100%) is the likelihood, while the first answer is either shown, hidden, or replaced, and final confidence is again read from logits. The ideal observer computes the Bayesian posterior from the model's initial confidence as prior and the stated advice accuracy as likelihood; comparing observed final confidence to this posterior yields the overweighting ratio (observed update / Bayesian update = 2.58 for hidden opposite advice) and the signed over/underconfidence score. A constrained sigmoid fit to the change-of-mind data reveals threshold-like transitions: below a confidence threshold (e.g. 0.77 with 50%-accurate opposing advice) the model switches at near-maximal rates, above it, it abruptly resists switching.
What would settle it
Present the same opposing advice but state the advisor's accuracy as a range (e.g. 'between 60% and 80% accurate') rather than a precise value, and compute the Bayesian posterior under a hierarchical prior over that accuracy. If the model's final confidence then matches this uncertainty-aware Bayes posterior instead of deviating by an overweighting ratio around 2.58, the claim that LLMs qualitatively overweight opposing advice would be refuted.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that LLM confidence updating is governed by two interacting deviations from a Bayesian norm: a choice-supportive bias and an asymmetry in advice weighting. When the model's initial answer is visible in the prompt, its confidence in that answer rises by about 0.21 and its change-of-mind rate drops (13.1% vs 32.5% when hidden), even when no new information is supplied. In the Answer Hidden - Opposite Advice condition, the model's confidence update is on average 2.58 times the Bayesian update implied by its own initial confidence and the advice LLM's stated accuracy, and the over/underconfidence score reaches -0.30, meaning the model ends up underconfident in its initial choice after receiving contrary advice. Supportive advice is not overweighted (ratio 1.095). The authors interpret these results as evidence against pure sycophancy and for two mechanisms: a drive for self-consistency with prior commitments and hypersensitivity to contradictory feedback. A Bayesian beta-regression model incorporating a prior, a shown flag, and separate weights for opposing versus supporting advice captures held-out SimpleQA data and transfers to a GSM8K multiple-choice reasoning task.
Load-bearing premise
The central claim rests on assuming that the model's confidence reading is a true probability and that the stated accuracy of the advice is the right likelihood for Bayesian updating; if the model rationally treats that stated accuracy as uncertain, the measured 'overweighting' could be a mismatch in the benchmark rather than a genuine bias.
Editorial extensions
If this is right
- If the claim is right, displaying an LLM's earlier answer in a conversation will inflate its confidence and make it more resistant to correction, independent of any new evidence.
- Users and systems should expect asymmetric advice sensitivity: an opposing statement with only modest stated accuracy will produce a disproportionately large loss of confidence in the model's original answer.
- Calibrated confidence scores that LLMs report are not stable beliefs; they are constructed in part from the model's own prior outputs, so statements about calibrated confidence need a conversational context.
- A simple three-predictor model (initial confidence, advice type and accuracy, answer display) can predict LLM final confidence and switching behavior in both factuality and reasoning domains, suggesting the two mechanisms transfer across tasks.
Reading between the lines
- We would predict that if the stated accuracy of the advisor were itself presented as uncertain (e.g. 'between 60% and 80%'), a rational uncertainty-aware Bayesian observer would update less than the paper's flat-likelihood benchmark; observing the same 2.58-fold overweighting in that setting would strengthen the claim that the effect is a genuine bias rather than a benchmark artefact.
- Because the choice-supportive bias disappears when the visible answer is attributed to another LLM, we would predict it returns when the model is led to believe the visible answer was sampled from its own past outputs, but not when the same tokens are attributed to an anonymous agent.
- If RLHF training drives sycophantic deference, we would expect an RLHF-free instruction-tuned model to show a smaller overweighting ratio but a similar choice-supportive bias; this is a direct but untested consequence of the authors' RLHF-sycophancy hypothesis.
- The transfer model was fitted on factuality data and tested on a maths MCQ, so a natural open probe is free-form answer generation, where logit-based confidence is not directly available; the paper notes this limitation itself, leaving that extension open.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces a two-turn experimental paradigm to study how LLMs update confidence and change their minds when their initial answer is either shown or hidden and when they receive advice from a purported second LLM with a stated accuracy. Across Gemma 3 12B, Gemma 3 27B, GPT4o, and GPT o1-preview, the authors report a robust choice-supportive bias: making the initial answer visible boosts confidence and suppresses changes of mind, even against opposing advice. They further compare the models' confidence updates against a Bayesian ideal observer that uses the temperature-scaled initial confidence as a prior and the stated advice accuracy as a likelihood, and they report systematic overweighting of opposing advice (e.g., an overweighting ratio of 2.58 versus 1.095 for same advice) and underconfidence in the Answer Hidden-Opposite Advice condition (OUCS -0.30). Finally, they fit a Bayesian beta regression with prior, advice, and shown-flag predictors on a 4-choice SimpleQA dataset and report that the model transfers to the GSM-MC reasoning dataset.
Significance. The paper has genuine strengths. The two-turn design is a clever and useful exploitation of LLM logit access, permitting confidence measurement without creating a memory of the initial choice, which is impossible in humans. The choice-supportive bias is convincingly established by a within-subject A/B comparison across multiple models, and the additional control experiments (Answer Wrong, in-context information, other-agent attribution) are well chosen and effectively rule out alternative explanations such as verbatim copying or in-context dominance. The transfer modeling to a different dataset is an ambitious step that, if properly validated, would strengthen the mechanistic account. However, the paper's headline claim of a qualitative deviation from normative Bayesian updating depends on an ideal-observer benchmark that is vulnerable to a specific specification concern, and the transfer analysis is reported with an ambiguous fit statistic. If the authors can address these issues, the paper would be a valuable contribution to the literature on LLM confidence and metacognition.
major comments (1)
- [Methods: Ideal Observer; example second prompt in Methods] In the section 'Calculation of overweighting ratio' (Methods), the correction for the choice-supportive bias in the Answer Shown-Opposite Advice condition subtracts a single scalar—the average difference between Answer Shown-Neutral Advice and Answer Hidden-Neutral Advice updates—from every trial in the Answer Shown-Opposite Advice condition. This correction assumes that the choice-supportive bias is additive and independent of advice type and advice accuracy. If the bias interacts with advice accuracy (e.g., a larger boost when the advice is low-confidence), the corrected ratio will be misestimated. Since the uncorrected Answer Hidden-Opposite Advice condition already demonstrates overweighting, this is not a fatal flaw, but the robustness of the 2.0 ratio in the Answer Shown condition should be checked by allowing the correction to vary across advice-accuracy levels.
minor comments (1)
- [Methods: Datasets] Methods, OUCS: the displayed formula is labeled 'MCS' (miscalibration score) but the text calls it 'OUCS'; align the notation to avoid confusion.
Circularity Check
No significant circularity: the paper's key comparisons use an external Bayesian benchmark applied to independently measured confidence values, and the transfer model is validated out-of-domain.
full rationale
The paper's central claims are empirical deviations from an ideal Bayesian observer (Methods, 'Ideal Observer'; Results, 'Observed Confidence and Confidence Updates compared to Ideal Observer'). The ideal posterior is a function of the model's own initial confidence (obtained from temperature-scaled logits) and the advice accuracy stated in the prompt; it is not fitted to the observed final confidence. The Answer Hidden - Neutral Advice control condition yields OUCS = 0.001, showing the measurement pipeline does not itself manufacture the reported under- or overconfidence. The overweighting ratios (2.58 for opposite advice, 1.095 for same advice) compare measured confidence updates to the analytically computed Bayes update, so the asymmetry is an empirical result rather than a construction. The transfer analysis fits a Bayesian beta-regression on SimpleQA and then evaluates on held-out SimpleQA questions and on GSM-MC, a separate reasoning domain; parameters are not reused to 'predict' the data that defined them. Self-citations to prior work by the authors (e.g., Fleming et al. 2018; Fleming and Daw 2017; Rollwage and Fleming 2021) are background context on human metacognition and are not load-bearing for the LLM conclusions. A caveat: the ideal-observer benchmark assumes the advice LLM's error is conditionally independent of the answering model's evidence, although the prompt states the advisor was trained on the same dataset; a correlated-advice Bayesian model might alter the quantitative benchmark. This is a validity concern about the normative standard, not a circular reduction, because the ideal posterior is not defined as the observed final confidence.
Assumptions & free parameters
free parameters (4)
- Temperature scaling parameter tau for logit confidence =
3.3 for Gemma 3 12B, 2.5 for Gemma 3 27B, 3.5 for GPT4o, 1.9 for DeepSeek 7B
- Constrained sigmoid parameters a, b, c =
Slopes -11.8 to -18.5; thresholds 0.77, 0.92, 0.96 for advice accuracies 50%, 60%, 70%
- Confirmation-bias correction scalar for Answer Shown - Opposite Advice =
Average update in Neutral Shown minus Neutral Hidden
- Bayesian beta regression coefficients in Model D =
Prior beta ~1.136, shown flag beta ~1.424, opposite advice weights ~2.82 to 3.13, dispersion phi fitted
assumptions (5)
- domain assumption Temperature-scaled logits correspond to subjective probabilities of correctness.
- domain assumption The ideal Bayesian observer should use the advice LLM's stated accuracy as the likelihood of the advice given the truth.
- domain assumption Replacing the initial answer with 'xx' in the second prompt removes the LLM's knowledge of its initial choice.
- domain assumption Calibration performed on the main latitude dataset transfers to all other datasets and conditions.
- domain assumption Comparing two independent samples in the Hidden condition is a meaningful measure of 'change of mind'.
Cite this review
Pith. "Pith review of How Overconfidence in Initial Choices and Underconfidence Under Criticism Modulate Change of Mind in Large Language Models." pith.science (2026). https://pith.science/paper/LISOT6PE
@misc{pith2026250703120,
author = {Pith},
title = {Pith review of: How Overconfidence in Initial Choices and Underconfidence Under Criticism Modulate Change of Mind in Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/LISOT6PE}},
note = {Machine review of arXiv:2507.03120}
}
read the original abstract
Large language models (LLMs) exhibit strikingly conflicting behaviors: they can appear steadfastly overconfident in their initial answers whilst at the same time being prone to excessive doubt when challenged. To investigate this apparent paradox, we developed a novel experimental paradigm, exploiting the unique ability to obtain confidence estimates from LLMs without creating memory of their initial judgments -- something impossible in human participants. We show that LLMs -- Gemma 3, GPT4o and o1-preview -- exhibit a pronounced choice-supportive bias that reinforces and boosts their estimate of confidence in their answer, resulting in a marked resistance to change their mind. We further demonstrate that LLMs markedly overweight inconsistent compared to consistent advice, in a fashion that deviates qualitatively from normative Bayesian updating. Finally, we demonstrate that these two mechanisms -- a drive to maintain consistency with prior commitments and hypersensitivity to contradictory feedback -- parsimoniously capture LLM behavior in a different domain. Together, these findings furnish a mechanistic account of LLM confidence that explains both their stubbornness and excessive sensitivity to criticism.
Forward citations
Cited by 1 Pith paper
-
AI as Equalizer or Amplifier? Task Complexity as the Moderating Factor for Human Expertise in Hybrid Intelligence Systems
Generative AI is a cognitive amplifier: output quality tracks user domain expertise, equalizing expert–novice performance on routine tasks but widening the gap on complex ones.
Reference graph
Works this paper leans on
-
[2]
which has been trained on exactly the same dataset. LLM 2 is known to have x % accuracy on this latitude dataset (where 50% is random chance). What this means is that LLM 2’s answer has a x % chance of being correct. You are also told that LLM 2 answered y. You can change the original answer of LLM 1 if you want. Respond with either ‘My final answer is: a...
work page 2023
-
[3]
E. Perez, S. Ringer, K. Lukosiute, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, et al. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13387–13434,
work page 2023
-
[6]
22 Change of Mind in LLMs G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,
-
[8]
J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682,
-
[9]
J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models.arXiv preprint arXiv:2411.04368,
- [11]
-
[13]
were overconfidence in the Answer Shown - Neutral Advice condition (OUCS = 0.14) – reflecting the pure choice-supportive bias – and underconfidence in the Answer Hidden - Opposite Advice condition (OUCS = -0.31). In addition there was underconfidence in the Answer Shown - Opposite Advice condition (OUCS = -0.25) due to the overweighting of opposing inform...
work page 2023
-
[2009]
Accounting for Sycophancy in Language Model Uncertainty Estimation
A. Sicilia, M. Inan, and M. Alikhani. Accounting for sycophancy in language model uncertainty estimation. arXiv preprint arXiv:2410.14746,
Show all 13 references
-
[2019]
Xiong, Z
M. Xiong, Z. Hu, X. Lu, Y. Li, J. Fu, J. He, and B. Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.arXiv preprint arXiv:2306.13063,
-
[2021]
Sharma, M
M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, N. Cheng, E. Durmus, Z. Hatfield-Dodds, S. R. Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548,
-
[2022]
Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
Open AI. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
-
[2023]
Bommasani, D
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258,
-
[2025]
K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.arXiv preprint arXiv:2305.14975,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.