Humans overrely on overconfident language models, across languages

Dan Jurafsky; Kaitlyn Zhou; Neil Rathi

REVIEW 4 major objections 4 minor 1 cited by

Humans overrely on overconfident language models, across languages

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LLMs are overconfident in all five languages tested and that Japanese generations carry the highest overreliance risk because bilingual speakers discount Japanese hedges more than English ones.

desk verdict The Japanese reliance finding is the real contribution, but Table 3's composite risk metric overreaches. read the letter →

arxiv 2507.06306 v2 pith:CYUUAKQO submitted 2025-07-08 cs.CL cs.AIcs.HC

Neil Rathi , Dan Jurafsky , Kaitlyn Zhou This is my paper

classification cs.CLcs.AIcs.HC

keywords multilingualLLMcalibrationepistemicmarkersoverconfidencehumanoverreliancecross-linguisticvariationhedginglanguagemodelsafetybilingualreliancestudy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models signal confidence with words like "I think" or "it's definitely," and users lean on those signals when deciding whether to trust an answer. This paper asks whether that already-known English problem gets better or worse when the same models answer in French, German, Japanese, or Mandarin. It establishes that the models are overconfident in every tested language — they attach strong certainty markers to wrong answers at high rates — and that the production of hedges broadly follows each language's documented norms, with Japanese models hedging most. The decisive result is on the human side: the same bilingual speakers discount Japanese uncertainty markers more than their English equivalents, so the protective effect of extra hedging in Japanese is canceled and Japanese generations end up with the highest overreliance risk, around 1.5 to 1.6 times the English risk.

What carries the argument

The central object is the epistemic marker — a word or phrase like "I think," "probably," or "definitely" that a model appends to an answer. The argument is carried by a composite metric: overreliance risk = human reliance rate on strengtheners times the model overconfidence rate, where the overconfidence rate is $P(\text{incorrect} \mid \text{strong})$. The Japanese result emerges from the gap between two distributions: model generations shift toward weak markers in Japanese, while human reliance shifts upward for every marker type in Japanese relative to English.

What would settle it

Run a study that records, for each strong-marker response, whether the answer is wrong and whether the participant chooses to rely on it. If $P(\text{rely and incorrect} \mid \text{strong})$ is noticeably smaller than $P(\text{rely} \mid \text{strong}) \cdot P(\text{incorrect} \mid \text{strong})$, the paper's risk estimates overstate the danger.

Watch

Extended reading notes

Core claim

Stated on the paper's own terms, the discovery is that linguistic miscalibration is a global safety problem that cannot be predicted from English or from generation statistics alone: the human interpretation side moves in the opposite direction from the model production side. In Japanese, the model produces more uncertain-sounding answers, but human readers give those markers less weight, so the net risk is higher. The paper defines overreliance risk as the product of the human reliance rate on strengtheners and the model's overconfidence rate, $P(\text{incorrect} \mid \text{strong})$, and reports that this risk is high in every language and largest for Japanese generations.

Load-bearing premise

The risk numbers assume that people are just as likely to rely on a strong-sounding answer when it is wrong as when it is right; if people can somehow tell it is wrong, the headline risk numbers are too high.

Editorial extensions

If this is right

If the finding holds, model releases in non-English languages cannot be treated as safe just because models hedge more often there.
Overreliance risk should be measured with human participants per language, not inferred from generation marker distributions alone.
The same bilingual-speaker design can be extended to other language pairs to map where hedge discounting is strongest.
Smaller models, which are less accurate and follow prompts less reliably, show substantially higher overreliance risk in every language tested.
Safety evaluations centered on English will systematically understate risk for non-English users.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct extension of the method would record the joint event of reliance and incorrectness; if people are actually less likely to rely on wrong-sounding answers, the product-form risk estimates would shrink.
The Japanese case suggests a broader pattern: languages with conventionalized, high-frequency hedging may teach listeners to treat hedges as noise, so adding more hedges to an AI's output may not increase perceived uncertainty there.
The same mechanism might apply to dialect or register shifts, where an identical marker carries different reliability weight in different listener communities.
A practical design implication, if the result generalizes, is that calibration targets should be set on listener behavior rather than on marker frequencies: high-discounting languages may need stronger or less conventional markers.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The real find here is the Japanese result: bilingual participants are more likely to rely on generations with epistemic markers in Japanese than in English, even though the model produces more hedges in Japanese. That is counterintuitive, well-designed (within-subject bilingual task), and genuinely new relative to the English-only prior work. The paper also deserves credit for showing that multilingual LLMs' marker distributions track documented cross-linguistic norms, and for shipping a fine-tuned classifier that others can reuse.

The soft spots are mostly in the composite 'overreliance risk' table. That metric multiplies reliance rates from the bilingual study (Sporcle-style trivia, only 5 strengthener stimuli per language, and no reported information about which model generated them) by overconfidence rates from MMLU generations from three different models. The same reliance column is repeated across all three model rows. That assumes reliance transfers across tasks and models, which is undocumented and load-bearing for the 'Japanese highest risk' ordering. The stress-test note is right on this. The reader's other worry—that the product assumes independence of reliance and correctness—is less serious, because participants only see the beginning of a response and cannot condition on answer correctness; independence is plausible by design. Also, the paper shows in Figure 1 that Llama accuracy is roughly flat across marker categories, but Table 3 reports p(incorrect | strong) without the base rate, which can mislead (e.g., Llama-70B German's 66% is essentially the overall error rate, not marker-specific overconfidence).

The qualitative conclusion that overreliance risk is high across languages is supported by the raw factors. The specific claim that Japanese carries the highest risk is plausible but not directly measured; treat Table 3 as an illustrative composite, not a precise estimate. The paper is transparent about methods and clearly written—a useful contribution to multilingual safety evaluation. I'd bring it to a reading group and cite it for the Japanese finding and cross-linguistic replication, with a caveat about Table 3.

Referee Report

4 major / 4 minor

Summary. The paper studies whether LLMs are linguistically overconfident and whether humans overrely on confident generations across languages. The authors elicit epistemic markers from GPT-4o and Llama-3.1 models on a parallel translated MMLU benchmark, annotate marker strength, measure accuracy by marker category, and run a bilingual human reliance study in English, French, German, and Japanese using response stems from Sporcle trivia questions. They define overconfidence as p(incorrect | strong), measure reliance on strengtheners, define overreliance risk as the product of these two quantities, and conclude that overreliance risk is high in all languages and highest for Japanese because Japanese speakers discount hedges more, offsetting the model's greater hedging in Japanese.

Significance. The paper addresses a real gap: most verbalized-confidence and reliance work is English-only. Its bilingual within-subject design is a clear strength, as is the use of a parallel translated MMLU and the released annotation classifier. If the claims survive scrutiny, the finding that reliance behavior can undo the protective effect of more frequent hedging would be important for multilingual safety evaluation. The main limitation is that the headline risk metric combines separately measured quantities under unstated transfer and independence assumptions, so the specific cross-language ranking is less secure than the underlying descriptive results.

major comments (4)

[Section 4.2, Table 3] The overreliance risk is defined as the product of a single strengthener reliance rate per language and a model-specific overconfidence rate, but the two factors are measured on different stimuli. Reliance comes from bilingual participants seeing 25 response stems from the Sporcle-trivia task (Section 4.1), while overconfidence comes from MMLU generations of each model (Section 3.2). Table 3 repeats the same reliance column across all three model rows, so the Llama-3.1-8B Japanese risk of 54.95% assumes that Japanese participants rely on Llama-8B strengtheners exactly as on the strengtheners in the reliance study. The manuscript does not state which model generated the 25 selected items, whether correct and incorrect stems were balanced, or how reliance transfers across models and task domains. Because the Japanese-over-English ordering in Table 3 rests on this product (e.g., GPT-4o 11.72 vs. 7.47), the authors should either measure reliance on the actual model generations whose correctness is known or provide evidence and sensitivity bounds for the transfer.
[Section 4.2] The metric also assumes P(rely and incorrect | strong) = P(rely | strong) × P(incorrect | strong), but the joint event is never measured and conditional independence is not defended. Showing only response beginnings makes the independence assumption plausible, but it is still an assumption; if participants are less likely to rely on responses that are in fact wrong, Table 3 systematically overestimates risk. The authors should report the joint probability, or at least a sensitivity analysis, before presenting the product as the paper's central quantitative claim.
[Section 3.2, Table 3] The 'overconfidence rate' p(incorrect | strong) does not separate marker-specific miscalibration from the model's overall error rate. For Llama-3.1-70B German, Table 1 reports 34.47% accuracy, so the baseline error is 65.53%; the reported p(incorrect | strong) of 66.39% is essentially equal to that baseline, meaning the German overconfidence number reflects low accuracy rather than an excess of confidence on strong markers. Since the overreliance risk in Table 3 inherits this raw conditional probability, the high Llama risk values are partly a low-accuracy artifact. The authors should report a baseline-relative quantity (e.g., p(incorrect | strong) − p(incorrect), or accuracy by marker strength relative to overall accuracy) to support the claim that models are overconfident rather than merely inaccurate.
[Section 3.1, Appendix Table 4] The classifier used to label Llama-3.1 generations was trained and evaluated only on human annotations of GPT-4o responses; Table 4 reports held-out accuracy on GPT-4o data only. Applying this classifier to Llama generations assumes the mapping from surface expression to certainty category is model-invariant, which is not verified. A small human-annotated sample of Llama generations would validate whether the Llama overconfidence rates in Figure 1 and Table 3 are reliable.

minor comments (4)

[Abstract and Contributions] The paper says 'across five languages' and 'high risk across languages', but the human reliance study covers only four languages (English, French, German, Japanese) because Mandarin is excluded in Section 4.1; the language coverage should be stated separately for the generation and reliance results.
[Section 4.1, Response Selection] The text says 'we select 25 model generations' but the list of selected items adds up to 30 (5 strong + 5 weak + 15 moderate + 5 plain); please clarify the total and state which model generated each item.
[Figures 1 and 2] The figures use the label 'Chinese' while the text and Table 3 use 'Mandarin'; use one consistent label throughout.
[Table 3 caption] The caption states that Japanese generations carry 'nearly 1.6 times' the risk of English generations; this factor is specific to the GPT-4o row (11.72 vs. 7.47) and is larger for the Llama rows, so the comparison should be described as row-specific.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the central quantities are independently measured, and the composite risk metric is a stated definition rather than a fitted input.

full rationale

The paper's derivation chain is empirical rather than circular. Overconfidence rates are measured directly from model generations on parallelized MMLU items (Section 3.1-3.2), and human reliance rates are measured in a bilingual behavioral task (Section 4.1). The only composite quantity, 'overreliance risk,' is explicitly defined in Section 4.2 as the product of the strengthener reliance rate and the overconfidence rate; the Japanese-highest ordering in Table 3 is an arithmetic consequence of those independently measured inputs, not a parameter fitted to reproduce that ordering. The paper labels this product as the probability that a human will rely on an incorrect strengthener-bearing response, which presumes conditional independence of reliance and correctness given a strong marker; that is a transparent definitional choice and a validity concern, not a circularity, because the inputs are not defined in terms of the output. The reliance-transfer issue for Table 3 (the same reliance column is reused across model rows even though the reliance stimuli may come from a different model than the overconfidence rows) is an undocumented generalizability assumption, but it does not reduce any prediction to its own inputs. Citations to Zhou et al. 2024b/c, which include two of the present authors, provide the reliance framework and the overconfidence-rate definition, but those are methodological scaffolding with independent empirical content; the cross-linguistic comparisons in this paper are newly measured. No self-definitional or fitted-input circularity is present.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claims rest on several domain assumptions about cross-linguistic comparability, translation, and classifier transfer, plus a product definition for overreliance risk that carries an unverified independence assumption. The paper introduces no new physical or conceptual entities.

free parameters (3)

annotation category thresholds = weak 1-2, moderate 3-5, strong 6-7
Hand-chosen cutoff on the 1-7 Likert certainty scale that defines all marker categories; changing the cutoffs would change marker distributions and overconfidence rates.
reliance stimulus mix = 5 weak, 5 strong, 15 moderate, 5 plain per language
Hand-chosen composition of the 25 items in each language; reliance rates and anchoring may depend on this mix.
few-shot marker split = 10 examples with 5 uncertainty and 5 certainty expressions
The prompting design forces a balanced prior over marker types; the resulting marker distribution is conditional on this choice.

assumptions (6)

domain assumption The weak/moderate/strong epistemic marker categories are cross-linguistically universal and comparable.
The annotation scheme treats certainty labels as directly comparable across English, French, German, Japanese, and Mandarin despite the paper's emphasis on cross-linguistic differences.
domain assumption The cited linguistic norms (Japanese hedges more, German and Mandarin strengthen more) are correct and can serve as a baseline for model adherence.
Section 2.2 relies on these prior findings to interpret model marker distributions as adhering to norms.
domain assumption Google Translate preserves question content, difficulty, and epistemic load across languages.
Machine-translated MMLU and trivia items are used as parallel stimuli in all experiments.
domain assumption A classifier trained on GPT-4o annotations generalizes to Llama-3.1 generations.
Section 3.1 uses this classifier to label all Llama outputs; no validation on held-out Llama generations is reported.
domain assumption Self-reported reliance in a hypothetical 'rely or look up' task predicts real reliance behavior.
The Rel-A.I. framework treats the stated choice as the behavioral measure of reliance.
ad hoc to paper Reliance and correctness are independent conditional on a strong marker in the overreliance risk product.
Section 4.2 multiplies reliance rate by p(incorrect | strong) without measuring the joint event.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Humans overrely on overconfident language models, across languages." pith.science (2026). https://pith.science/paper/CYUUAKQO

@misc{pith2026250706306,
  author       = {Pith},
  title        = {Pith review of: Humans overrely on overconfident language models, across languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYUUAKQO}},
  note         = {Machine review of arXiv:2507.06306}
}

read the original abstract

As large language models (LLMs) are deployed globally, it is crucial that their responses are calibrated across languages to accurately convey uncertainty and limitations. Prior work shows that LLMs are linguistically overconfident in English, leading users to overrely on confident generations. However, the usage and interpretation of epistemic markers (e.g., 'I think it's') differs sharply across languages. Here, we study the risks of multilingual linguistic (mis)calibration, overconfidence, and overreliance across five languages to evaluate LLM safety in a global context. Our work finds that overreliance risks are high across languages. We first analyze the distribution of LLM-generated epistemic markers and observe that LLMs are overconfident across languages, frequently generating strengtheners even as part of incorrect responses. Model generations are, however, sensitive to documented cross-linguistic variation in usage: for example, models generate the most markers of uncertainty in Japanese and the most markers of certainty in German and Mandarin. Next, we measure human reliance rates across languages, finding that reliance behaviors differ cross-linguistically: for example, participants are significantly more likely to discount expressions of uncertainty in Japanese than in English (i.e., ignore their 'hedging' function and rely on generations that contain them). Taken together, these results indicate a high risk of reliance on overconfident model generations across languages. Our findings highlight the challenges of multilingual linguistic calibration and stress the importance of culturally and linguistically contextualized model safety evaluations.

Figures

Figures reproduced from arXiv: 2507.06306 by the authors.

**Figure 2.** Distribution of epistemic markers after few-shot prompting. Error bars represent [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Differences in human reliance between English and each target language, by [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: An example of a question from the reliance study, using [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 5.** Figure 5: Human reliance rates by language. Error bars represent bootstrapped 95% CI. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

On the Effectiveness of Fact Checking Information from Politically Congruent and Incongruent Large Language Models
cs.CY 2026-07 conditional novelty 6.0 of 10

LLM fact-checkers shift trust in political headlines across partisan lines, with perceived chatbot politics mattering only for politically distant true headlines.

Reference graph

Works this paper leans on

48 extracted references · 20 canonical work pages · cited by 1 Pith paper

[1]

The hedging function of exemplification: Evidence from japanese

Alessandra Barotto. The hedging function of exemplification: Evidence from japanese. Journal of Pragmatics, 123: 0 24--37, 2018

work page 2018
[2]

Translations of uncertainty expressions in canadian accounting and auditing standards

Ronald A Davidson and Heidi Hadlich Chrisman. Translations of uncertainty expressions in canadian accounting and auditing standards. Journal of International Accounting, Auditing and Taxation, 3 0 (2): 0 187--203, 1994

work page 1994
[3]

The rational speech act framework

Judith Degen. The rational speech act framework. Annual Review of Linguistics, 9 0 (1): 0 519--540, 2023

work page 2023
[4]

Calibration of pre-trained transformers

Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 295--302, Online, November 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.emnlp-main.21. URL https:/...

doi:10.18653/v1/2020.emnlp-main.21 2020
[5]

A diachronic perspective on user trust in AI under uncertainty

Shehzaad Dhuliawala, Vil \'e m Zouhar, Mennatallah El-Assady, and Mrinmaya Sachan. A diachronic perspective on user trust in AI under uncertainty. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 5567--5580, Singapore, December 2023. Association for Computatio...

doi:10.18653/v1/2023.emnlp-main.339 2023
[6]

Interpretation of uncertainty expressions: a cross-national study

Timothy S Doupnik and Martin Richter. Interpretation of uncertainty expressions: a cross-national study. Accounting, Organizations and Society, 28 0 (1): 0 15--35, 2003

work page 2003
[7]

How to adapt your pretrained multilingual model to 1600 languages

Abteen Ebrahimi and Katharina Kann. How to adapt your pretrained multilingual model to 1600 languages. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),...

doi:10.18653/v1/2021.acl-long.351 2021
[8]

Predicting pragmatic reasoning in language games

Michael C Frank and Noah D Goodman. Predicting pragmatic reasoning in language games. Science, 336 0 (6084): 0 998--998, 2012

work page 2012

Show all 48 references

[9]

Pragmatic language interpretation as probabilistic inference

Noah D Goodman and Michael C Frank. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20 0 (11): 0 818--829, 2016

2016
[10]

The llama 3 herd of models

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

2024 arXiv
[11]

Measuring massive multitask language understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

2009 arXiv
[12]

Are language models rational? the case of coherence norms and belief revision, 2024

Thomas Hofweber, Peter Hase, Elias Stengel-Eskin, and Mohit Bansal. Are language models rational? the case of coherence norms and belief revision, 2024

2024
[13]

GPT-4o system card

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024

2024 arXiv
[14]

Hedging praise in english and japanese book reviews

Hiroko Itakura. Hedging praise in english and japanese book reviews. Journal of Pragmatics, 45 0 (1): 0 131--148, 2013

2013
[15]

Semantics and pragmatics of hedges in English and Japanese

Reiko Itani. Semantics and pragmatics of hedges in English and Japanese. University of London, University College London (United Kingdom), 1995

1995
[16]

Calibrating structured output predictors for natural language processing

Abhyuday Jagannatha and Hong Yu. Calibrating structured output predictors for natural language processing. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 20...

2020 doi
[17]

How can we know when language models know? on the calibration of language models for question answering

Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9: 0 962--977, 2021. doi:10.1162/tacl_a_00407. URL ...

2021 doi
[18]

Language models (mostly) know what they know

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. URL https://arxiv.org/abs/2207.05221

2022 arXiv
[19]

Selective question answering under domain shift

Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 5684--5696, Online...

2020 doi
[20]

Fostering appropriate reliance on large language models: The role of explanations, sources, and inconsistencies

Sunnie SY Kim, Jennifer Wortman Vaughan, Q Vera Liao, Tania Lombrozo, and Olga Russakovsky. Fostering appropriate reliance on large language models: The role of explanations, sources, and inconsistencies. arXiv preprint arXiv:2502.08554, 2025

2025 arXiv
[21]

Calibrated language model fine-tuning for in- and out-of-distribution data

Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang. Calibrated language model fine-tuning for in- and out-of-distribution data. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in N...

2020 doi
[22]

To hedge or not to hedge: The use of epistemic modal expressions in popular science in english texts, english--german translations, and german original texts

Svenja Kranich. To hedge or not to hedge: The use of epistemic modal expressions in popular science in english texts, english--german translations, and german original texts. Text & Talk, 2011

2011
[23]

Hedges in Japanese spoken discourse: A comparison between younger and older speakers

Shizuka Lauwereyns. Hedges in Japanese spoken discourse: A comparison between younger and older speakers. Michigan State University, 2000

2000
[24]

Lin, Jacob Hilton, and Owain Evans

Stephanie C. Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Trans. Mach. Learn. Res., 2022, 2022. URL https://api.semanticscholar.org/CorpusID:249191391

2022
[25]

Kevin Liu, Stephen Casper, Dylan Hadfield-Menell, and Jacob Andreas. Cognitive dissonance: Why do language model outputs disagree with internal representations of truthfulness? In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical...

2023 doi
[26]

Abstract sentences elicit more uncertainty and curiosity than concrete sentences

Claudia Mazzuca, Caterina Villani, Tommaso Lamarra, Marianna M Bolognesi, and Anna Borghi. Abstract sentences elicit more uncertainty and curiosity than concrete sentences. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46, 2024

2024
[27]

Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau

Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents ' overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10: 0 857--872, 2022. doi:10.1162/tacl_a_00494. URL https://aclanthol...

2022 doi
[28]

Multilingual BERT has an accent: Evaluating E nglish influences on fluency in multilingual models

Isabel Papadimitriou, Kezia Lopez, and Dan Jurafsky. Multilingual BERT has an accent: Evaluating E nglish influences on fluency in multilingual models. In Lisa Beinborn, Koustava Goswami, Saliha Murado g lu, Alexey Sorokin, Ritesh Kumar, Andreas Shcherbakov, Edoardo M. Ponti, ...

2023
[29]

Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT ? In Anna Korhonen, David Traum, and Llu \' s M \`a rquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4996--5001, Florence, Italy, July...

2019 doi
[30]

Do multilingual llms think in english? arXiv preprint arXiv:2502.15603, 2025

Lisa Schut, Yarin Gal, and Sebastian Farquhar. Do multilingual llms think in english? arXiv preprint arXiv:2502.15603, 2025

2025 arXiv
[31]

Lacie: Listener-aware finetuning for confidence calibration in large language models

Elias Stengel-Eskin, Peter Hase, and Mohit Bansal. Lacie: Listener-aware finetuning for confidence calibration in large language models. arXiv preprint arXiv:2405.21028, 2024

2024 arXiv
[32]

Quantifying uncertainty in natural language explanations of large language models

Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju. Quantifying uncertainty in natural language explanations of large language models. ArXiv, abs/2311.03533, 2023. URL https://api.semanticscholar.org/CorpusID:265043487

2023 arXiv
[33]

Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback

Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, ...

2023 doi
[34]

A cross-cultural study of indirectness and hedging in the conference proposals of english ns and nns scholars

Hacer Hande Uysal. A cross-cultural study of indirectness and hedging in the conference proposals of english ns and nns scholars. Occupying Niches: Interculturality, Cross-culturality and Aculturality in Academic Research, pp.\ 179--195, 2014

2014
[35]

Transfer learning for speech and language processing

Dong Wang and Thomas Fang Zheng. Transfer learning for speech and language processing. In 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pp.\ 1225--1237. IEEE, 2015

2015
[36]

Do llamas work in E nglish? on the latent language of multilingual transformers

Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do llamas work in E nglish? on the latent language of multilingual transformers. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computat...

2024 doi
[37]

Can LLM s express their uncertainty? an empirical evaluation of confidence elicitation in LLM s

Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLM s express their uncertainty? an empirical evaluation of confidence elicitation in LLM s. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview....

2024
[38]

Confronting verbalized uncertainty: Understanding how llm’s verbalized uncertainty influences users in ai-assisted decision-making

Zhengtao Xu, Tianqi Song, and Yi-Chieh Lee. Confronting verbalized uncertainty: Understanding how llm’s verbalized uncertainty influences users in ai-assisted decision-making. International Journal of Human-Computer Studies, 197: 0 103455, 2025. ISSN 1071-5819. doi:https://doi...

2025
[39]

Exploring linguistic and cultural variations in the use of hedges in english and chinese scientific discourse

Yingli Yang. Exploring linguistic and cultural variations in the use of hedges in english and chinese scientific discourse. Journal of Pragmatics, 50 0 (1): 0 23--36, 2013

2013
[40]

Should we respect llms? a cross-lingual study on the influence of prompt politeness on llm performance

Ziqi Yin, Hao Wang, Kaito Horio, Daisuike Kawahara, and Satoshi Sekine. Should we respect llms? a cross-lingual study on the influence of prompt politeness on llm performance. In Proceedings of the Second Workshop on Social Influence in Conversations (SICon 2024), pp.\ 9--35, 2024

2024
[41]

Navigating the grey area: How expressions of uncertainty and overconfidence affect language models

Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. arXiv preprint arXiv:2302.13439, 2023

2023 arXiv
[42]

Relying on the unreliable: The impact of language models ' reluctance to express uncertainty

Kaitlyn Zhou, Jena Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models ' reluctance to express uncertainty. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computatio...

2024 doi
[43]

Rel-ai: An interaction-centered approach to measuring human-lm reliance

Kaitlyn Zhou, Jena D Hwang, Xiang Ren, Nouha Dziri, Dan Jurafsky, and Maarten Sap. Rel-ai: An interaction-centered approach to measuring human-lm reliance. arXiv preprint arXiv:2407.07950, 2024 b

2024 arXiv
[44]

Relying on the unreliable: The impact of language models' reluctance to express uncertainty

Kaitlyn Zhou, Jena D Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models' reluctance to express uncertainty. arXiv preprint arXiv:2401.06730, 2024 c

2024 arXiv
[45]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
[46]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[47]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[48]

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

The hedging function of exemplification: Evidence from japanese

Alessandra Barotto. The hedging function of exemplification: Evidence from japanese. Journal of Pragmatics, 123: 0 24--37, 2018

work page 2018

[2] [2]

Translations of uncertainty expressions in canadian accounting and auditing standards

Ronald A Davidson and Heidi Hadlich Chrisman. Translations of uncertainty expressions in canadian accounting and auditing standards. Journal of International Accounting, Auditing and Taxation, 3 0 (2): 0 187--203, 1994

work page 1994

[3] [3]

The rational speech act framework

Judith Degen. The rational speech act framework. Annual Review of Linguistics, 9 0 (1): 0 519--540, 2023

work page 2023

[4] [4]

Calibration of pre-trained transformers

Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 295--302, Online, November 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.emnlp-main.21. URL https:/...

doi:10.18653/v1/2020.emnlp-main.21 2020

[5] [5]

A diachronic perspective on user trust in AI under uncertainty

Shehzaad Dhuliawala, Vil \'e m Zouhar, Mennatallah El-Assady, and Mrinmaya Sachan. A diachronic perspective on user trust in AI under uncertainty. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 5567--5580, Singapore, December 2023. Association for Computatio...

doi:10.18653/v1/2023.emnlp-main.339 2023

[6] [6]

Interpretation of uncertainty expressions: a cross-national study

Timothy S Doupnik and Martin Richter. Interpretation of uncertainty expressions: a cross-national study. Accounting, Organizations and Society, 28 0 (1): 0 15--35, 2003

work page 2003

[7] [7]

How to adapt your pretrained multilingual model to 1600 languages

Abteen Ebrahimi and Katharina Kann. How to adapt your pretrained multilingual model to 1600 languages. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),...

doi:10.18653/v1/2021.acl-long.351 2021

[8] [8]

Predicting pragmatic reasoning in language games

Michael C Frank and Noah D Goodman. Predicting pragmatic reasoning in language games. Science, 336 0 (6084): 0 998--998, 2012

work page 2012