Investigating the Effects of Cognitive Biases in Prompts on Large Language Model Outputs

Stanley Kok; Yan Sun

REVIEW 4 major objections 5 minor 36 references

Investigating the Effects of Cognitive Biases in Prompts on Large Language Model Outputs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Cognitive biases embedded in prompts consistently reduce LLM accuracy across tasks and models.

desk verdict A competent prompt-sensitivity study that overreaches by labeling explicit wrong-answer hints 'cognitive biases'; the effects are real but the interpretation is not. read the letter →

arxiv 2506.12338 v1 pith:5FUYR6QR submitted 2025-06-14 cs.CL

Yan Sun , Stanley Kok This is my paper · ORCID

classification cs.CL

keywords cognitivebiasesconfirmationbiasavailabilitylargelanguagemodelspromptinjectionchain-of-thoughtpromptingattentionweightsquestionanswering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that injecting common cognitive biases, specifically confirmation bias and availability bias, into otherwise neutral prompts lowers the accuracy of large language models on multiple-choice Q&A tasks. It proposes a framework that starts from an unbiased prompt, appends explicit bias phrases, and measures the accuracy gap on general and financial benchmarks. Across GPT-3.5, GPT-4, Vicuna, and Mistral, the authors report consistent accuracy drops, with the largest reaching about 24 percentage points. They also report that in open models the attention weight paid to the wrongly suggested answer rises sharply when the bias is present. If true, this means ordinary user phrasing, not just adversarial prompt attacks, can steer LLM outputs away from correct answers.

What carries the argument

The framework is a controlled prompt-contrast setup: take an unbiased multiple-choice prompt as baseline; append a hand-written bias phrase at the end; ask the model to answer with chain-of-thought and a forced "The answer is: (X)" format; then compare accuracy and attention. The bias phrases are the operative instrument: "Suggested Answer (A)/(B)" and "Many Wrong Answers" instantiate confirmation bias, while "Negative Recall," "Positive Recall," and "Positive Reference" instantiate availability bias. The attention analysis uses last-layer attention weights from the final prompt token toward the answer tokens A and B, comparing biased and unbiased prompts on Vicuna and Mistral.

What would settle it

Run the same benchmark set with "I think the answer is (B), but I'm curious to hear what you think" replaced by a matched non-first-person hint such as "A previous user answered (B)" or "The answer key says (B)". If the accuracy drops are essentially identical, the measured effect is token- or instruction-level rather than specifically cognitive; if the drops vanish, the first-person belief framing is the active ingredient.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that cognitive biases embedded in natural-language prompts are a systematic source of LLM unfaithfulness: appending sentences such as "I think the answer is (B), but I'm curious to hear what you think" or "The most recent news article I read said ..., so that is likely to be true" moves models toward the suggested option and reduces accuracy relative to an unbiased baseline. The effect appears in closed-source models (GPT-3.5 and GPT-4 on the Sports Understanding task) and open-source models (Vicuna-7b and Mistral-7b on BIG-Bench Hard and FinQA), across plausibility, causal judgment, navigation, and financial reasoning tasks. In attention analyses for Vicuna and Mistral, the final prompt token allocates more attention to the wrong option under the biased prompt, and attention to the wrong option climbs as the output is generated, correlating with incorrect conclusions.

Load-bearing premise

The load-bearing premise is that the appended phrases such as "I think the answer is (B)" actually express cognitive biases the way everyday users would, rather than operating as direct answer hints or token-level priming that would make the drops an instruction-following effect rather than a cognitive-bias effect.

Editorial extensions

If this is right

Accuracy gaps of 2 to 24 percentage points mean that even a single appended belief phrase can make LLMs measurably less reliable in Q&A settings.
Bias-aware prompt design, such as instructing models to set aside or flag a user's stated preference, should reduce these reliability losses.
The attention result suggests a diagnostic: elevated attention toward a candidate answer after a biased phrase can mark an unreliable response before the final answer is generated.
Because the effect appears across GPT-3.5, GPT-4, Vicuna, and Mistral, the paper argues that it is not an artifact of one model family or training recipe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same protocol could test whether the effect is genuinely cognitive or partly token-priming by comparing "I think the answer is (B)..." with a matched neutral suggestion such as "A random user answered (B)..."
If simple appended phrases move accuracy, longer conversational context that expresses a user's prior belief may exert similar or stronger pulls, extending the risk beyond binary Q&A to chatbot and advisory applications.
The findings point to a testable mitigation: add an instruction to ignore the user's stated belief and measure whether the accuracy gap closes.
A quantitative relationship between the strength or number of suggested-answer tokens and the accuracy drop would let developers predict which prompts need safeguards before deployment.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this paper measures something real—LLM accuracy drops when you append explicit false assertions to prompts—but it mislabels that something as 'cognitive biases.' The injection phrases in Table 1 are direct answer hints or negations, not natural expressions of confirmation or availability bias. So the central claim in the abstract and contributions overreaches.

What's new and useful: the authors build a systematic framework, run four LLMs across three BBH tasks and FinQA, compute standard errors and significance, and attempt an attention-weight analysis. That is a solid empirical skeleton. The finding that open-source models are more susceptible than GPT-4 is worth knowing, and the attention shift toward the wrong answer token is a suggestive illustration, even if it is only one example.

Soft spots, in order of severity. First, construct validity. 'I think the answer is (B)...' is an opinion, not confirmation bias. '(A)(A)...' is not an availability cue. The evaluation section even says they focus on cases where the bias points to an incorrect answer, so every injection is engineered to contradict the ground truth. Any mechanism that makes the model weight the user's assertion—sycophancy, instruction-following, token priming—produces the same drops. Without neutral controls that match the phrasing but remove the answer suggestion, the study cannot distinguish 'cognitive bias' from generic answer-priming. Second, the consistency claim is overstated: GPT-4's accuracy rose 0.3% under Suggested Answer (A), and Mistral's FinQA drop under Suggested Answer (A) is not significant. Third, the attention analysis rests on a single example per model, reports differences of a few thousandths, and only covers open-source models. Fourth, no code or data is released, so the FinQA sample selection (144 from the full set) cannot be audited.

The citation pattern is honest: they cite Turpin et al. and position themselves as an extension. They also explicitly limit the scope to explicit biases and binary-choice tasks. So the paper is not incoherent—it's just over-ambitious in its interpretation.

Who gets value: researchers studying prompt robustness, sycophancy, and human-AI interaction, and the IS community. It deserves a serious referee: the empirical design is reproducible in principle and the topic is relevant. But a referee should push hard on the construct-validity issue and ask for either a reframing as 'explicit answer suggestions' or a genuinely neutral control set. I'd accept it for review with major revisions, not desk-reject.

Referee Report

4 major / 5 minor

Summary. The paper proposes a framework for injecting 'cognitive biases' (confirmation and availability bias) into prompts by appending explicit statements such as 'I think the answer is (B), but I'm curious to hear what you think' to otherwise unbiased prompts. The framework is evaluated on binary-choice question-answering tasks from BIG-Bench Hard (Sports Understanding, Causal Judgment, Navigate) and FinQA, using GPT-3.5, GPT-4, Vicuna, and Mistral, with accuracy compared between unbiased and biased prompts. The authors report that biased prompts mostly reduce accuracy, with t-test significance flags, and they analyze attention weights in one example per open-source model, finding increased attention to the wrongly suggested option. The paper concludes that cognitive biases in prompts consistently degrade LLM performance and that attention shifts reveal the underlying mechanism.

Significance. If the central claim were supported, the paper would offer a useful, systematic demonstration that simple bias-like phrases in prompts reduce LLM accuracy across models and domains, with practical implications for bias-aware prompt design and for managerial oversight of LLM deployments. The paper has clear strengths: it uses four LLMs including both closed- and open-source models, two benchmark datasets, multiple injection types, deterministic generation at temperature 0, and it reports standard errors for the accuracy differences. The attention analysis is a creative attempt to open the black box of how prompts affect model internals. However, the paper's value is currently gated by the construct validity of the 'cognitive bias' injections and by the overstatement of the consistency of the results; the empirical regularity may be real, but it is not yet established as a cognitive-bias effect.

major comments (4)

[Table 1 and Evaluation Metrics] The study operationalizes 'cognitive biases' with explicit statements that always assert the incorrect answer (e.g., 'I think the answer is (B), but I'm curious to hear what you think'; Evaluation Metrics states 'we focus on cases where the cognitive bias points towards an incorrect answer'). Under this design, any mechanism that makes the model follow the user's explicit claim—instruction-following, sycophancy, or token-level priming—would produce the observed accuracy drops without engaging the psychological constructs of confirmation or availability bias. The manuscript does not include neutral control conditions that vary the content of the injected statement while removing the bias framing (e.g., a non-opinionated assertion, a suggestion of the correct answer, or an arbitrary repeated token) to show that the effect is specific to 'cognitive bias' phrasing. As written, the experiments demonstrate that models are swayed by explicit wrong-answer statements, but they do not support the abstract's claim that 'cognitive biases in prompts' degrade performance.
[Abstract, Contributions, Tables 2 and 5] The claim that biased prompts 'consistently lead to poorer performance across domains and LLMs' is contradicted by the authors' own results: GPT-4's accuracy increases by +0.3% under Suggested Answer (A) in Table 2, and Mistral's FinQA accuracy drops by only 2.44% under Suggested Answer (A) with a standard error of 3.96 (not significant). Additionally, the contribution claiming 'significantly degrade the performance of all LLMs (GPT-3.5, GPT-4, Vicuna, and Mistral)' is not supported by the coverage: GPT-3.5 and GPT-4 are tested only on the Sports Understanding task, so no cross-domain claim for the GPT models is established. The abstract and contributions should be qualified to reflect the actual pattern of mostly negative but not universally significant effects.
[Analysis of Attention Weights, Tables 6-7, Figure 4] The attention-weight analysis is based on a single example for Vicuna and a single example for Mistral. No confidence intervals, repeated examples, or significance tests are provided for the reported differences (e.g., Δα_B = +0.008981 for Vicuna and +0.022405 for Mistral). The claim that bias leads to a 'significant increase in attention weights towards the incorrect answer' is therefore not supported by the evidence presented. The analysis should be aggregated over many examples, ideally with a statistical test, before any mechanistic conclusion is drawn.
[Evaluation Metrics and Tables 2, 4, 5] The paper reports standard errors and t-test significance stars but does not specify what statistical test is performed (e.g., paired vs. independent, one- vs. two-sided), nor whether multiple comparisons across injection types, tasks, and models are corrected. Without this information, the '*' significance flags cannot be interpreted, and some flagged differences (e.g., Vicuna Positive Recall on Sports Understanding, -9.68% with SE 4.24*) may not survive correction. Please provide the test details or clarify that the stars are uncorrected.

minor comments (5)

[Table 4, Navigate row] The text reports Vicuna's Navigate baseline as 51.97%, but the table lists 50.32%; the same value 51.97% is also used for the Causal Judgment baseline, suggesting a copy-paste error. Please correct the inconsistency.
[Table 3] The manual check table is internally confusing: the model's final answer is rendered as '(B) implausible', but the task options are presumably (A) implausible and (B) plausible. Please reconcile the option labels and the marked wrong answer.
[Experimental Setup and Table 1] The exact unbiased prompt template and the construction of the 'Many Wrong Answers' injection (number of repetitions, whether it repeats (A) or (B), and how it is appended to the prompt) are not specified. Please provide full prompt templates and injection details in an appendix for reproducibility.
[Datasets] In the introduction, BIG-Bench Hard is cited as 'Tafjord et al. (2022)' while the dataset section correctly cites Suzgun et al. (2022); the Tafjord citation appears to be an error.
[Analysis of Attention Weights and Figure 4] The text says the framework focuses on biases placed at the end of prompts, but Figure 4 is described as showing injection positions 'Tail, Middle, Head'; please clarify whether middle/head positions were actually tested or whether this is a figure-labeling artifact.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the empirical protocol measures accuracy and attention directly, with no fitted parameter or self-citation chain reducing the result to its inputs.

full rationale

The paper's derivation chain is empirical rather than analytic: it constructs unbiased and biased prompts, runs four LLMs on benchmark subsets, and directly measures accuracy and attention weights. No parameter is fitted from the target results, no result is defined in terms of the quantity it is said to predict, and the paper does not rely on a load-bearing self-citation or imported uniqueness theorem. The Evaluation Metrics statement that the study focuses on cases where the injected bias points toward an incorrect answer could raise a construct-validity question (the injections are essentially explicit wrong-answer suggestions, so the observed drops may reflect sycophancy or answer priming rather than psychological bias), but that is a threat to external validity and interpretation, not circularity: a model could in principle ignore the injected text, and whether it obeys is an empirical fact measured against independent benchmark labels. The attention-weight analysis is likewise a direct measurement on open-source models and does not use the accuracy results to derive itself. Under the stated review standard, no specific reduction of a claim to its own inputs can be exhibited, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim relies on several domain assumptions about experimental isolation, attention interpretability, and dataset representativeness, but there are no fitted parameters or invented entities.

assumptions (4)

domain assumption The difference in accuracy between the unbiased and biased prompts isolates the effect of the injected bias.
This assumes no other confounds differ between conditions; the experimental design varies only the appended bias text.
domain assumption Attention weights are interpretable as model focus and are causally relevant to output choice.
The attention analysis in Section 6 treats attention weight increases as evidence of the model favoring the wrong answer; this is a contested assumption in interpretability research.
domain assumption The selected tasks and samples (3 of 23 BBH tasks, 144 FinQA samples) are representative of general and financial Q&A.
The paper generalizes from these subsets to 'across multiple benchmark datasets' without a selection rationale or a broader evaluation.
standard math Temperature 0 yields deterministic model outputs.
The paper uses temperature 0 for all models, which typically makes decoding greedy, but does not guarantee determinism across all APIs or implementations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Investigating the Effects of Cognitive Biases in Prompts on Large Language Model Outputs." pith.science (2026). https://pith.science/paper/5FUYR6QR

@misc{pith2026250612338,
  author       = {Pith},
  title        = {Pith review of: Investigating the Effects of Cognitive Biases in Prompts on Large Language Model Outputs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5FUYR6QR}},
  note         = {Machine review of arXiv:2506.12338}
}

read the original abstract

This paper investigates the influence of cognitive biases on Large Language Models (LLMs) outputs. Cognitive biases, such as confirmation and availability biases, can distort user inputs through prompts, potentially leading to unfaithful and misleading outputs from LLMs. Using a systematic framework, our study introduces various cognitive biases into prompts and assesses their impact on LLM accuracy across multiple benchmark datasets, including general and financial Q&A scenarios. The results demonstrate that even subtle biases can significantly alter LLM answer choices, highlighting a critical need for bias-aware prompt design and mitigation strategy. Additionally, our attention weight analysis highlights how these biases can alter the internal decision-making processes of LLMs, affecting the attention distribution in ways that are associated with output inaccuracies. This research has implications for Al developers and users in enhancing the robustness and reliability of Al applications in diverse domains.

Figures

Figures reproduced from arXiv: 2506.12338 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 27 canonical work pages

[1]

Investigating the Effects of Cognitive Biases in Prompts on Large Language Model Outputs

Systems (ICIS) Association for Information Systems AIS Electronic Library (AISeL) ICIS 2024 Proceedings International Conference on Information December 2024 Investigating the Effects of Cognitive Biases in Prompts on Large Language Model Outputs Yan Sun National University of Singapore, yansun@comp.nus.edu.sg Stanley Kok National University of Singapore,...

work page 2024
[3]

Let’s think step-by-step,

We analyze the attention weights of LLMs to understand how specific forms of confirmation bias affect the distribution of the weights. This analysis sheds light on the underlying mechanism that drives biased outcomes, laying the groundwork for greater transparency and reliability in LLMs. Related Work In this section, we provide a brief overview of prior ...

work page 2023
[4]

Cognitive Biases in Human-LLM Interaction. (The dashed arrow represents our focus.) Output LLM Prompt ? Human Cognitive Bias Effects of Cognitive Biases on Language Model Outputs Forty-Fifth International Conference on Information Systems, Bangkok, Thailand 2024 3 A recent line of research has just begun to explore the effect of biased prompts on LLM outp...

work page 2024
[5]

A” and “B

Impact of Bias Types on Open-Source LLMs’ Performance on the FinQA Dataset (Note: *p<5% **p<1% ***p<0.1%) The Many Wrong Answer bias causes a decline in performance for Mistral by 4.47% and a larger drop for Vicuna by 12.84%. This could indicate that Mistral, despite the drop, maintains a certain resilience against misleading information, whereas Vicuna s...

work page
[6]

and Mistral (Jiang et al. 2023)). Our empirical results reveal that biased prompts consistently lead to poorer performance across domains and LLMs. To delve deeper into the impact of biases, we analyze how attention weights shift between a potential answer (e.g., option A or B in Figure

work page 2023
[7]

In our study, we observed notable differences in how biases affect the outputs of different LLMs

Comparison of Token’s Last-Token Attention on Mistral Forty-Fifth International Conference on Information Systems, Bangkok, Thailand 2024 14 Effects of Cognitive Biases on Language Model Outputs With the accuracy drops shown in the previous tables and a noticeable divergence in the model’s attention, it implies that a user’s prior misunderstanding can be ...

work page 2024
[8]

and Mistral-7b-v2 model (Jiang et al. 2023)1. Due to cost constraints, we test GPT-3.5 and GPT-4 only on the Sports Understanding task in the BBH dataset. For the Vicuna and Mistral models, we run on both BBH and FinQA datasets. To ensure replicable outcomes, we employ a temperature setting of 0 during solution generation for all LLMs. In addition, the ma...

work page 2023
[9]

Let’s think step by step

Specif- ically, the Suggested Answer types for confirmation bias—both (A) and (B)— subtly imply a correct answer, enticing the model to validate the suggested option. We also include an extreme scenario for confirmation bias, named Many Wrong Answers. For example, (A)(A)...(A)(A) abruptly introduces incorrect choices and 1In this paper, we will refer to V...

work page 2024

Show all 36 references

[10]

Wrong Answer in Bold; Attitude Change underlined

GPT-4’s Unfaithful Answers by Suggested Answer (B). Wrong Answer in Bold; Attitude Change underlined. Effects of Cognitive Biases on Language Model Outputs Forty-Fifth International Conference on Information Systems, Bangkok, Thailand 2024 10 Open-Source LLMs The performance o...

2024
[11]

Here, the presence of many wrong answers results in a marked accuracy reduction for Mistral by 19.13% and a less pronounced but still significant decline for Vicuna by 4.35%

Impact of Bias Types on Open-Source LLMs’ Performance on the BBH Dataset (Note: *p<5% **p<1% ***p<0.1%) When analyzing the causal judgment task, we see a baseline accuracy of 59.57% for Mistral and 51.97% for Vicuna. Here, the presence of many wrong answers results in a marked...

2024
[15]

unboxing

Attentions to Option (A/B) across the Mistral’s Outputs (Injection Position: Tail, Middle, Head) Forty-Fifth International Conference on Information Systems, Bangkok, Thailand 2024 15 Effects of Cognitive Biases on Language Model Outputs notably confirmation and availability b...

2024
[17]

Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios,

“Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios,” Journal of Medical Systems (47:1) 2023, p

2023
[19]

It has been accepted for inclusion in ICIS 2024 Proceedings by an authorized administrator of AIS Electronic Library (AISeL)

https://aisel.aisnet.org/icis2024/aiinbus/aiinbus/19 This material is brought to you by the International Conference on Information Systems (ICIS) at AIS Electronic Library (AISeL). It has been accepted for inclusion in ICIS 2024 Proceedings by an authorized administrator of A...

2024
[20]

BERT: Pre-training of Deep Bidirectional Trans- formers for Language Understanding,

“BERT: Pre-training of Deep Bidirectional Trans- formers for Language Understanding,” in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–...

2019
[22]

Mistral 7B,

“Mistral 7B,” in: arXiv preprint arXiv:2310.06825, Kim, S., Joo, S. J., Kim, D., Jang, J., Ye, S., Shin, J., and Seo, M

arXiv
[23]

The CoT Collection: Improv- ing Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning,

“The CoT Collection: Improv- ing Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning,” ArXiv (abs/2305.14045). Kliegr, T., Bahnık, Š., and Fürnkranz, J

arXiv
[24]

Large Language Models are Zero-Shot Reasoners,

“Large Language Models are Zero-Shot Reasoners,” ArXiv (abs/2205.11916). Korteling, J. E., Paradies, G. L., and Sassen-van Meer, J. P

arXiv
[25]

Revealing the Dark Secrets of BERT,

“Revealing the Dark Secrets of BERT,” in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4365–4374. Lanham, T. et al

2019
[26]

Measuring Faithfulness in Chain-of-Thought Reasoning,

“Measuring Faithfulness in Chain-of-Thought Reasoning,” ArXiv (abs/2307.13702). Liu, V. and Chilton, L. B

arXiv
[27]

Design guidelines for prompt engineering text-to-image generative models,

“Design guidelines for prompt engineering text-to-image generative models,” in: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–23. Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M., and Callison-Burch, C

2022
[28]

Faithful Chain-of-Thought Reasoning,

“Faithful Chain-of-Thought Reasoning,” ArXiv (abs/2301.13379). Madaan, A. and Yazdanbakhsh, A

arXiv
[29]

Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango,

“Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango,” ArXiv (abs/2209.07686). Nickerson, R. S

arXiv
[30]

Automating Customer Service using LangChain: Building custom open- source GPT Chatbot for organizations,

“Automating Customer Service using LangChain: Building custom open- source GPT Chatbot for organizations,” ArXiv (abs/2310.05421). Parcalabescu, L. and Frank, A

arXiv
[32]

Addressing cognitive bias in medical language models,

“Addressing cognitive bias in medical language models,” in: arXiv preprint arXiv:2402.08113, Sennrich, R., Haddow, B., and Birch, A

arXiv
[33]

Neural Machine Translation of Rare Words with Subword Units,

“Neural Machine Translation of Rare Words with Subword Units,” in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany: Association for Computational Linguistics, 2016, pp. 1715–

2016
[34]

Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning,

“Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning,” in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2078–2093. Teubner, T., Flath, C. M., Weinhardt, C., van der Aalst, W., and Hinz, O

2022
[35]

Welcome to the Era of ChatGPT et Al

“Welcome to the Era of ChatGPT et Al.” Business & Information Systems Engineering (65:2) 2023, pp. 95–101. Touvron, H. et al

2023
[36]

Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts,

“Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts,” in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–21

2023
[2007]

Attitudes as object–evaluation associations of varying strength,

“Attitudes as object–evaluation associations of varying strength,” Social cognition (25:5), pp. 603–637. Effects of Cognitive Biases on Language Model Outputs Forty-Fifth International Conference on Information Systems, Bangkok, Thailand 2024 16 Fazio, R. H. and Olson, M. A

2024
[2016]

A” or “B

to efficiently manage out-of-vocabulary words by merging frequent character sequences into subwords, optimizing vocabulary size. In our investigation of how cognitive biases influence language model outputs, particular attention is given to the processing of the last token in ...

2024
[2019]

What Does BERT Look at? An Analysis of BERT’s Attention,

“What Does BERT Look at? An Analysis of BERT’s Attention,” in: Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpret- ing Neural Networks for NLP, pp. 276–286. Clark, P., Tafjord, O., and Richardson, K

2019
[2020]

Language Models are Few-Shot Learners,

“Language Models are Few-Shot Learners,” ArXiv (abs/2005.14165). Cascella, M., Montomoli, J., Bellini, V., and Bignami, E

2005 arXiv
[2021]

FinQA: A Dataset of Numerical Reasoning over Financial Data,

“FinQA: A Dataset of Numerical Reasoning over Financial Data,” in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3697–3711. Chiang, W.-L. et al

2021
[2022]

2021)) covering different domains, and conduct experiments on both closed-source LLMs (GPT-3.5 and GPT-4 (OpenAI et al

and FinQA (Chen et al. 2021)) covering different domains, and conduct experiments on both closed-source LLMs (GPT-3.5 and GPT-4 (OpenAI et al. 2024)) and open-source models (Vicuna (Chiang et al

2021
[2023]

Santi Cazorla called for the screen

by allowing humans to interact with artificial intelligence systems in en- tirely new ways. Humans communicate with LLMs by providing textual inputs called prompts, which in turn guide LLMs to generate surprisingly coherent and plausible replies and explanations. Fueled by thi...

2011
[2024]

On measuring faithfulness or self-consistency of natural language explanations,

“On measuring faithfulness or self-consistency of natural language explanations,” in: Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (ACL 2024), to appear. Effects of Cognitive Biases on Language Model Outputs Forty-Fifth Internationa...

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Investigating the Effects of Cognitive Biases in Prompts on Large Language Model Outputs

Systems (ICIS) Association for Information Systems AIS Electronic Library (AISeL) ICIS 2024 Proceedings International Conference on Information December 2024 Investigating the Effects of Cognitive Biases in Prompts on Large Language Model Outputs Yan Sun National University of Singapore, yansun@comp.nus.edu.sg Stanley Kok National University of Singapore,...

work page 2024

[2] [3]

Let’s think step-by-step,

We analyze the attention weights of LLMs to understand how specific forms of confirmation bias affect the distribution of the weights. This analysis sheds light on the underlying mechanism that drives biased outcomes, laying the groundwork for greater transparency and reliability in LLMs. Related Work In this section, we provide a brief overview of prior ...

work page 2023

[3] [4]

Cognitive Biases in Human-LLM Interaction. (The dashed arrow represents our focus.) Output LLM Prompt ? Human Cognitive Bias Effects of Cognitive Biases on Language Model Outputs Forty-Fifth International Conference on Information Systems, Bangkok, Thailand 2024 3 A recent line of research has just begun to explore the effect of biased prompts on LLM outp...

work page 2024

[4] [5]

A” and “B

Impact of Bias Types on Open-Source LLMs’ Performance on the FinQA Dataset (Note: *p<5% **p<1% ***p<0.1%) The Many Wrong Answer bias causes a decline in performance for Mistral by 4.47% and a larger drop for Vicuna by 12.84%. This could indicate that Mistral, despite the drop, maintains a certain resilience against misleading information, whereas Vicuna s...

work page

[5] [6]

and Mistral (Jiang et al. 2023)). Our empirical results reveal that biased prompts consistently lead to poorer performance across domains and LLMs. To delve deeper into the impact of biases, we analyze how attention weights shift between a potential answer (e.g., option A or B in Figure

work page 2023

[6] [7]

In our study, we observed notable differences in how biases affect the outputs of different LLMs

Comparison of Token’s Last-Token Attention on Mistral Forty-Fifth International Conference on Information Systems, Bangkok, Thailand 2024 14 Effects of Cognitive Biases on Language Model Outputs With the accuracy drops shown in the previous tables and a noticeable divergence in the model’s attention, it implies that a user’s prior misunderstanding can be ...

work page 2024

[7] [8]

and Mistral-7b-v2 model (Jiang et al. 2023)1. Due to cost constraints, we test GPT-3.5 and GPT-4 only on the Sports Understanding task in the BBH dataset. For the Vicuna and Mistral models, we run on both BBH and FinQA datasets. To ensure replicable outcomes, we employ a temperature setting of 0 during solution generation for all LLMs. In addition, the ma...

work page 2023

[8] [9]

Let’s think step by step

Specif- ically, the Suggested Answer types for confirmation bias—both (A) and (B)— subtly imply a correct answer, enticing the model to validate the suggested option. We also include an extreme scenario for confirmation bias, named Many Wrong Answers. For example, (A)(A)...(A)(A) abruptly introduces incorrect choices and 1In this paper, we will refer to V...

work page 2024