Pith. sign in

REVIEW 3 major objections 8 minor 2 cited by

Acquiescence Bias in Large Language Models

T0 review · 3 major / 8 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Large language models answer 'no' even when 'no' means agreement

desk verdict Solid data and clean exposition undermined by a baseline mismatch in the disagree condition that kills the abstract's 'regardless' claim; the narrower English reformatting effect is real but needs to be reported as such. read the letter →

arxiv 2509.08480 v1 pith:XLKRA5IJ submitted 2025-09-10 cs.CL

Daniel Braun This is my paper · ORCID
classification cs.CL
keywords acquiescencebiaslargelanguagemodelspromptsensitivitysurveydesignresponseyes/noquestionsnegationlegalNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether large language models show acquiescence bias, the human tendency to agree with survey statements regardless of content. Across five models, nine binary tasks, and three languages, it finds the opposite: when A/B choices are converted into yes/no questions, models shift toward the token "no," and they do so even in disagreement-phrased prompts where "no" logically signals agreement. The effect is consistent across models and tasks in English, while German and Polish show significant but patternless prompt influence. The paper concludes that prompt phrasing materially changes LLM outputs and that LLMs are poorly suited to simulating human survey responses.

What carries the argument

The instrument is a paired set of five prompt conditions built from the same binary questions: a neutral A/B choice, a yes/no question on the first option, an agreement question, a negated-agreement question, and a disagreement question. By counting absolute "no" responses across conditions and testing differences with McNemar's test, the design isolates the effect of question phrasing from the content of the question. The disagreement condition is the logical hinge: because "no" there signals agreement with the statement, a persistent rise in "no" responses separates a token-level response bias from a semantic agreement or disagreement bias.

What would settle it

Re-analyze the English results in Table 1 using the neutral option-A rate, not the option-B rate, as the baseline for the disagreement condition's "no" responses. If the disagreement-condition "no" rate is not consistently above the neutral A rate across the five models, the claim of a no-bias independent of semantic meaning fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that LLMs display a bias toward the surface response "no" that is independent of the semantic content of that answer. In the neutral condition models choose between two options; converting the first option into a yes/no question raises the number of responses equivalent to the second option by 31 to 203 percent for English. Asking "Do you agree...?" produces the same increase in "no" answers, and asking "Do you disagree...?" where "no" means agreement with the statement still produces more "no" answers for most models. Because the bias tracks the token rather than the logical meaning, the paper argues it is neither acquiescence nor its opposite, and it reads the result as evidence that LLM responses are not primarily driven by reasoning.

Load-bearing premise

The conclusion rests on comparing "no" answers in the disagreement condition with the neutral condition's second option, even though semantically that "no" matches the neutral condition's first option; if that baseline is wrong, the apparent no-bias may disappear.

Editorial extensions

If this is right

  • Converting survey items from A/B choices into yes/no format will push LLM outputs toward "no" in English, even when the correct answer is the first option.
  • LLMs are unreliable stand-ins for human survey respondents, at least on legal-domain binary tasks, because they do not reproduce the human acquiescence pattern.
  • Prompt phrasing is a material variable in LLM evaluation: adding "do you agree" or "don't you agree" changes accuracy in both directions depending on the task's base error distribution.
  • The consistency of the no-bias across model sizes suggests it is not a quirk of one architecture or scale, within the tested set.
  • The finding undercuts the assumption that large models answer such questions by reasoning, since logically opposite phrasings converge on the same surface answer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The no-bias could be a property of the response format: models may be primed to emit the second token of a two-token set, and a test that randomly permutes the order of "Yes" and "No" would separate a true no-bias from an order effect.
  • A natural extension is to test the same five conditions in non-legal or conversational domains, where the distribution of true yes/no answers differs and where the paper's own limitation note leaves the question open.
  • The German and Polish results, where prompt effects are significant but directionless, may indicate that negation and politeness morphology interact with the bias; this goes beyond what the paper claims.
  • One could reverse the answer options so that the first option is "No" and see whether the bias tracks the word "no" or the position of the second option.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper investigates whether large language models (LLMs) reproduce human acquiescence bias, the tendency to agree with survey statements regardless of content. It evaluates five models (Llama-3.1-8B, Mistral-Small-24B, Gemma-2-27B, Llama-3.3-70B, GPT-4o) on nine legal-domain tasks in English, German, and Polish, under five prompt conditions: neutral A/B choice, yes/no, agreement, negated agreement, and disagreement. The authors report that prompt phrasing significantly changes responses, that no consistent human-like acquiescence bias is found, and that English models display a bias toward the response 'no' regardless of whether that response indicates agreement or disagreement, leading them to conclude that LLMs are not well suited to simulate human survey responses.

Significance. If the central claim were supported, the paper would contribute a large-scale, multilingual negative result of interest to prompt design and to the use of LLMs as survey simulators. The study has notable strengths: more than 37,975 question variations and 152,000 responses, five models in three languages, publicly available code and data, and per-task statistical tests. However, the headline claim rests on a baseline comparison that is semantically invalid in the disagreement condition; after correcting that comparison, the evidence contradicts the 'regardless' formulation. The paper's remaining contributions—prompt sensitivity, absence of acquiescence bias, and a narrower response-mode bias in certain English conditions—are defensible but require substantial reanalysis and reframing.

major comments (3)
  1. [Section 4 / Table 1; Section 3.2] The disagree-condition rows in Table 1 benchmark 'No' responses against the neutral condition's B count, but Section 3.2 states that in the disagreement prompt 'no' implies agreement with the question and is therefore equivalent to option A. The correct baseline is neutral A, not B. Recomputing the English relative changes with neutral A as baseline yields -52% (Llama-3.1-8B), -73% (Mistral-24B), -46% (Gemma-2-27B), -85% (Llama-3.3-70B), and -31% (GPT-4o), instead of the +190%, +6%, +292%, -83%, and +17% reported in the table. Every model produces fewer agreement-equivalent 'No' responses in the disagree condition than 'A' responses in the neutral condition. The disagree-condition evidence therefore does not support the abstract and conclusion claim that LLMs display a bias toward 'no' regardless of whether it indicates agreement or disagreement; in fact, it points in the opposite direction.
  2. [Section 4, metric switch] The paper discloses a post hoc switch from the pre-specified positive-rate metric to the absolute number of 'no' responses ('Instead of focusing on positives, we decided to focus...'). Because the new metric is introduced after observing the results and is then applied without condition-specific semantic anchors, the central 'no'-bias claim inherits the baseline mismatch described above. The pre-specified positive-rate results (Appendix D) should be reported as the primary analysis, and any 'no'-count analysis should use the semantically equivalent anchor in each condition: neutral B for the yes/no, agreement, and (with appropriate caveats) negated-agreement conditions, and neutral A for the disagreement condition.
  3. [Section 4 / Table 1, claim of consistency] Even under the paper's own B-baseline, the English disagree row for Llama-3.3-70B is -83% (vs neutral B), and the paper acknowledges 'all but one model,' so the asserted consistency is already weak. Once the semantically correct A-baseline is used, none of the five models shows an increase in agreement-equivalent responses in the disagree condition. The 'regardless of whether it indicates agreement or disagreement' conclusion thus collapses, and the paper's central contribution would need to be narrowed to a response-mode bias in the conditions where 'no' corresponds to the B option in English.
minor comments (8)
  1. [Table 3 and Section 3.1] The corpus name is misspelled as 'Legelbench' in Table 3 and in Section 3.1; it should be 'Legalbench.'
  2. [Section 4] The phrase 'pă0.05' appears to be a rendering error; it should read 'p < 0.05.'
  3. [Section 4] There is a typo, 'Lllama-3.3-70B,' in the paragraph discussing Table 2.
  4. [Appendix C] Several JSON keys contain spaces (e.g., 'n e g a t e d _ a g r e e _ p r o m p t' and 'd e f i n i t i o n _ c l a s s i f i c a t i o n'); if these are not formatting artifacts, they should be corrected to valid identifiers.
  5. [Appendix C] The key 'neutral_anwers' is a typo for 'neutral_answers'.
  6. [Section 3.2] The example in the Neutral condition renders 'V oid' with a space; this should be 'Void.'
  7. [Abstract / Introduction / Conclusion] The abstract says the bias is observed 'across different models, tasks, and languages,' but the 'no' bias is only claimed for English; the abstract should align with the English-only finding reported in the results and conclusion.
  8. [Table 1] The relative-change percentages in the German and Polish disagree rows use the same mismatched baseline (neutral B) and should be recalculated or relabeled, even though the paper does not claim a consistent pattern in those languages.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the study's claim is an empirical measurement against external benchmarks, with no fitted parameters or derivation chain; the only self-citation is a corpus provenance reference.

full rationale

The paper is an empirical measurement study: it counts LLM responses under five prompt conditions and reports the counts in Table 1. There is no fitted parameter, no uniqueness theorem, and no derivation chain whose output could reduce to its input by construction. The central claim (a bias toward answering 'no') is a summary of observed response counts, and those counts came from independently sourced benchmarks (Legalbench, LEPISZCZE) plus the AGB-DE corpus. The only self-citation, Braun and Matthes (2024), is used solely to identify the German dataset; it does not supply any load-bearing theoretical premise or define the outcome metric. The disagree-condition rows in Table 1 do raise a serious baseline-validity concern: the paper itself states in Section 3.2 that in the disagreement prompt 'no' implies agreement and is therefore equivalent to option A, yet Table 1 brackets the disagree 'No (A)' counts against the neutral B counts. That is a measurement/interpretation flaw that could undermine the reported pattern, but it is not circularity in the derivation sense: the counts are not entailed by the hypothesis, and they could have contradicted the claim (as Llama-3.3-70B's -83% in the English disagree row does). The result is externally checkable through the released code and responses, so the argument is not self-supporting. Score 2 reflects the minor non-load-bearing self-citation, not any circular step.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no fitted free parameters or invented entities. The central claim rests on assumptions about the validity of prompt-based comparisons as measures of response bias, the comparability of the 'no' token across conditions, and the generalizability of legal-domain English results to broader LLM behavior.

assumptions (5)
  • domain assumption The neutral A/B condition is an unbiased baseline for measuring response bias; any shift in the distribution of responses under rephrasing is attributable to bias rather than to changes in task interpretation or difficulty.
    The entire analysis compares 'yes/no' and 'agree/disagree' counts to 'A/B' counts from the neutral prompt. If rephrasing changes the task rather than the response tendency, the comparison is invalid. Section 3.2 introduces the condition design.
  • ad hoc to paper The 'No' response token in the disagreement condition can be compared to the neutral condition's second option (B) to test a no-answer bias.
    In the disagree prompt, 'No' is semantically equivalent to option A (agreement with the statement), not to B. The paper nonetheless counts 'No' responses against the neutral B baseline (Table 1), an assumption specific to this analysis.
  • domain assumption The rephrased prompts preserve the semantic content and difficulty of the original classification tasks, so response differences are not caused by rephrasing artifacts.
    Prompts are designed to be concise and minimally different, but the yes/no condition changes the task from forced-choice classification to question answering. Section 3.2 states this is intentional, but it is assumed rather than validated.
  • domain assumption Single stochastic samples at temperature 1.0 provide a stable enough estimate of each model's response distribution for McNemar significance tests.
    Each item is sampled once per model and condition. The paper argues the large sample size makes the pattern robust, but no repeated sampling or variance estimate is provided. Section 3.3 sets temperature to 1.0; limitations acknowledge randomness.
  • domain assumption Findings from legal-domain tasks in three languages, with consistent patterns only in English, support general conclusions about LLM suitability for survey simulation.
    The conclusion 'LLMs are not well suited to simulate human responses' goes beyond the legal-domain English-only evidence. The limitations section acknowledges this but the abstract states it without qualification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Acquiescence Bias in Large Language Models." pith.science (2026). https://pith.science/paper/XLKRA5IJ

@misc{pith2026250908480,
  author       = {Pith},
  title        = {Pith review of: Acquiescence Bias in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XLKRA5IJ}},
  note         = {Machine review of arXiv:2509.08480}
}
read the original abstract

Acquiescence bias, i.e. the tendency of humans to agree with statements in surveys, independent of their actual beliefs, is well researched and documented. Since Large Language Models (LLMs) have been shown to be very influenceable by relatively small changes in input and are trained on human-generated data, it is reasonable to assume that they could show a similar tendency. We present a study investigating the presence of acquiescence bias in LLMs across different models, tasks, and languages (English, German, and Polish). Our results indicate that, contrary to humans, LLMs display a bias towards answering no, regardless of whether it indicates agreement or disagreement.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Tag Questions and the Generational Reversal of Sycophancy Across 45 Language Models

    cs.CL 2026-07 conditional novelty 8.0 of 10

    Across 45 LLMs, the 'right?' tag effect flips from sycophantic to resistant over four years of releases, while the 'maybe?' tag raises agreement in every model — anti-sycophancy training is grammar-keyed and one-sided.

  2. Benchmarking LLM Competence on Logical Inference over Probability Operators

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Only 9 of 29 tested LLMs beat a chance-level competence floor on valid/invalid inferences over probability operators like "probably" and "might"; most answer from a fixed yes/no bias.

Reference graph

Works this paper leans on

26 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [1]

    Sotiris Anagnostidis and Jannis Bulian. 2024. How susceptible are llms to influence in prompts? arXiv preprint arXiv:2408.11865

    arXiv 2024

  2. [2]

    Lukasz Augustyniak, Kamil Tagowski, Albert Sawczyn, Denis Janiak, Roman Bartusiak, Adrian Szymczak, Arkadiusz Janz, Piotr Szyma\' n ski, Marcin W a troba, Miko aj Morzy, Tomasz Kajdanowicz, and Maciej Piasecki. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/890b206ebb79e550f3988cb8db936f42-Paper-Datasets_and_Benchmarks.pdf This is the wa...

    work page 2022

  3. [3]

    Daniel Braun and Florian Matthes. 2024. https://doi.org/10.18653/v1/2024.acl-long.559 AGB - DE : A corpus for the automated legal assessment of clauses in G erman consumer contracts . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10389--10405, Bangkok, Thailand. Association for Co...

    doi:10.18653/v1/2024.acl-long.559 2024

  4. [4]

    Daniel Danner, Julian Aichholzer, and Beatrice Rammstedt. 2015. Acquiescence in personality questionnaires: Relevance, domain specificity, and stability. Journal of Research in Personality, 57:119--130

    work page 2015

  5. [5]

    Jessica Maria Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, and Zexue He. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.739 Cognitive bias in decision-making with LLM s . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12640--12653, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.findings-emnlp.739 2024

  6. [6]

    Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.452 L aw B ench: Benchmarking legal knowledge of large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proc...

    doi:10.18653/v1/2024.emnlp-main.452 2024

  7. [7]

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher R \'e , Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. 2024. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36

    2024

  8. [8]

    Patrick Haller, Jannis Vamvas, and Lena Ann J \"a ger. 2024. Yes, no, maybe? revisiting language models' response stability under paraphrasing for the assessment of political leaning. In First Conference on Language Modeling

    work page 2024

Show all 26 references
  1. [9]

    Seth J Hill and Margaret E Roberts. 2023. Acquiescence bias inflates estimates of conspiratorial beliefs and political misperceptions. Political Analysis, 31(4):575--590

    2023

  2. [10]

    Thanmay Jayakumar, Fauzan Farooqui, and Luqman Farooqui. 2023. https://doi.org/10.18653/v1/2023.nllp-1.22 Large language models are legal but they are not: Making the case for a powerful L egal LLM . In Proceedings of the Natural Legal Language Processing Workshop 2023, pages ...

    2023 doi

  3. [11]

    Jill M Johnson, Dennis N Bristow, Kenneth C Schneider, et al. 2004. Did you not understand the question or not? an investigation of negatively worded questions in survey research. Journal of Applied Business Research (JABR), 20(1)

    2004

  4. [12]

    Erik Jones and Jacob Steinhardt. 2022. Capturing failures of large language models via human cognitive biases. Advances in Neural Information Processing Systems, 35:11785--11799

    2022

  5. [13]

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.29 Benchmarking cognitive biases in large language models as evaluators . In Findings of the Association for Computational Linguistics: ACL ...

    2024 doi

  6. [14]

    Krosnick

    Jon A. Krosnick. 1999. https://doi.org/10.1146/annurev.psych.50.1.537 Survey research . Annual Review of Psychology, 50(Volume 50, 1999):537--567

    1999 doi

  7. [15]

    Manikanta Loya, Divya Sinha, and Richard Futrell. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.241 Exploring the sensitivity of LLM s' decision-making capabilities: Insights from prompt variations and hyperparameters . In Findings of the Association for Computational ...

    2023 doi

  8. [16]

    Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153--157

    1947

  9. [17]

    Beatrice Rammstedt and Richard F Farmer. 2013. The impact of acquiescence on the evaluation of personality structure. Psychological assessment, 25(4):1137

    2013

  10. [18]

    Matthew Renze. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.432 The effect of sampling temperature on problem solving in large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7346--7356, Miami, Florida, USA. Associatio...

    2024 doi

  11. [19]

    Maribel Romero and Chung-Hye Han. 2004. On negative yes/no questions. Linguistics and philosophy, 27(5):609--658

    2004

  12. [20]

    Paul R \"o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. 2024. https://doi.org/10.18653/v1/2024.acl-long.816 Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large lan...

    2024 doi

  13. [21]

    Cor Steging, Silja Renooij, and Bart Verheij. 2025. https://arxiv.org/abs/2505.01539 Parameterized argumentation-based reasoning tasks for benchmarking generative language models . Preprint, arXiv:2505.01539

    2025 arXiv

  14. [22]

    Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. Do llms exhibit human-like response biases? a case study in survey design. Transactions of the Association for Computational Linguistics, 12:1011--1026

    2024

  15. [23]

    index of political efficacy?

    James D Wright. 1975. Does acquiescence bias the" index of political efficacy?". The Public Opinion Quarterly, 39(2):219--226

    1975

  16. [24]

    Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.108 P ro SA : Assessing and understanding the prompt sensitivity of LLM s . In Findings of the Association for Computational Linguistics: EMN...

    2024 doi

  17. [25]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  18. [26]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.