Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

More or Less Wrong: A Benchmark for Directional Bias in LLM Comparative Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Comparative words in a prompt — 'more', 'less', 'equal' — reliably steer LLM predictions toward the framing term, even when that answer is wrong, and the bias grows when the prompt names a gender or race.

desk verdict A useful new bias benchmark, but the headline 'consistent' steering claim is contradicted by its own Table 1 and the prompt controls are too loose to support 'mere presence'. read the letter →

arxiv 2506.03923 v1 pith:UITEON5U submitted 2025-06-04 cs.CL

Mohammadamin Shafiei , Hamidreza Saffari , Nafise Sadat Moosavi This is my paper
classification cs.CL
keywords directionalframingbiascomparativereasoningLLMevaluationMathCompbenchmarkDirErrmetricchain-of-thoughtmitigationdemographicpromptsensitivity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that large language models answer simple comparison questions with a directional bias: when a prompt contains the word 'more', models over-predict 'more'; when it contains 'less', they over-predict 'less'; and likewise for 'equal', even when the underlying quantities are identical and the answer is wrong. To make the effect measurable, the authors build MathComp, a set of 300 comparison scenarios, each asked in fourteen framing variants, and test six models from three families using a new directional-error metric that counts which way a model errs. They find that the drift is pronounced in ordinary one-word answers, shrinks under chain-of-thought prompting but does not disappear, and grows again when the person being described carries a gender or race label. The paper's point is that standard accuracy metrics hide a systematic steering of LLM reasoning, so evaluation must count the direction of errors, not just their rate.

What carries the argument

The machinery is the benchmark-and-metric pair: MathComp's 14-way prompt grid and the directional error metric $\mathrm{DirErr}(y)$, the proportion of instances in which the model predicts label $y$ while the true label is not $y$, divided by all instances for which $y$ would be an error. This turns wrongness into a signed quantity, so a heat-map across framing variants can show whether 'more'-framed prompts push mistakes toward 'more'. The grid crosses three framing styles (neutral, direct, indirect) with three cue terms ('more', 'less', 'equal') and two positions (beginning, end), then adds demographic identity insertion, so the only thing that varies between cells is the lexical cue and its placement.

What would settle it

Re-run the experiment with the indirect framings' asserted content removed while keeping the cue word — for example, replace '[Person B] spends more time on home maintenance than [Person A] in several instances' with a content-free sentence such as 'The scenario is more involved than it first appears' — and measure $\mathrm{DirErr}$ on the same 300 scenarios. If the directional drift toward 'more' largely disappears, the effect is carried by the asserted premise rather than the mere presence of the word; if it persists, the lexical cue itself is the steering mechanism.

Watch

Extended reading notes

Core claim

The central discovery is that a comparative term in a prompt acts as a directional steering cue: the mere presence of 'more', 'less', or 'equal' reliably shifts a model's predictions toward that label even when the label is false. The evidence comes from 300 controlled comparison scenarios with objectively correct answers, each presented in neutral, direct, and indirect framings, placed at the beginning or the end of the prompt, and answered under one-word, JSON, free-form chain-of-thought, and structured chain-of-thought output formats. A new metric, directional error ($\mathrm{DirErr}$), counts for each label the share of all opportunities on which the model wrongly chooses it; across framing conditions the cue word inflates its own $\mathrm{DirErr}$, with 'equal' framings pushing over-prediction of 'equal' from near zero to as high as 94 percent, and with the smallest models exceeding 90 percent drift in several conditions. Demographic descriptors such as 'a woman' or 'a Black person' amplify the drift despite identical arithmetic, and in some 'less'-framed conditions the demographic priors override the cue and push errors toward 'more'. Chain-of-thought reasoning in free form reduces the bias substantially, but structured JSON reasoning remains susceptible, with models often solving the arithmetic correctly and then casting the correct answer in the cue word's frame.

Load-bearing premise

The indirect framing variants are treated as logically equivalent to the neutral and direct ones, but they insert an asserted comparative fact into the prompt; a model that simply believes that stated premise is not committing a purely lexical framing error, so the measured directional drift mixes true wording effects with rational sensitivity to injected information.

Editorial extensions

If this is right

  • Accuracy-only evaluation is blind to directional drift: two models with the same accuracy can err in opposite directions, so benchmarks should report per-direction error rates.
  • Chain-of-thought is a partial mitigation only: free-form reasoning dampens the drift, but structured JSON reasoning can preserve or reintroduce it, so structured formats need separate auditing.
  • Fairness measurements that ignore framing are confounded: demographic descriptors shift answers more when a comparative cue is present, so identity effects must be measured with framing held fixed.
  • Directional drift shrinks with model size but persists even in the largest tested models, which stay sensitive to 'equal' framings; scaling alone is not a fix.
  • The benchmark doubles as a diagnostic: the $\mathrm{DirErr}$ profile across the fourteen variants exposes each model's specific bias pattern, which aggregate accuracy hides.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The indirect framings assert a comparative fact inside the prompt — '[Person B] spends more time ... in several instances' — so a model that trusts the stated premise is partly responding to new information, not merely to the word; the reported drift is therefore an upper bound on the purely lexical effect.
  • If the steering mechanism is general, the same $\mathrm{DirErr}$ methodology should reproduce in other closed-set answer spaces (yes/no, larger/smaller, high/low), a cheap extension that would test how far beyond 'more', 'less', and 'equal' the phenomenon reaches.
  • The demographic amplification suggests models hold content priors, such as 'women spend more time on caregiving', and that the cue word interacts with those priors; holding the math fixed and varying only the task category would map where the priors are strongest.
  • A practical consequence: prompts that ask for comparisons should state the neutral wording first and avoid echoing any answer class, because even a correct phrasing of the answer inside the prompt pulls the model's output.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MathComp, a benchmark of 300 comparative math scenarios, and evaluates six GPT, Claude, and Qwen models under 14 prompt variants that vary the comparative term (more/less/equal), framing style (neutral/direct/indirect), and framing position. The central metric, DirErr, measures the proportion of incorrect answers that fall in a given direction, and the paper reports directional framing bias, amplification by demographic descriptors, and partial mitigation from chain-of-thought prompting. The dataset and templated code are released.

Significance. If the central claim were cleanly identified, this would be a useful diagnostic resource: the tasks have objective ground truth, labels are symbolically verified, multiple model families are covered, and the directional error metric goes beyond accuracy. The authors also deserve credit for releasing the dataset and templated generation code, which supports reproducibility and follow-up work. However, the current analysis is primarily descriptive: it lacks inferential statistics, a chance baseline, and a control for extra semantic premises in the indirect framing variants. As a result, the quantitative magnitude and the claimed universality of the effect are not yet established.

major comments (4)
  1. [§3.1, Figure 1] The indirect framing variants are not logically equivalent to the neutral or direct variants. In the indirect-more template, the prompt first asserts that Person B spends more time than Person A in several instances before asking the comparison; in the Figure 1 example the gold label is Equal and Person B exceeds Person A in only one of three sub-activities, so the asserted premise is unentailed and in fact false. A model that answers "more" may be deferring to the asserted proposition rather than being steered by the word "more." Since DirErr (Section 4.4) is pooled across direct and indirect variants, the measured drift is a mixture of lexical priming, question-form changes, and premise-acceptance behavior, and the abstract's "mere presence of a comparative term" conclusion is not identified. A control condition that removes the extra proposition while keeping the comparative token is required.
  2. [§4.4, §5, Table 1] No significance tests, confidence intervals, or chance baseline are reported. Under uniform random guessing among the three labels, DirErr(y) has expectation 1/3 (about 33.3%) on the non-gold instances, so values such as Sonnet 3.7's 20.19% for more:Direct (Begin) in Table 1 are below chance, not evidence of drift. The claim in Section 5 that "the mere presence of a comparative term ... reliably biases predictions toward that term" is therefore unsupported without per-condition binomial tests or a multilevel model that accounts for scenario and model. This same gap affects the abstract's "consistent" framing-bias claim.
  3. [§6, Table 1] The demographic-amplification claim is not consistently borne out by the reported table. For Sonnet 3.7, more:Indirect (End) has Std=24.88% but White=11.74%, and neutral (Begin) has Std=26.29% with most demographic groups below that value. The text says the pattern "holds consistently across both Sonnet and GPT-4o-mini," but no interaction test or confidence interval is provided, and cell-level comparisons are noisy without error bars. The nonlinear interaction claim in Section 6 needs a regression or ANOVA-style analysis rather than visual inspection of tables.
  4. [§7, Table 8] The free-form chain-of-thought results are extracted by GPT-4o-mini, which is itself one of the six evaluated models and is shown elsewhere to exhibit strong directional bias (e.g., more:Indirect (End) DirErr=95.77% in Table 11). No human agreement study or validation of the extraction is reported, so the "substantial mitigation" conclusion may be confounded by the extractor's own bias. At minimum, an extraction-quality evaluation or an alternative extractor is needed before the Section 7 mitigation claim can be accepted.
minor comments (5)
  1. [§4.4] The displayed equation for DirErr is referred to as "Eq. 4.4" but has no equation number; please number it.
  2. [References] There are typographical issues in the reference list, such as "V oronov" (should be "Voronov") and "T able 3" in the appendix.
  3. [Table 7] The instruction templates in Table 7 contain inconsistent quoting and formatting around the example JSON object; please clean these up.
  4. [§6, Appendix A.1] The category "African" is used alongside "Black" as a race/ethnicity descriptor without a definition; this conflates a continental/national descriptor with a racial category and should be clarified.
  5. [§4.1] The use of Claude Sonnet 3.7 to generate the dataset and then evaluating the same model is a potential source of dataset bias; this is not fatal because labels are symbolically verified, but it should be acknowledged explicitly in the Limitations section.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: MathComp's directional-bias measurements are empirical, labels are symbolically verified, and the indirect-variant content confound is a validity limitation, not a circular derivation.

full rationale

The paper's central claim is an empirical correlation between comparative prompt terms and model answer distributions, quantified by the DirErr metric over held-out model outputs. No parameter is fitted and then renamed as a prediction, and no result is derived from an assumption that already contains the conclusion. The 'logically equivalent' description of the prompt variants is an input assumption of the experimental design, not a theorem the paper derives; even if the indirect-nore/less prompts inject extra asserted comparative premises (a genuine internal-validity confound, since a model answering 'more' after being told '[Person B] spends more time... than [Person A]' may be deferring to the assertion rather than being lexically steered), the measurements do not force the conclusion by construction: models can and do answer with the gold label, and the paper reports large variation across models, framings, and positions. The gold labels are independently established: the appendix states that 'After validation, we used a Python script to compute final totals for each individual and compare them automatically,' so the labels do not depend on the evaluated models' outputs. The self-involvements—Claude Sonnet 3.7 generating scenarios and also being evaluated, and GPT-4o-mini extracting free-form CoT answers—do not reduce the target result to its inputs: the generator's content is filtered and symbolically verified, and the extractor is used only to read out final labels from rationales, not to define the bias being measured. The cited prior work by an overlapping author (Saffari et al., 2025) is background related work, not a load-bearing uniqueness theorem or an imported ansatz. Accordingly, no circular step is identifiable, and the paper's directional-drift findings are self-contained empirical measurements; the indirect-variant issue should be treated as a correctness/validity risk rather than circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters appear; the analysis is empirical. The load-bearing assumptions are the logical equivalence of the indirect prompts, the representativeness of the 300-scenario dataset for subgroup claims, and the reliability of GPT-4o-mini as a CoT answer extractor.

assumptions (3)
  • domain assumption The indirect framing variants are logically equivalent to the direct and neutral variants despite containing explicit comparative assertions.
    Section 3.1 and Figure 1 treat Indirect 'contextually suggestive' primes (for example, '[Person B] spends more time ... in several instances') as equivalent framings, but they inject new asserted information that a model could rationally use.
  • domain assumption The 300 MathComp scenarios are representative enough to support general claims about framing bias and demographic subgroup effects.
    Section 3 and the Limitations section admit the dataset is small (300), and the demographic subgroup analyses use even smaller subsets.
  • domain assumption GPT-4o-mini can reliably extract labels from free-form CoT responses without introducing its own directional bias.
    Section 4.1 and Table 8 describe using GPT-4o-mini as the answer extractor for free-form CoT, yet this model is also one of the six evaluated models and its own bias in the extraction role is not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of More or Less Wrong: A Benchmark for Directional Bias in LLM Comparative Reasoning." pith.science (2026). https://pith.science/paper/UITEON5U

@misc{pith2026250603923,
  author       = {Pith},
  title        = {Pith review of: More or Less Wrong: A Benchmark for Directional Bias in LLM Comparative Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UITEON5U}},
  note         = {Machine review of arXiv:2506.03923}
}
read the original abstract

Large language models (LLMs) are known to be sensitive to input phrasing, but the mechanisms by which semantic cues shape reasoning remain poorly understood. We investigate this phenomenon in the context of comparative math problems with objective ground truth, revealing a consistent and directional framing bias: logically equivalent questions containing the words ``more'', ``less'', or ``equal'' systematically steer predictions in the direction of the framing term. To study this effect, we introduce MathComp, a controlled benchmark of 300 comparison scenarios, each evaluated under 14 prompt variants across three LLM families. We find that model errors frequently reflect linguistic steering, systematic shifts toward the comparative term present in the prompt. Chain-of-thought prompting reduces these biases, but its effectiveness varies: free-form reasoning is more robust, while structured formats may preserve or reintroduce directional drift. Finally, we show that including demographic identity terms (e.g., ``a woman'', ``a Black person'') in input scenarios amplifies directional drift, despite identical underlying quantities, highlighting the interplay between semantic framing and social referents. These findings expose critical blind spots in standard evaluation and motivate framing-aware benchmarks for diagnosing reasoning robustness and fairness in LLMs.

Figures

Figures reproduced from arXiv: 2506.03923 by the authors.

Figure 1
Figure 1. Comparison of prompt framing effects on response patterns for time-based home maintenance tasks. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Directional error percentages (DirErr %) for one-word answers under framing variation. Each heat-map [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Directional error percentages (DirErr % under chain-of-thought prompting with the framing clause placed [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Directional error percentages (DirErr %) for JSON-formatted answers (the second instruction type) under [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Directional error percentages (DirErr % under chain-of-thought prompting (the third instruction type). [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Directional error percentages (DirErr % under chain-of-thought prompting (the fourth instruction type) [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: DirErr % for sonnet 3.7, the best model on average while including Asian and African races, when the [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: DirErr % for sonnet 3.7, the best model on average while including White and Black races, when the [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: DirErr % for sonnet 3.7, the best model on average while including Hispanic race, when the framing [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: DirErr % for sonnet 3.7, the best model on average while including Woman and Man, when the framing [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: DirErr % for GPT4O-mini on average while including Asian and African races, when the framing [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: DirErr % for GPT4O-mini on average while including White and Black races, when the framing [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: DirErr % for GPT4O-mini on average while including Hispanic race, when the framing variations are [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: DirErr % for GPT4O-mini on average while including Woman and Man, when the framing variations [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

Reference graph

Works this paper leans on

39 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157

    arXiv 2024

  2. [2]

    Bowen Cao, Deng Cai, Zhisong Zhang, Yuexian Zou, and Wai Lam. 2024. On the worst prompt performance of large language models. arXiv preprint arXiv:2406.10248

    arXiv 2024

  3. [3]

    Anastasiia Demidova, Hanin Atwany, Nour Rabih, Sanad Sha’ban, and Muhammad Abdul-Mageed. 2024. John vs. ahmed: Debate-induced bias in multilingual llms. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 193--209

    work page 2024

  4. [4]

    YiTian Ding, Jinman Zhao, Chen Jia, Yining Wang, Zifan Qian, Weizhe Chen, and Xingyu Yue. 2025. https://aclanthology.org/2025.trustnlp-main.36/ Gender bias in large language models across multiple languages: A case study of C hat GPT . In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pages 552--579, Albuquerque, New Mexico. Associati...

    work page 2025

  5. [5]

    James N Druckman. 2001. Evaluating framing effects. Journal of economic psychology, 22(1):91--101

    work page 2001

  6. [6]

    Stephen Flusberg and Kevin J. Holmes. 2024. https://escholarship.org/uc/item/1f0095g2 Linguistic framing in large language models . volume 46

    work page 2024

  7. [7]

    Gallegos, Ryan A

    Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. https://doi.org/10.1162/coli_a_00524 Bias and fairness in large language models: A survey . Computational Linguistics, 50(3):1097--1179

    doi:10.1162/coli_a_00524 2024

  8. [8]

    Jingjing Gong, Yan Zhang, Zheng Yang, Yonghua Huang, Jun Feng, and Weiwei Zhang. 2013. The framing effect in medical decision-making: a review of the literature. Psychology, health & medicine, 18(6):645--653

    work page 2013

Show all 39 references
  1. [9]

    Jiasheng Gu, Hongyu Zhao, Hanzi Xu, Liangyu Nie, Hongyuan Mei, and Wenpeng Yin. 2023. https://doi.org/10.18653/v1/2023.findings-acl.875 Robustness of learning from task instructions . In Findings of the Association for Computational Linguistics: ACL 2023, pages 13935--13948, T...

    2023 doi

  2. [10]

    Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2024. https://openreview.net/forum?id=kGteeZ18Ir Bias runs deep: Implicit reasoning biases in persona-assigned LLM s . In The Twelfth International Conference...

    2024

  3. [11]

    Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398

    2023 arXiv

  4. [12]

    Itay Itzhak, Gabriel Stanovsky, Nir Rosenfeld, and Yonatan Belinkov. 2024. https://doi.org/10.1162/tacl_a_00673 Instructed to bias: Instruction-tuned language models exhibit emergent cognitive bias . Transactions of the Association for Computational Linguistics, 12:771--785

    2024 doi

  5. [13]

    Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki, and Timothy Baldwin. 2024. Evaluating gender bias in large language models via chain-of-thought prompting. arXiv preprint arXiv:2401.15585

    2024 arXiv

  6. [14]

    Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024. https://doi.org/10.18653/v1/2024.acl-long.163 GSM -plus: A comprehensive benchmark for evaluating the robustness of LLM s as mathematical problem solvers . In Proceedings of the 62nd Annual Meeting of the ...

    2024 doi

  7. [15]

    Ruixi Lin and Hwee Tou Ng. 2023. https://doi.org/10.18653/v1/2023.findings-acl.324 Mind the biases: Quantifying cognitive biases in language model prompting . In Findings of the Association for Computational Linguistics: ACL 2023, pages 5269--5281, Toronto, Canada. Association...

    2023 doi

  8. [16]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. https://openreview.net/forum?id=KUNzEQMWU7 Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts . I...

    2024

  9. [17]

    Middleton, and Mahesan Niranjan

    Junyu Mao, Stuart E. Middleton, and Mahesan Niranjan. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.258 Do prompt positions really matter? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4102--4130, Mexico City, Mexico. Association for C...

    2024 doi

  10. [18]

    Marta Marchiori Manerba, Karolina Stanczak, Riccardo Guidotti, and Isabelle Augenstein. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.812 Social bias probing: Fairness benchmarking for language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural ...

    2024 doi

  11. [19]

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. https://doi.org/10.1162/tacl_a_00681 State of what art? a call for multi-prompt LLM evaluation . Transactions of the Association for Computational Linguistics, 12:933--949

    2024 doi

  12. [20]

    Mohammad

    Saif M. Mohammad. 2020. https://doi.org/10.18653/v1/2020.acl-main.702 Gender gap in natural language processing research: Disparities in authorship and citations . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7860--7870, Onl...

    2020 doi

  13. [21]

    Huy Nghiem, John Prindle, Jieyu Zhao, and Hal Daum \'e Iii. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.413 you gotta be a doctor, lin : An investigation of name-based bias of large language models in employment recommendations . In Proceedings of the 2024 Conference on ...

    2024 doi

  14. [22]

    Andreas Opedal, Alessandro Stolfo, Haruki Shirakami, Ying Jiao, Ryan Cotterell, Bernhard Sch\" o lkopf, Abulhair Saparov, and Mrinmaya Sachan. 2024. https://proceedings.mlr.press/v235/opedal24a.html Do language models exhibit the same cognitive biases in problem solving as hum...

    2024

  15. [23]

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. https://doi.org/10.18653/v1/2022.findings-acl.165 BBQ : A hand-built bias benchmark for question answering . In Findings of the Association for ...

    2022 doi

  16. [24]

    Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri. 2025. Benchmarking prompt sensitivity in large language models. In European Conference on Information Retrieval, pages 303--313. Springer

    2025

  17. [25]

    Hamidreza Saffari, Mohammadamin Shafiei, Donya Rooein, Francesco Pierri, and Debora Nozza. 2025. https://aclanthology.org/2025.findings-naacl.337/ Can I introduce my boyfriend to my grandmother? evaluating large language models capabilities on I ranian social norm classificati...

    2025

  18. [26]

    Alejandro Salinas, Amit Haim, and Julian Nyarko. 2024. What's in a name? auditing large language models for race and gender bias. arXiv preprint arXiv:2402.14875

    2024 arXiv

  19. [27]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324

    2023 arXiv

  20. [28]

    Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. https://doi.org/10.18653/v1/D19-1339 The woman worked as a babysitter: On biases in language generation . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9...

    2019 doi

  21. [29]

    Jasivan Sivakumar and Nafise Sadat Moosavi. 2023. https://doi.org/10.18653/v1/2023.acl-long.838 FERMAT : An alternative to accuracy for numerical reasoning . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag...

    2023 doi

  22. [30]

    Jiuding Sun, Chantal Shaib, and Byron C Wallace. 2024. https://openreview.net/forum?id=g9diuvxN6D Evaluating the zero-shot robustness of instruction-tuned language models . In The Twelfth International Conference on Learning Representations

    2024

  23. [31]

    Anton Voronov, Lena Wolf, and Max Ryabinin. 2024. https://doi.org/10.18653/v1/2024.findings-acl.375 Mind your format: Towards consistent evaluation of in-context learning improvements . In Findings of the Association for Computational Linguistics: ACL 2024, pages 6287--6310, B...

    2024 doi

  24. [32]

    Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.243 kelly is a warm person, joseph is a role model : Gender biases in LLM -generated reference letters . In Findings of the Association for C...

    2023 doi

  25. [33]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Proc...

    2022

  26. [34]

    Qian Wu and Han Zheng. 2025. https://doi.org/10.1016/j.jretconser.2025.104250 Consumers' questions as nudges: Comparing the effect of linguistic cues on llm chatbot and human responses . Journal of Retailing and Consumer Services, 84:104250

    2025

  27. [35]

    Ryutaro Yamauchi, Sho Sonoda, Akiyoshi Sannai, and Wataru Kumagai. 2023. Lpml: llm-prompting markup language for mathematical reasoning. arXiv preprint arXiv:2309.13078

    2023 arXiv

  28. [36]

    Zhicheng Yang, Jinghui Qin, Jiaqi Chen, and Xiaodan Liang. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.104 Unbiased math word problems benchmark for mitigating solving bias . In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1401--1408, ...

    2022 doi

  29. [37]

    Jie Zeng, Qianyu He, Qingyu Ren, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, and Fei Yu. 2025. Order matters: Investigate the position bias in multi-constraint instruction following. arXiv preprint arXiv:2502.17204

    2025 arXiv

  30. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.