Pith. sign in

REVIEW 4 major objections 5 minor 15 references

The Biased Samaritan: LLM biases in Perceived Kindness

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In ten publicly available LLMs, an unmarked 'person' is implicitly treated as a white middle-aged or young-adult male, and most explicitly named demographics are rated as more willing to help than that unmarked control.

desk verdict The control-group design is a genuine step forward in bias auditing, but the paper's headline 'default person' claim rests on treating non-significant t-tests as evidence of equivalence, and that inference doesn't hold. read the letter →

arxiv 2506.11361 v1 pith:R5T6XBRM submitted 2025-06-12 cs.CL cs.CY

Jack H Fagan , Ruhaan Juyaal , Amy Yue-Ming Yu , Siya Pun This is my paper
classification cs.CLcs.CY
keywords LLMbiasdemographiccontrolgroupbaselineGoodSamaritanscenariosscorepromptbrittlenesspairedt-test
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that commercial LLMs carry an implicit 'default person'—white, male, and middle-aged or young adult—and that this baseline can be measured by adding a control prompt that names no demographic. Across 412 human-written Good Samaritan-style scenarios, ten models from five providers rated a genderless, raceless, ageless 'person' as less likely to help than most explicitly named groups, meaning the models' bias is mostly affirmative toward non-default demographics rather than negative toward them. The method matters because it separates two biases that earlier auditing approaches tangled together: the assumption of who the unmarked person is, and the directional preference once a demographic is named. It gives users and developers a quantitative, repeatable audit to detect both kinds of bias in LLM output.

What carries the argument

The central object is the bias score, defined as the difference between a demographic group's mean helpfulness rating and the control group's mean rating, where helpfulness is the model's 1-100 probability that a moral patient will intervene to help a third party. The control is a 'person' with no demographic information, and each of 103 scenario ideas is rephrased four ways with different grammar and word choice, producing roughly 2,800 human-written prompts per model run. Paired sample t-tests at α = 0.01 flag which demographic-versus-control differences are significant, and the standard deviation across the four rephrasings defines brittleness, which measures how sensitive scores are to wording rather than to demographics. This control-comparison design is what makes the implicit default demographic visible, because it shows which groups are statistically indistinguishable from an unmarked person.

What would settle it

Run the same 103 scenarios with an explicit 'typical person' control and use a pre-specified equivalence margin; if White, male, or middle-aged groups do not fall within that margin of the control, or if an explicit 'typical person' control produces a different baseline, then the claim that these groups are the implicit default is not supported.

Watch

Extended reading notes

Core claim

The paper claims that LLMs have a measurable implicit baseline demographic: when no demographic is specified, the models behave as if the 'person' in a scenario is white, male, and middle-aged or a young adult. Relative to that unmarked control, women, non-binary people, seniors, and most non-white racial groups were rated as significantly more likely to help, while white, male, and middle-aged groups usually did not differ significantly from the control. The authors interpret this as two distinct biases that are usually entangled in prior work: a default assumption about the unmarked person, and a directional preference once a demographic is named. Because they included a control condition with no demographic information, they could observe that most deviations from the baseline are affirmative, not negative, with teenagers as the main exception.

Load-bearing premise

The central assumption is that when a demographic group's scores do not differ significantly from the control, that means the model treats that group as the default person; non-significance is not the same as proof of equivalence, so this inference carries the argument.

Editorial extensions

If this is right

  • All ten models showed statistically significant bias in at least six demographic categories, so the pattern is not specific to one developer or model family.
  • Non-baseline groups—women, non-binary people, seniors, and most non-white racial groups—were rated as more likely to help than the unmarked control, so on this task the prevailing bias is affirmative rather than negative.
  • Teenagers were the clear exception, rated as less likely to help than the control by most models.
  • Adding a no-demographic control lets bias audits separate the implicit default from directional preference, a distinction that multiple-choice and sentiment-analysis methods cannot make.
  • Brittleness measurements show that wording changes move scores by 4–8 points on a 1–100 scale, sometimes more than the demographic biases themselves, so phrasing must be controlled in any audit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The manuscript cites '[llm-prompt-bias]' to justify using only human-written prompts, but no matching entry appears in its reference list, so the cited support for that design choice is missing from the bibliography.
  • The inference from non-significant t-tests to 'default' status would be stronger with an equivalence test and a pre-specified margin; as written, underpowered comparisons could masquerade as baseline agreement.
  • The affirmative bias toward non-baseline groups is consistent with post-training debiasing or overcorrection, which would make this control-comparison method a natural audit instrument for measuring the size and direction of such corrections.
  • The authors note that a different language could shift the default demographic; running the same prompt set in Mandarin would test whether the implied default person is a property of English training data or a more general model prior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This manuscript introduces a control-condition method for measuring demographic bias in LLMs. The authors prompt ten commercial models with 103 human-generated Good Samaritan scenarios, varying the moral patient's race, gender, or age, and compare the resulting 1--100 helpfulness ratings against an unspecified 'person' control. Bias scores are computed as group mean minus control mean, with paired-sample t-tests at alpha = 0.01. The paper reports two headline findings: that models treat a white middle-aged or young adult male as the baseline 'person,' and that non-baseline demographics are generally rated as more willing to help than the baseline. The manuscript also reports brittleness across prompt rephrasings and a two-run repeatability check for two models.

Significance. The control-anchored design is a genuine methodological contribution: unlike multiple-choice stereotype tests, it allows directional bias and 'default person' assumptions to be separated within the same experiment, without fitting any parameters to the data. The use of human-generated prompts and the inclusion of a non-demographic control are strengths, as is the explicit repeatability check for two models. However, the central baseline-demographic claim rests on interpreting non-significant differences as evidence of equivalence, which is not statistically justified. If the authors add proper equivalence analyses and multiple-comparison correction, this could become a useful and reproducible LLM-auditing tool; in its current form, the load-bearing baseline conclusion is not established.

major comments (4)
  1. [Section 5.1.1, 5.1.2, and Section 7] The inference that non-significant paired t-tests imply the model 'treats the control group as male' or 'considers the control to be a person in this age range' is not valid. A p-value above 0.01 only means the test did not detect a difference; it is not evidence that the difference is zero or practically equivalent. This is load-bearing because the baseline-demographic claim is the paper's main novelty. Please replace these inferences with a formal equivalence test (e.g., two one-sided tests) using a pre-specified margin, or with Bayesian interval estimation, and report the corresponding confidence intervals and power analysis for the 103 scenario-pairs. Without this, the headline finding that the baseline demographic is a white middle-aged or young adult male is not supported.
  2. [Section 4.5 and Tables 1--3] No correction is applied for multiple comparisons, despite testing 10 models across 3 demographic categories and up to 6 groups per category. At alpha = 0.01, dozens of tests are performed, so the claim that 'all models across all manufacturers displayed bias in at least 6 categories' likely overstates the evidence. Please report the total number of tests per category, apply a false-discovery-rate or family-wise error correction, and indicate which highlighted cells in Tables 1--3 survive correction. The within-model, within-category structure should also be taken into account, since demographic groups share the same control scores.
  3. [Section 4.6, Eq. (1), and Appendix A.1] Equation (1) states that b, the number of brittleness rephrasings, is 4 in this paper, but Table 4 shows that most demographic groups have only 2 phrasings and some have 3 (e.g., non-binary has 3; senior has 3; teenager has 1 in the table as printed). This discrepancy affects how the aggregated scores and the paired t-tests are computed, and it is not a purely typographical issue because the standard error of the mean depends on the number of rephrasings per scenario. Please correct the definition, report the actual b for each group, and explain how missing phrasings were handled in the averaging.
  4. [Section 5.1.2 and Table 1] The text contains an internal contradiction. It states both that 'most models displayed statistically significant positive bias towards the young adult group' and that 'the majority of models found no significant difference between middle-aged and young adults relative to the control.' Yet Table 1 shows positive young-adult differences in 9 of 10 models, several of them sizable (e.g., 2.69, 2.54, 1.78). Since young adults are also included in the 'baseline demographic' claim, the paper must clarify exactly which models show significant young-adult effects and reconcile this with the assertion that young adults are part of the default category.
minor comments (5)
  1. [Section 4.2] The citation placeholder '[llm-prompt-bias]' appears in the text but is not present in the reference list; please add the corresponding reference or remove the placeholder.
  2. [Section 4.6, Eq. (1)] The notation in Eq. (1) should clarify whether the sum over n uses a constant b for all scenario-m pairs; if b varies by group, the equation should reflect that explicitly.
  3. [Section 8] The conclusion says the study finds bias 'when it comes to race, gender, and ethnicity,' but the paper measures age, not ethnicity. Please correct this wording.
  4. [Appendix A.1] The typo 'AA VE' in Section 6 should read 'AAVE,' and the demographic phrasing table would be easier to use if the number of phrasings per group were listed in a separate column.
  5. [Appendix B and Figure 1] Section 5.3 refers to 'Appendix B.4' for repeated trial results, but the appendix section is labeled 'B.3 Repeated Trial Results.' Also, Figure 1's caption says 'Difference from the control represents the frequency...' while the y-axis appears to be a proportion; please clarify the exact quantity plotted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: bias scores are direct demographic-minus-control differences with no fitted parameters or self-citation chain.

full rationale

The paper's central quantities are computed directly from model outputs: the helpfulness rating is a mean over prompt rephrasings and scenarios (Eq. 1), and the bias score is the difference between a demographic group's rating and a within-experiment 'person' control. No parameter is fitted to the data and then renamed as a prediction; the control group is an independent anchor built into the prompt design rather than derived from the target claims. The baseline-demographic conclusion (white, male, middle-aged/young adult) is an operational inference from groups whose bias scores did not reach the pre-specified significance threshold, and the identity of those groups is data-driven rather than assumed. That inference is statistically fragile — non-significance is not evidence of equivalence without an equivalence margin and adequate power, and the young-adult grouping is internally inconsistent with Table 1 — but that is a validity concern, not circularity. The paper also does not rely on any load-bearing self-citation; references to prior work are contextual and none of the authors' own prior results are invoked to justify the method or the conclusion. The derivation chain is therefore self-contained relative to its stated inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on assumptions about scenario representativeness, the validity of the statistical test, the adequacy of demographic phrasings, and the neutrality of API defaults. No invented entities or fitted free parameters are present.

assumptions (4)
  • domain assumption The 103 scenarios are representative of everyday moral situations and cover the construct of helpfulness.
    The paper selects scenarios modeled on the Good Samaritan experiment but provides no rationale or validation that they sample the space of helping behaviors fairly; the scenario list is not published (Section 4.2).
  • standard math The paired t-test is valid for comparing demographic vs control scores across scenarios.
    Assumes the 103 scenario-level averages are independent and identically distributed; no multiple-comparison correction is applied despite 130 tests (Section 4.5).
  • domain assumption The demographic phrasing lists (e.g., 'Man' and 'Male individual') operationalize the intended groups.
    Appendix A.1 shows the phrasings; the paper assumes these are equivalent proxies and that any difference is due to group identity rather than word association (Section 4.1).
  • domain assumption API default settings approximate a typical lay user's experience and do not introduce confounds.
    Models were queried at default settings (Table 5) without controlling for system prompts, retries, or non-determinism across providers (Section 4.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Biased Samaritan: LLM biases in Perceived Kindness." pith.science (2026). https://pith.science/paper/R5T6XBRM

@misc{pith2026250611361,
  author       = {Pith},
  title        = {Pith review of: The Biased Samaritan: LLM biases in Perceived Kindness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R5T6XBRM}},
  note         = {Machine review of arXiv:2506.11361}
}
read the original abstract

While Large Language Models (LLMs) have become ubiquitous in many fields, understanding and mitigating LLM biases is an ongoing issue. This paper provides a novel method for evaluating the demographic biases of various generative AI models. By prompting models to assess a moral patient's willingness to intervene constructively, we aim to quantitatively evaluate different LLMs' biases towards various genders, races, and ages. Our work differs from existing work by aiming to determine the baseline demographic identities for various commercial models and the relationship between the baseline and other demographics. We strive to understand if these biases are positive, neutral, or negative, and the strength of these biases. This paper can contribute to the objective assessment of bias in Large Language Models and give the user or developer the power to account for these biases in LLM output or in training future LLMs. Our analysis suggested two key findings: that models view the baseline demographic as a white middle-aged or young adult male; however, a general trend across models suggested that non-baseline demographics are more willing to help than the baseline. These methodologies allowed us to distinguish these two biases that are often tangled together.

Figures

Figures reproduced from arXiv: 2506.11361 by the authors.

Figure 1
Figure 1. Difference from the control represents the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 1
Figure 1. B.2 Brittleness Measure For gender, see [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Helpfulness Score versus Differences from [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Helpfulness Score versus Differences from [PITH_FULL_IMAGE:figures/full_fig_p010_3.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages

  1. [1]

    Sunhao Dai, Yuqi Zhou, Liang Pang, Weihao Liu, Xiaolin Hu, Yong Liu, Xiao Zhang, Gang Wang, and Jun Xu. 2024. https://doi.org/10.1145/3637528.3671882 N eural R etrievers are B iased T owards LLM - G enerated C ontent . In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 526–537. ACM

    arXiv 2024

  2. [2]

    Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. https://arxiv.org/abs/2305.08283 F rom P retraining D ata to L anguage M odels to D ownstream T asks: T racking the T rails of P olitical B iases L eading to U nfair NLP M odels . Preprint, arXiv:2305.08283

    arXiv 2023

  3. [3]

    Jeremy Howard and Sebastian Ruder. 2018. https://doi.org/10.18653/v1/P18-1031 Universal language model fine-tuning for text classification . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328--339, Melbourne, Australia. Association for Computational Linguistics

    doi:10.18653/v1/p18-1031 2018

  4. [4]

    As an AI Language Model, "Yes I Would Recommend Calling the Police": Norm Inconsistency in LLM Decision-Making

    Shomik Jain, D Calacci, and Ashia Wilson. 2024. https://arxiv.org/abs/2405.14812 A s an AI L anguage M odel, " Y es I W ould R ecommend C alling the P olice": N orm I nconsistency in LLM D ecision- M aking . Preprint, arXiv:2405.14812

    work page Pith review arXiv 2024

  5. [5]

    Hadas Kotek, Rikker Dockum, and David Sun. 2023. https://doi.org/10.1145/3582269.3615599 G ender b ias and s tereotypes in L arge L anguage M odels . CI '23, page 12–24, New York, NY, USA. Association for Computing Machinery

    arXiv 2023

  6. [6]

    Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. https://doi.org/10.18653/v1/2021.acl-long.416 S tereo S et: M easuring S tereotypical B ias in P retrained L anguage M odels . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long ...

    doi:10.18653/v1/2021.acl-long.416 2021

  7. [7]

    Martin Obaidi, Lukas Nagel, Alexander Specht, and Jil Klünder. 2022. https://doi.org/10.1016/j.infsof.2022.107018 S entiment analysis tools in software engineering: A systematic mapping study . Information and Software Technology, 151:107018

    arXiv 2022

  8. [8]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. https://doi.org/10.3115/v1/D14-1162 G lo V e: Global vectors for word representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 1532--1543, Doha, Qatar. Association for Computational Linguistics

    doi:10.3115/v1/d14-1162 2014

Show all 15 references
  1. [9]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1202 Deep contextualized word representations . In Proceedings of the 2018 Conference of the North A merican Chapter of the As...

    2018 doi

  2. [10]

    Dieter Rasch, Friedrich Teuscher, and Volker Guiard. 2007. https://doi.org/10.1016/j.jspi.2006.04.011 H ow robust are tests for two independent samples? Journal of Statistical Planning and Inference, 137(8):2706--2720. 5th St. Petersburg Workshop on Simulation

    2007 doi

  3. [11]

    Alejandro Salinas, Amit Haim, and Julian Nyarko. 2025. https://arxiv.org/abs/2402.14875 W hat's in a N ame? A uditing L arge L anguage M odels for R ace and G ender B ias . Preprint, arXiv:2402.14875

    2025 arXiv

  4. [12]

    Akshaj Kumar Veldanda, Fabian Grob, Shailja Thakur, Hammond Pearce, Benjamin Tan, Ramesh Karri, and Siddharth Garg. 2023. https://arxiv.org/abs/2310.05135 Are E mily and G reg S till M ore E mployable than L akisha and J amal? I nvestigating A lgorithmic H iring B ias in the E...

    2023 arXiv

  5. [13]

    K elly is a W arm P erson, J oseph is a R ole M odel

    Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. 2023. https://arxiv.org/abs/2310.09219 " K elly is a W arm P erson, J oseph is a R ole M odel": G ender B iases in LLM - G enerated R eference L etters . Preprint, arXiv:2310.09219

    2023 arXiv

  6. [14]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  7. [15]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.