Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

The Effects of Demographic Instructions on LLM Personas

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM sexism labels skew female; persona prompts don't fix it

desk verdict The female-bias headline is not settled: the alpha computation mixes LLM alignment with demographic-group internal agreement, so the central claim may be a variance artifact; still, this is a useful small study that deserves peer review. read the letter →

arxiv 2505.11795 v1 pith:I744IR65 submitted 2025-05-17 cs.IR

Angel Felipe Magnossão de Paula , J. Shane Culpepper , Alistair Moffat , Sachin Pathiyan Cherumanal , Falk Scholer , Johanne Trippas This is my paper · ORCID
classification cs.IR
keywords sexismdetectionperspectivismlargelanguagemodelsdemographicbiaspersonapromptingannotationdisagreementcontentmoderationKrippendorffalpha
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models can be steered into judging tweets from the perspective of a particular demographic group, and whether those models already carry a demographic slant. Using the EXIST 2023 collection, where each tweet carries labels from six annotators stratified by gender and age, the authors find that five LLMs (GPT-3.5, GPT-4, GPT-4o, Mistral, and Qwen) all agree more with female annotators than with male annotators when deciding whether a tweet is sexist. Age patterns are less consistent, with different models aligning with different age groups. The paper then tests demographic 'persona' instructions that tell the model to adopt a gender and age group, and finds the effects inconsistent and unpredictable: some models move toward the instructed group, others move away. The authors conclude that demographic-based persona prompting cannot be relied on as a debiasing method for sexism classification.

What carries the argument

The load-bearing object is the EXIST 2023 collection, which preserves annotation disagreement: each of 7,958 tweets is labeled by six annotators stratified by gender and age, so a demographic group's perspective can be read directly from its label distribution. The paper's measure is Krippendorff's $\alpha$, an inter-annotator agreement coefficient, computed between each model's labels and each demographic subgroup's labels, with bootstrap confidence intervals below 0.001. The intervention is persona prompting: a prompt that states 'your demographic information is: sex X, age group Y' before asking the model to classify tweets as sexist or not. That setup lets the paper ask whether the instruction moves the model's labels toward the instructed group's labels.

What would settle it

Run the same experiment on a held-out panel of annotators per demographic group, using several phrasings of the same persona instruction; if instructed-persona models improve agreement with their target group more often than they lower it across models and phrasings, the paper's conclusion that demographic personas cannot be relied on is overturned.

Watch

Extended reading notes

Core claim

The central discovery is that LLM-based sexism classification is not perspective-neutral: every one of the five tested models agrees more closely, in Krippendorff's $\alpha$, with the labels of female annotators than with male annotators on the EXIST 2023 sexism-detection task. For example, GPT-3.5 reaches $\alpha=0.415$ against female annotators and $0.371$ against male annotators, while GPT-4o reaches $0.228$ against female and $0.191$ against male annotators. Age-group alignment is model-specific: GPT-3.5 and Mistral align most with the 46+ group, GPT-4 and Qwen with 23–45, and GPT-4o with 18–22. Instructing the model to adopt a demographic persona changes agreement in both directions, improving it for some model–group pairs and worsening it for others, so the paper concludes that demographic-based persona prompting cannot be relied on to mitigate bias.

Load-bearing premise

The load-bearing assumption is that agreement between the model's labels and one demographic subgroup's labels, measured by a single alpha value, is a valid sign both of alignment with that demographic and of whether the model actually adopted the instructed persona; a model could take on a persona and still disagree with the specific, small group of annotators in EXIST.

Editorial extensions

If this is right

  • An LLM-based moderation pipeline that labels sexist tweets will, under these results, reproduce a perspective closer to the female annotators in EXIST, meaning the set of flagged tweets will differ from what a male-leaning system would flag.
  • Adding a demographic instruction is not a reliable way to steer that perspective: outcome changes are model- and group-specific, so persona prompting should not be treated as a debiasing control.
  • Because the age group with highest agreement differs by model, claims about LLM demographic bias on this task need to be model-specific rather than general.
  • When LLMs are used as stand-ins for human annotators, their demographic alignment is part of the measurement and should be reported; otherwise reported quality scores may actually measure closeness to one group's perspective.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the gender finding is that the same female-leaning gap should appear in other subjective classification tasks, such as hate-speech or sentiment labeling; if it does not, the effect may be specific to sexism content rather than a general demographic default.
  • The persona results are tied to one optimized prompt template, so it remains open whether richer persona descriptions, in-context examples, or instruction-tuned variants would produce reliable persona adoption; that would narrow, rather than overturn, the paper's negative conclusion.
  • The authors' framing implies a practical diagnostic the paper does not develop: moderation systems could report which demographic group a model's judgments align with, and treat large deviations from the female-aligned default as a trigger for human review.
  • If the female-aligned default reflects pretraining-data composition rather than something about sexism, then models trained with different data mixes should show different gap sizes; comparing models of the same family trained on different data would test that.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether LLMs exhibit demographic bias when classifying tweets as sexist, and whether demographic persona prompting can mitigate such bias. Using the EXIST 2023 dataset, which contains multiple annotations per tweet stratified by gender and age, the authors compute Krippendorff's alpha between each of five LLMs (GPT-3.5, GPT-4, GPT-4o, Mistral, Qwen) and human annotator subgroups. They report that all five models agree more with female annotators than with male annotators, and that age patterns are inconsistent. They then add gender- and age-specific persona instructions to the prompts and find that the effects on agreement are mixed and unpredictable, concluding that demographic-based persona prompting cannot be relied upon to mitigate bias in sexism classification.

Significance. If the results are correct, the paper provides a valuable empirical result: LLM-based sexism moderation inherits a female-skewed judgment perspective, and simple demographic persona prompting is not a dependable debiasing intervention. The study's strengths include the use of a multi-annotator perspectivist dataset, evaluation across five models spanning both API and open-source families, and explicit bootstrap resampling. However, the central measure of agreement is confounded by within-group annotator agreement, so the headline RQ1 claim requires re-analysis. If the pattern persists after controlling for this confound, the paper would be an important contribution to perspectivist IR and fairness evaluation.

major comments (3)
  1. [Section 3, 'Bias Analysis'; Tables 2 and 3] The Krippendorff's alpha values comparing an LLM to a demographic subgroup are computed on a reliability matrix that contains multiple human annotations per tweet (three female and three male; two per age group). In the standard formulation, the observed disagreement includes pairwise disagreements among the human annotators within the subgroup. If female annotators are more internally consistent than male annotators, alpha_F can exceed alpha_M even when the LLM's labels are no closer to female judgments. The reported cross-group human alpha of 0.477 does not control for this. The authors must recompute alignment using an LLM–human pairwise measure (e.g., average alpha between the LLM and each individual annotator) or a model that separates within-group human disagreement from LLM–human agreement. Without this, the RQ1 claim that LLMs 'exhibit bias toward female annotators' is not supported.
  2. [Section 4, first paragraph; Table 2] The claimed consistency across all five models rests on the confounded alpha values. For example, GPT-3.5 alpha_F = 0.415 versus alpha_M = 0.371, a difference of 0.044. The paper states that all bootstrap confidence intervals are 'smaller than 0.001' but provides no interval values or details on the bootstrap procedure for the differences. The authors should report the confidence intervals for the alpha differences and, more importantly, for the recomputed pairwise measure suggested above, so that the significance of the gender pattern can be assessed.
  3. [Section 1 and Section 5] The statement that the LLMs 'seemed to be incapable of doing so' (adopting personas) is too strong relative to the evidence. The experiments use one specific prompt template and a single set of demographic labels; failure to move agreement with the EXIST annotators could result from a weak persona intervention or from the metric's insensitivity, rather than from a fundamental inability to adopt personas. The paper's more cautious formulation in Section 5 -- 'inconsistent and unpredictable outcomes... cannot be relied upon' -- is defensible and should replace the 'incapable' framing throughout.
minor comments (5)
  1. [Section 3, 'Bias Analysis'] The sentence 'All of the measured Krippendorff's alpha coefficients had confidence intervals smaller than 0.001' is ambiguous; please clarify whether this refers to interval width and report the bootstrap method and at least one concrete interval.
  2. [Section 3, 'Data'] The text says the training and development subsets were merged, but Table 1 reports 7,958 total tweets; please clarify whether this is the train+dev size and how the test set was handled.
  3. [Section 3, 'Prompt Creation'] The prompt development process is unclear: three candidate prompts are first tested on 20 tweets, then the prompt is further optimized with o1-preview; please state explicitly which version was used as the baseline prompt in all experiments.
  4. [Table 2] Please describe how the cross-group human alpha of 0.477 was computed (e.g., treating all six annotators as coders, or aggregating per group); this will help readers interpret the between-group consistency and the subsequent LLM comparisons.
  5. [Table 3 caption] The age-group subscripts in the model names, such as GPT-3.518−22, are visually ambiguous; consider using a clearer notation, for example GPT-3.5[18-22], in the table and text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central agreement measures are computed from fixed LLM outputs against an external human-annotation benchmark, with no fitted parameter or self-citation chain defining the result.

full rationale

The paper's derivation chain is self-contained in the relevant sense. RQ1 measures Krippendorff's alpha between LLM-generated labels and the pre-existing EXIST 2023 demographic subgroup annotations; no model parameter is fitted to the demographic-agreement outcome, and the human labels are external to the LLM prompt construction. RQ2 similarly compares the same external benchmark before and after adding demographic persona text to the prompt. The only prompt-development step selected among three candidates using a 75% inter-model consistency criterion on twenty tweets, which is a prompt-engineering choice, not a fit to the female-vs-male agreement values that support the paper's conclusion. Citations to the authors' prior sexism-detection work and to multi-stage LLM relevance-assessment work are background references and do not carry the load-bearing argument. Concerns about whether Krippendorff's alpha conflates within-group human agreement with LLM-human alignment are measurement-validity critiques, not cases where the derived quantity equals an input by definition or where a fitted parameter is renamed as a prediction. Therefore no circular step is exhibited.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities or fitted constants; its load-bearing assumptions are about dataset representativeness, the validity of the agreement metric, the choice of demographic axes, and the effectiveness of persona prompting as a manipulation.

assumptions (5)
  • domain assumption EXIST 2023 human annotations are a valid source of ground truth for perceived sexism across gender and age groups.
    Used throughout as the benchmark for LLM agreement; if the annotator pool is not representative of the gender or age groups, the bias measurement is externally invalid.
  • standard math Krippendorff's alpha between one LLM label set and a demographic subgroup's labels is a meaningful agreement measure for comparing perspectives.
    The metric is standard, but its use here assumes that comparing a subgroup's aggregate labels to a single model output captures the intended agreement; the paper does not justify the coding scheme or sample size behind the reported alpha values.
  • domain assumption Gender, coded as binary, and age, coded as three bands, are the relevant demographic axes for subjectivity in sexism judgments.
    The study varies only these attributes and does not test education, language, or intersectional identities; the conclusion is scoped to these axes.
  • ad hoc to paper A one-line demographic instruction in the prompt is a sufficient intervention to induce a persona in an LLM.
    The paper assumes persona prompting is an effective manipulation; the null result may instead mean the manipulation is too weak, not that models cannot adopt personas.
  • domain assumption Absence of public ground-truth labels for EXIST 2023 prevents LLM training contamination for this task.
    Stated in the footnote of Section 1; it assumes contamination requires label exposure, but tweets themselves are public and could appear in training data, and the argument conflates label availability with data availability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Effects of Demographic Instructions on LLM Personas." pith.science (2026). https://pith.science/paper/I744IR65

@misc{pith2026250511795,
  author       = {Pith},
  title        = {Pith review of: The Effects of Demographic Instructions on LLM Personas},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I744IR65}},
  note         = {Machine review of arXiv:2505.11795}
}
read the original abstract

Social media platforms must filter sexist content in compliance with governmental regulations. Current machine learning approaches can reliably detect sexism based on standardized definitions, but often neglect the subjective nature of sexist language and fail to consider individual users' perspectives. To address this gap, we adopt a perspectivist approach, retaining diverse annotations rather than enforcing gold-standard labels or their aggregations, allowing models to account for personal or group-specific views of sexism. Using demographic data from Twitter, we employ large language models (LLMs) to personalize the identification of sexism.

Figures

Figures reproduced from arXiv: 2505.11795 by the authors.

Figure 1
Figure 1. Prompt structure. Bold text indicates the parts that were added/varied in the various experiments, as extensions beyond the “baseline” prompt. classifying text as sexist, highlighting an inherent gender-based bias, with GPT-3.5 demonstrating the highest agreement with female annotators, and GPT-4o the lowest. Additionally, as we progress from GPT-3.5 to GPT-4 and subsequently to GPT-4o, we observe a decrease in agre… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Demographic Prompting at Scale: When More Attributes Hurt LLM--Human Agreement

    cs.CL 2026-07 conditional novelty 6.5 of 10

    Across five subjective tasks and five open-source LLMs, demographic prompting improves human agreement only for 1–3 high-signal, directionally coherent attributes and degrades under the full attribute set.

Reference graph

Works this paper leans on

32 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    T. D. Aguda, S. Siddagangappa, E. Kochkina, S. Kaur, D. Wang, and C. Smiley. Large language models as financial data annotators: A study on effectiveness and efficiency. In Proceedings of the 2024 Joint International Conference on Com- putational Linguistics, Language Resources and Evaluation , pages 10124–10145,

    work page 2024

  2. [2]

    Basta, M

    C. Basta, M. R. Costa-jussà, and N. Casas. Evaluating the underlying gender bias in contextualized word embeddings. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing , pages 33–39, 2019. doi: 10.18653/v1/W19- 3805

    doi:10.18653/v1/w19- 2019

  3. [3]

    E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , page 610–623,

    work page 2021

  4. [4]

    Bigdeli, N

    A. Bigdeli, N. Arabzadeh, S. Seyedsalehi, B. Mitra, M. Zihayat, and E. Bagheri. De-biasing relevance judgements for fair ranking. In Advances in Information Retrieval, pages 350–358. Springer Nature Switzerland, 2023

    work page 2023

  5. [5]

    M. R. Costa-jussà, C. Hardmeier, W. Radford, and K. Webster, editors.Proceedings of the First Workshop on Gender Bias in Natural Language Processing . 2019

    work page 2019

  6. [6]

    S. Dai, C. Xu, S. Xu, L. Pang, Z. Dong, and J. Xu. Bias and unfairness in information retrieval systems: New challenges in the LLM era. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , page 6437–6447,

    work page

  7. [7]

    A. F. M. de Paula and R. F. da Silva. Detection and classification of sexism on social media using multiple languages, transformers, and ensemble models. In IberLEF@ SEPLN, 2022

    work page 2022

  8. [8]

    A. F. M. de Paula, R. F. da Silva, and I. B. Schlicht. Sexism prediction in Spanish and English tweets using monolingual and multilingual BERT and ensemble models. arXiv preprint arXiv:2111.04551, 2021

    work page Pith review arXiv 2021

Show all 32 references
  1. [9]

    doi: 10.1145/3637528.3671458

  2. [10]

    Frenda, G

    S. Frenda, G. Abercrombie, V. Basile, A. Pedrani, R. Panizzon, A. T. Cignarella, C. Marco, and D. Bernardi. Perspectivist approaches to natural language process- ing: A survey. Language Resources and Evaluation , pages 1–28, 2024

    2024

  3. [11]

    Hasler, M

    L. Hasler, M. Halvey, and R. Villa. Augmented test collections: A step in the right direction. arXiv preprint arXiv:1501.06370, 2015

    2015 arXiv

  4. [12]

    Faggioli, L

    G. Faggioli, L. Dietz, C. L. A. Clarke, G. Demartini, M. Hagen, C. Hauff, N. Kando, E. Kanoulas, M. Potthast, B. Stein, and H. Wachsmuth. Perspectives on large language models for relevance judgment. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of In...

    2023

  5. [13]

    Kepel and K

    D. Kepel and K. Valogianni. Autonomous prompt engineering in large language models. arXiv preprint arXiv:2407.11000, 2024

    2024 arXiv

  6. [14]

    Krieg, E

    K. Krieg, E. Parada-Cabaleiro, M. Schedl, and N. Rekabsaz. Do perceived gender biases in retrieval results affect relevance judgements? In Advances in Bias and Fairness in Information Retrieval , pages 104–116. Springer International Publishing, 2022

    2022

  7. [15]

    Hutchinson, V

    B. Hutchinson, V. Prabhakaran, E. Denton, K. Webster, Y. Zhong, and S. Denuyl. Social biases in NLP models as barriers for persons with disabilities. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 5491–5501, 2020. doi: 10.1865...

    2020 doi

  8. [16]

    A. Liu, M. Diab, and D. Fried. Evaluating large language model biases in persona- steered generation. arXiv preprint arXiv:2405.20253, 2024

    2024 arXiv

  9. [17]

    Lunardi, D

    R. Lunardi, D. La Barbera, and K. Roitero. The elusiveness of detecting political bias in language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , page 3922–3926, 2024. doi: 10.1145/ 3627673.3680002

    2024

  10. [18]

    Kurita, N

    K. Kurita, N. Vyas, A. Pareek, A. W. Black, and Y. Tsvetkov. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing , pages 166–172, 2019. doi: 10.18653/ v1/W19-3823

    2019

  11. [19]

    Plaza, J

    L. Plaza, J. Carrillo-de Albornoz, R. Morante, E. Amigó, J. Gonzalo, D. Spina, and P. Rosso. Overview of EXIST 2023: sEXism Identification in Social NeTworks. In Advances in Information Retrieval, pages 593–599. Springer Nature Switzerland, 2023

    2023

  12. [20]

    H. A. Rahmani, C. Siro, M. Aliannejadi, N. Craswell, C. L. A. Clarke, G. Faggioli, B. Mitra, P. Thomas, and E. Yilmaz. LLM4Eval: Large language model for eval- uation in IR. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Informatio...

    2024

  13. [21]

    Plaza, J

    L. Plaza, J. Carrillo-de Albornoz, R. Morante, E. Amigó, J. Gonzalo, D. Spina, and P. Rosso. Overview of EXIST 2023 – Learning with disagreement for sexism identification and characterization. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 316–...

    2023

  14. [22]

    Salminen, C

    J. Salminen, C. Liu, W. Pian, J. Chi, E. Häyhänen, and B. J. Jansen. Deus ex machina and personas from large language models: Investigating the composition of AI- generated persona descriptions. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , ...

    2024

  15. [23]

    Call me sexist, but

    M. Samory, I. Sen, J. Kohne, F. Flöck, and C. Wagner. “Call me sexist, but... ”: Revisiting sexism detection using psychological scales and adversarial samples. In Proceedings of the International AAAI Conference on Web and Social Media , volume 15, pages 573–584, 2021

    2021

  16. [24]

    Rodríguez-Sánchez, J

    F. Rodríguez-Sánchez, J. Carrillo de Albornoz, and L. Plaza. Automatic classifi- cation of sexism in social networks: An empirical study on Twitter data. IEEE Access, 8:219563–219576, 2020. doi: 10.1109/ACCESS.2020.3042604

    2020

  17. [25]

    Sclar, Y

    M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr. Quantifying language models’ sensi- tivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324, 2023

    2023 arXiv

  18. [26]

    Thomas, S

    P. Thomas, S. Spielman, N. Craswell, and B. Mitra. Large language models can accurately predict searcher preferences. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , page 1930–1940, 2024. doi: 10.1145/3626772.3657707

    1930

  19. [27]

    J. A. Schnabel, J. Trippas, F. Scholer, and D. Hettiachchi. Multi-stage large language model pipelines can outperform GPT-4o in relevance assessment. In Proceedings of the ACM Web Conference , 2025. doi: 10.1145/3701716.3715488

    2025

  20. [28]

    a helpful assistant

    M. Zheng, J. Pei, L. Logeswaran, M. Lee, and D. Jurgens. When “a helpful assistant” is not really helpful: Personas in system prompts do not improve performances of large language models. InFindings of the Association for Computational Linguistics Conference on Empirical Metho...

    2024 doi

  21. [29]

    Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba. Large language models are human-level prompt engineers. In Proceedings of the 11th International Conference on Learning Representations , 2023. URL https://iclr.cc/ virtual/2023/poster/10850

    2023

  22. [30]

    P. Zhan, Z. Xu, Q. Tan, J. Song, and R. Xie. Unveiling the lexical sensitivity of LLMs: Combinatorial optimization for prompt enhancement. arXiv preprint arXiv:2405.20701, 2024

    2024 arXiv

  23. [2021]

    doi: 10.1145/3442188.3445922

  24. [2024]

    URL https://aclanthology.org/2024.lrec-main.885/

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.