Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that annotator demographics account for only about 8% of the variance in sexism labels, with tweet content dominating, and that persona-prompted LLMs do not reliably reproduce human judgments.

desk verdict Solid empirical work undermined by an unsupported central claim: the 8% demographic-variance number does not follow from the paper's own variance components. read the letter →

arxiv 2507.13138 v2 pith:5S3KLP44 submitted 2025-07-17 cs.CL

Hadi Mohammadi , Tina Shahedi , Pablo Mosteiro , Massimo Poesio , Ayoub Bagheri , Anastasia Giachanou This is my paper · ORCID
classification cs.CL
keywords sexismdetectionannotationreliabilitydemographicbiasmixed-effectsmodelvariancedecompositionpersonapromptingLLMannotationsSHAPexplainability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish where variability in sexism annotations comes from. Fitting a mixed-effects logistic model to a bilingual English/Spanish tweet dataset with six annotators per tweet, it finds that annotator demographics (gender, age, ethnicity, education, region) account for only about 8% of the observed variance in labeling, while tweet-level content dominates, with an intraclass correlation of 92.3%. It then tests whether giving LLMs demographic personas makes their labels closer to human ones; the result is that persona prompting usually fails to help and sometimes hurts. Adding token-level explanations (SHAP) to focus models on content-relevant words is more consistently useful, especially for smaller models. If right, the paper argues, fairness work in sexism detection should emphasize content-driven explanations and robust annotation protocols rather than demographic simulation.

What carries the argument

The central object is a Generalized Linear Mixed Model (GLMM) for binary labels, with crossed random intercepts for annotators and nested random intercepts for tweets within languages, plus inverse-frequency weighting to correct for demographic and label-class imbalance; this is what produces the 8%-vs-content variance split. The second mechanism is SHAP token attribution, which assigns each token an importance score and feeds the top tokens, bolded, into LLM prompts. Persona prompting is the third mechanism, inserting a demographic description into the prompt; the paper compares plain, persona, explainable, and combined scenarios.

What would settle it

Re-estimate the variance partition on the original unweighted labels, or compute the variance explained by the demographic fixed effects directly; if those demographics explain substantially more than $8\%$ (or if removing the rare-group exclusions moves the number well above $8\%$), the paper's central claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is a variance decomposition: in this sexism-detection dataset, demographic features of annotators contribute a statistically detectable but minor share—around 8%—of the variance in binary sexism judgments, whereas the tweet itself is the overwhelming driver (random-intercept variance 33.72 for tweets vs 5.54 for annotators and 0.30 for language; ICC 92.3%). A second claim is that instructing LLMs to adopt demographic personas does not reliably improve agreement with human labels over a plain classification prompt, and sometimes degrades it; guiding the model with SHAP-highlighted content tokens helps more, particularly for smaller models. The paper concludes that content-driven explainability and annotation protocols are a more promising route to fairness than persona simulation.

Load-bearing premise

The 8% claim rests on the assumption that the tweet-level random intercept truly isolates content effects from demographic effects after weighting and filtering, so that the leftover share can be attributed to demographics; if the tweet intercept silently absorbs demographic variance, or the weighting distorts the variance split, the central number changes.

Editorial extensions

If this is right

  • If the 8% figure is right, bias mitigation for sexism detection should concentrate on content and annotation protocol design rather than broad demographic reweighting or correction.
  • Persona prompting cannot be assumed to convert an LLM into a reliable stand-in for a demographic group's perspective; it can slightly reduce accuracy or F1.
  • SHAP-based highlighting of content-relevant tokens is the more robust intervention, with clear gains for small models.
  • The high tweet-level ICC implies that tweet content explains most of the disagreement structure, so aggregation schemes and soft-label training may matter more than demographic controls.
  • LLM annotators may display uniform stylistic patterns, so they cannot yet replace diverse human annotator pools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct pseudo-R-squared computation of the demographic fixed effects on the unweighted data would give a cleaner check on the 8% claim than the reported variance components alone.
  • The finding that coarse demographic personas fail suggests that how a persona is constructed matters: richer, experience-based vignettes might behave differently, but demographic labels alone are unlikely to capture subjective annotation behavior.
  • The variance decomposition implies that disagreement-aware approaches, such as modeling per-annotator perspectives or using soft labels, could be a more productive fairness direction than demographic stratification.
  • Repeating this design on other subjective tasks, such as toxicity or political-bias annotation, would show whether the 8%-level demographic variance and persona ineffectiveness are specific to sexism detection or general to subjective annotation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies variability in sexism-detection annotations from the EXIST 2024 dataset, using a mixed-effects logistic regression to separate demographic fixed effects from tweet-, annotator-, and language-level random effects. It reports that tweet content dominates (ICC 92.3%) and that demographic factors account for about 8% of variance. It then evaluates LLM-based annotation under four prompt scenarios: baseline, demographic persona, SHAP-highlighted tokens, and a combined persona+SHAP condition, across four LLMs. The main findings are that persona prompting rarely improves over baseline, while SHAP highlighting helps smaller models, and that SHAP token analysis shows reliance on content-specific terms. The authors argue that fairness efforts should focus on content rather than demographic simulation.

Significance. If the 8% demographic-variance claim were properly supported, the paper would provide a useful quantitative counterpoint to demographic-correction approaches in annotation. The negative results on persona prompting and the positive results for smaller models under SHAP highlighting are informative, though the latter are compromised by a potential evaluation loop. The paper provides code on GitHub, which supports reproducibility. However, because the central quantitative claim is not backed by a reported variance decomposition and is internally inconsistent with the reported ICC, the paper's headline message cannot be accepted as stated.

major comments (3)
  1. [§4.2 and Abstract] The claim that 'demographic factors account for 8% of the variance' is not supported by the reported variance components. The ICC of 92.3% is consistent with (33.72+5.54+0.30)/(33.72+5.54+0.30+3.29), where 3.29=π²/3 is the latent residual variance for a logit link; the remaining 7.7% is residual error, not demographics. The sentence 'The remaining 8% of the variance is explained by demographic variables and residual error' conflates residual error with demographic fixed effects. Please report a marginal R² or an equivalent variance decomposition that includes the fixed-effect variance (Xβ) to justify the 8%; as written, the number appears to be a misreading of the residual proportion.
  2. [§3.5 and §4.3] The SHAP-based important tokens used to construct GenXAI and GenPXAI prompts appear to be computed on the same test set used to evaluate those scenarios. If so, the highlighted tokens encode test-set label information, creating an evaluation loop that can inflate performance. Specify whether SHAP importance is computed on a training/validation split or on the test split; if the latter, re-run the GenXAI evaluation with tokens derived only from training data.
  3. [§3.2] The weighting and exclusion of rare demographic combinations (removing 69 of 725 annotators and reducing 266 to 56 combinations) can substantially change the variance partition, yet no sensitivity analysis is reported. The 8% claim (or whatever figure results from a correct decomposition) should be checked under alternative weighting and exclusion criteria, since the reported ICC and variance components are estimated on the filtered, weighted sample.
minor comments (5)
  1. [§3.8] The temperature settings used for LLM generation are not reported, despite the abstract and §3.8 mentioning 'various ... temperature settings'. Please list the temperatures used for each model and scenario.
  2. [§4.1] The odds ratios for English (OR = 0.84) and Spanish (OR = 1.95) tweets are not traceable to Table 1 or any reported model coefficient; please clarify the reference level and whether these come from a separate model or from coefficients not shown in the table.
  3. [Appendix D] The token lists include subword fragments (e.g., 'ches', 'stitute', 'ehan'), which suggests tokenizer artifacts; if word-level tokens are intended, please use word-level tokenization or clarify that these are BERT WordPiece tokens.
  4. [Figure 7] The heatmap labels 'TPR' and 'FNR' are confusing, the caption contains a typo ('Comapring' instead of 'Comparing'), and the meaning of the numeric scales is not explained. Please revise the figure and caption.
  5. [§3.3] The Shapley value formula is written with an unclear summation notation 'P T'⊆T\{t}'; the standard definition over all subsets should be stated more precisely to avoid ambiguity.

Circularity Check

2 steps flagged · score 6.0 of 10

The 8% demographic-variance figure is the complement of the random-effects ICC rather than a measured demographic share, and the SHAP guidance is fitted to the evaluation split.

  1. self definitional [Section 4.2, Fixed Effects Interpretation; also Abstract and Section 5.]
    "With an intraclass correlation coefficient (ICC) of 92%, the majority of the variance in labeling outcomes is attributed to tweet-level variability, with language contributing only a minor additional source of variation. The remaining 8% of the variance is explained by demographic variables and residual error."

    The '8%' is constructed as 100% minus the reported ICC. The ICC uses only the random-effect variances (tweet 33.72 + annotator 5.54 + language 0.30) over those plus the logit residual variance pi^2/3 ~ 3.29, giving 92.3%; demographic fixed effects are not part of this decomposition. The 'remaining 8%' is therefore essentially the residual-error share, not an estimate of demographic variance. The paper reports no marginal R^2 or fixed-effect variance partition that would justify assigning the leftover to demographics. Hence the claim that demographic factors account for 8% is a relabeling of 1 minus ICC, not a measured quantity.

  2. fitted input called prediction [Section 3.3 (SHAP token selection) and Section 4.4 (GenAI scenario evaluation).]
    "To find the most influential tokens, we calculate the SHAP importance SIt for each token t by averaging the absolute SHAP values across all instances Nt where the token appears, considering only the cases where the model’s prediction matches the true label ... These tokens are then used in GenXAI and GenPXAI scenarios ... We evaluate our approach on a 10% random sample of the dataset, comprising 326 English texts and 366 Spanish texts, covering all demographic groups."

    The 326-English/366-Spanish set in Section 4.4 is exactly the 10% test split used in Section 4.3, where SHAP importance is computed on 'the remaining 10%' for 'correctly classified YES instances'. The bolded tokens used as GenXAI/GenPXAI prompt guidance are therefore fitted to the true labels and model predictions of the very texts whose classification scores are then reported. The GenXAI versus GenAI comparison is not an independent evaluation of explanation guidance; it measures prompted label leakage. The fitted token highlights are effectively inputs constructed from the evaluation targets and then called a scenario result.

full rationale

The paper is mostly an empirical measurement study, and its self-citations (Mohammadi et al. 2024 for SHAP, Mohammadi et al. 2025 for LLM stylistic uniformity) are not load-bearing. The load-bearing problem is internal. The central claim that demographic factors explain 8% of variance is not supported by an independent fixed-effect variance estimate: the paper's own variance components give ICC = (33.72 + 5.54 + 0.30)/(33.72 + 5.54 + 0.30 + pi^2/3) = 92.3%, so the 'remaining 8%' is the residual/leftover component, which the paper then labels 'demographic variables and residual error' and the abstract reduces to 'demographic factors.' No marginal R^2 or demographic fixed-effect variance share is reported, so the headline number is a definitional complement, not a measured demographic contribution. Separately, the SHAP token highlights used in GenXAI/GenPXAI prompts are computed on the same 10% test split on which those scenarios are evaluated, making the XAI guidance an input fitted to the evaluation targets. These two reductions affect the two main quantitative messages of the paper; however, the underlying GLMM fit, coefficient table, and SHAP token lists remain real empirical outputs, so a middle score is appropriate.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The analysis relies on standard statistical assumptions for GLMMs and task-specific design choices such as the 2% exclusion threshold and SHAP importance threshold. No new theoretical entities are introduced.

free parameters (2)
  • Cumulative importance threshold Tc = 0.95
    Chosen by hand in Section 3.3 to select important tokens for SHAP-guided prompts; results depend on this threshold.
  • Rare-demographic exclusion threshold = 2% of annotators
    Ad hoc cutoff in Section 3.2; 69 of 725 annotators removed, affecting variance estimates.
assumptions (4)
  • standard math Random effects in GLMM are normally distributed with mean zero.
    Standard assumption for mixed models; invoked in Section 3.2.
  • domain assumption The crossed/nested random structure (language > tweet, annotator) correctly represents the data-generating process.
    Assumed in the model specification in Section 3.2.
  • domain assumption Inverse-frequency weighting removes demographic imbalance without biasing variance component estimates.
    Weights are applied to observations in the GLMM; no validation of this assumption is provided.
  • domain assumption SHAP values from the fine-tuned BERT model provide a valid guide to content relevance for LLM prompts.
    Used to select highlighted tokens in GenXAI/GenPXAI scenarios; no evaluation of this transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanation." pith.science (2026). https://pith.science/paper/5S3KLP44

@misc{pith2026250713138,
  author       = {Pith},
  title        = {Pith review of: Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5S3KLP44}},
  note         = {Machine review of arXiv:2507.13138}
}
read the original abstract

Understanding the sources of variability in annotations is crucial for developing fair NLP systems, especially for tasks like sexism detection where demographic bias is a concern. This study investigates the extent to which annotator demographic features influence labeling decisions compared to text content. Using a Generalized Linear Mixed Model, we quantify this inf luence, finding that while statistically present, demographic factors account for a minor fraction ( 8%) of the observed variance, with tweet content being the dominant factor. We then assess the reliability of Generative AI (GenAI) models as annotators, specifically evaluating if guiding them with demographic personas improves alignment with human judgments. Our results indicate that simplistic persona prompting often fails to enhance, and sometimes degrades, performance compared to baseline models. Furthermore, explainable AI (XAI) techniques reveal that model predictions rely heavily on content-specific tokens related to sexism, rather than correlates of demographic characteristics. We argue that focusing on content-driven explanations and robust annotation protocols offers a more reliable path towards fairness than potentially persona simulation.

Figures

Figures reproduced from arXiv: 2507.13138 by the authors.

Figure 1
Figure 1. We instruct LLMs to replicate human annotations for subjective NLP tasks from different perspectives using persona prompting and XAI techniques. Our results show that simulated personas alone may not sufficiently capture human subjectivity. XAI analysis confirms that tweet content plays a more significant role in model decisions. is not a straightforward task since it requires thoughtful consideration of the criteri… view at source ↗
Figure 2
Figure 2. Threshold vs. Number of Selected Tokens in both English and Spanish [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. displays the top 20 tokens by SHAP importance. In English, terms like slut, women, girls, and wife dominate, highlighting the model’s focus on overtly gendered and potentially insulting language. Similarly, in Spanish, tokens such as masculino, mujeres, feminist, mujer, mach, and sexual are highly ranked, indicating a strong reliance on explicit gendered terms and references to sexual characteristics or ideologies. … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: , we observed some variation across cate￾gories in both English and Spanish. For instance, certain groups exhibited broader ranges of unique tokens, potentially hinting at subtle cultural or lin￾guistic factors influencing how they engage with the text. However, consis…
Figure 5
Figure 5. Figure 5: Different demographic combinations that have the highest weight contributions across both label classes [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: The cumulative importance of the top 50 tokens in both English and Spanish. F LLMs Performance Comparison GenAI GenP GenXAI GenPXAI Scenario LLaMA 3.2 3B LLaMA 3.3 70B OpenAI 4o-mini OpenAI GPT-4o Model 0.50 0.47 0.53 0.59 0.66 0.64 0.64 0.65 0.79 0.78 0.78 0.77 0.76 0…
Figure 7
Figure 7. Figure 7: Comapring TPR and FNR across models, scenarios, and languages [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Do Large Language Models Understand Morality Across Cultures?

    cs.CL 2025-07 reject novelty 4.0 of 10

    Small language models compress cross-cultural moral differences, producing more uniformly permissive and less varied judgments than international survey data.

Reference graph

Works this paper leans on

45 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational linguistics, 34(4):555--596

    work page 2008

  2. [2]

    Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, Alexandra Uma, and 1 others. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st workshop on benchmarking: past, present and future, pages 15--21. Association for Computational Linguistics

    work page 2021

  3. [3]

    Tilman Beck, Hendrik Schuff, Anne Lauscher, and Iryna Gurevych. 2024. Sensitivity, performance, robustness: Deconstructing the effect of sociodemographic prompting. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2589--2615

    work page 2024

  4. [4]

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610--623

    2021

  5. [5]

    Laura Biester, Vanita Sharma, Ashkan Kazemi, Naihao Deng, Steven Wilson, and Rada Mihalcea. 2022. Analyzing the effects of annotator gender across nlp tasks. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP@ LREC2022, pages 10--19

    work page 2022

  6. [6]

    Herbert Bless and Klaus Fiedler. 2014. Social cognition: How individuals construct social reality. Psychology Press

    work page 2014

  7. [7]

    Jason Brownlee. 2018. A gentle introduction to early stopping to avoid overtraining neural networks. Machine Learning Mastery, 7

    work page 2018

  8. [8]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186

    2019

Show all 45 references
  1. [9]

    Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can ai language models replace human participants? Trends in Cognitive Sciences, 27(7):597--600

    2023

  2. [10]

    Elizabeth Excell and Noura Al Moubayed. 2021. Towards equal gender representation in the annotations of toxic language detection. GeBNLP 2021, page 55

    2021

  3. [11]

    Pieter Fivez, Walter Daelemans, Tim Van de Cruys, Yury Kashnitsky, Savvas Chamezopoulos, Hadi Mohammadi, Anastasia Giachanou, Ayoub Bagheri, Wessel Poelman, Juraj Vladika, and 1 others. 2024. The clin33 shared task on the detection of text generated by large language models. C...

    2024

  4. [12]

    Gabriel Freedman, Adam Dejl, Deniz Gorur, Xiang Yin, Antonio Rago, and Francesca Toni. 2024. Argumentative large language models for explainable and contestable decision-making. arXiv preprint arXiv:2405.02079

    2024 arXiv

  5. [13]

    Igor Grossmann, Matthew Feinberg, Dawn C Parker, Nicholas A Christakis, Philip E Tetlock, and William A Cunningham. 2023. Ai and the transformation of social science research. Science, 380(6650):1108--1109

    2023

  6. [14]

    Robert M Groves, Floyd J Fowler Jr, Mick P Couper, James M Lepkowski, Eleanor Singer, and Roger Tourangeau. 2011. Survey methodology. John Wiley & Sons

    2011

  7. [15]

    Vipul Gupta, Pranav Narayanan Venkit, Shomir Wilson, and Rebecca J Passonneau. 2024. Sociodemographic bias in language models: A survey and forward path. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 295--322

    2024

  8. [16]

    Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. Annollm: Making large language models to be better crowdsourced annotators. In NAACL (Industry Track)

    2024

  9. [17]

    Human feedback is not gold standard

    Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback is not gold standard. In The Twelfth International Conference on Learning Representations

  10. [18]

    Tiancheng Hu and Nigel Collier. 2024. Quantifying the persona effect in llm simulations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10289--10307

    2024

  11. [19]

    Tiancheng Hu, Yara Kyrychenko, Steve Rathje, Nigel Collier, Sander van der Linden, and Jon Roozenbeek. 2025. Generative language models exhibit social identity biases. Nature Computational Science, 5(1):65--75

    2025

  12. [20]

    a woman is more culturally knowledgeable than a man?

    Mahammed Kamruzzaman, Hieu Nguyen, Nazmul Hassan, and Gene Louis Kim. 2024. " a woman is more culturally knowledgeable than a man?": The effect of personas on cultural norm interpretation in llms. arXiv preprint arXiv:2409.11636

    2024 arXiv

  13. [21]

    Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew M Bean, Katerina Margatina, Rafael Mosquera-Gomez, Juan Ciro, Max Bartolo, Adina Williams, He He, and 1 others. 2024. The prism alignment dataset: What participatory, representative and individualised human feedback ...

    2024

  14. [22]

    Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability

    2011

  15. [23]

    Joseph Kwarteng, Gregoire Burel, Aisling Third, Tracie Farrell, and Miriam Fernandez. 2023. Understanding misogynoir: A study of annotators’ perspectives. In Proceedings of the 15th ACM Web Science Conference 2023, pages 271--282

    2023

  16. [24]

    Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process...

    2021

  17. [25]

    Benedicta Adokarley Lomotey. 2015. On sexism in language and language change--the case of peninsular spanish. Linguistik online, 70(1):167--183

    2015

  18. [26]

    Hariharan Manikandan, Yiding Jiang, and J Zico Kolter. 2023. Language models are weak learners. Advances in Neural Information Processing Systems, 36:50907--50931

    2023

  19. [27]

    Hadi Mohammadi, Anastasia Giachanou, and Ayoub Bagheri. 2024. A transparent pipeline for identifying sexism in social media: Combining explainability with model prediction. Applied Sciences, 14(19):8620

    2024

  20. [28]

    Hadi Mohammadi, Anastasia Giachanou, Daniel L Oberski, and Ayoub Bagheri. 2025. Explainability-based token replacement on llm-generated text. arXiv preprint arXiv:2506.04050

    2025

  21. [29]

    Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677--694

    2019

  22. [30]

    Maja Pavlovic and Massimo Poesio. 2024. The effectiveness of llms as annotators: A comparative overview and empirical analysis of direct representation. LREC-COLING 2024, page 100

    2024

  23. [31]

    Jiaxin Pei and David Jurgens. 2023. When do annotator demographics matter? measuring the influence of annotator demographics with the popquorn dataset. In The 17th Linguistic Annotation Workshop (LAW-XVII)@ ACL 2023, page 252

    2023

  24. [32]

    Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671--10682

    2022

  25. [33]

    Barbara Plank, Dirk Hovy, Anders Sogaard, and 1 others. 2014. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of EACL. Association for Computational Linguistics

    2014

  26. [34]

    Laura Plaza, Jorge Carrillo-de Albornoz, Enrique Amig \'o , Julio Gonzalo, Roser Morante, Paolo Rosso, Damiano Spina, Berta Chulvi, Alba Maeso, and V \' ctor Ruiz. 2024. Exist 2024: sexism identification in social networks and memes. In European Conference on Information Retri...

    2024

  27. [35]

    Alexandra Ralevski, Nadaa Taiyab, Michael Nossal, Lindsay Mico, Samantha N Piekos, and Jennifer Hadlock. 2024. Using large language models to annotate complex cases of social determinants of health in longitudinal clinical records. medRxiv

    2024

  28. [36]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “why should i trust you?” explaining the predictions of any classifier . In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135--1144

    2016

  29. [37]

    Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman. 2008. Labelme: a database and web-based tool for image annotation. International journal of computer vision, 77:157--173

    2008

  30. [38]

    a fer, Aidan Combs, Christopher Bagdon, Jiahui Li, Nadine Probol, Lynn Greschner, Sean Papay, Yarik Menchaca Resendiz, Aswathy Velutharambath, Amelie W \

    Johannes Sch \"a fer, Aidan Combs, Christopher Bagdon, Jiahui Li, Nadine Probol, Lynn Greschner, Sean Papay, Yarik Menchaca Resendiz, Aswathy Velutharambath, Amelie W \"u hrl, and 1 others. 2024. Which demographics do llms default to during annotation? arXiv preprint arXiv:2410.08820

    2024 arXiv

  31. [39]

    Narjes Tahaei and Sabine Bergler. 2024. Analysis of annotator demographics in sexism detection. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 376--383

    2024

  32. [40]

    Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2020. A case for soft loss functions. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pages 173--177

    2020

  33. [41]

    Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385--1470

    2021

  34. [42]

    Angelina Wang, Jamie Morgenstern, and John P Dickerson. 2024. Large language models cannot replace human participants because they cannot portray identity groups. CoRR

    2024

  35. [43]

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2):1--38

    2024

  36. [44]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  37. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.