Pith. sign in

REVIEW 4 major objections 5 minor 37 references

Dutch CrowS-Pairs: Adapting a Challenge Dataset for Measuring Social Biases in Language Models for Dutch

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Dutch CrowS-Pairs benchmark shows English models carry the most social bias.

desk verdict A genuinely useful Dutch CrowS-Pairs resource, but the cross-lingual bias claims are confounded and should not be taken at face value. read the letter →

arxiv 2507.16442 v1 pith:7KCH5VN2 submitted 2025-07-22 cs.CL

Elza Strazda , Gerasimos Spanakis This is my paper
classification cs.CL
keywords CrowS-PairsDutchsocialbiaslanguagemodelsevaluationpersonapromptingcross-lingualmasked
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Dutch CrowS-Pairs, a Dutch-language adaptation of the CrowS-Pairs challenge dataset: 1,463 sentence pairs across nine bias categories, each contrasting a stereotyped sentence with a minimally different anti-stereotyped counterpart. Using it, the authors measure how often Dutch, English, and French language models prefer the stereotyped sentence. They report that all models exhibit measurable bias, that English masked models score highest, French models sit in the middle, and Dutch models score lowest, and that prompting autoregressive models to adopt a bad persona raises bias scores while a good persona lowers them. The contribution is a public non-English benchmark plus a cross-lingual comparison that ties bias to training data, architecture, language, and prompt context.

What carries the argument

The measuring device is CrowS-Pairs' contrastive-pair protocol: each item pairs a stereotyped sentence with a minimally different anti-stereotyped counterpart, and the bias score is the fraction of pairs in which a model prefers the stereotyped sentence, with 50 percent defined as neutrality. Masked models are scored by pseudo-log-likelihood: each token is masked in turn and the model's probability for the original token is recorded, then aggregated into a per-model bias percentage. Autoregressive models are scored by prompting them to choose which of the two sentences is more likely. The adaptation machinery is translation with cultural localization: replacing U.S.-specific groups and units, localizing names, deleting 45 culturally irrelevant pairs, and repairing non-minimal pairs, double switches, and category mismatches so each Dutch pair contrasts on one bias dimension.

What would settle it

Evaluate all three language families on the intersection of semantically identical sentence pairs, or on literal translations of the same pairs, and check whether the English-over-French-over-Dutch ranking survives; if the ordering changes, the cross-lingual conclusion is an artifact of dataset content rather than model behavior.

Watch

Extended reading notes

Core claim

The paper's central claim is that the adapted dataset is a workable instrument for quantifying social bias in Dutch language models, and that measuring with it yields a ranked picture: among masked models, English ones are most biased (RoBERTa 65.14, BERT 61.45), French models are intermediate (CamemBERT 58.30, FlauBERT 55.02), and Dutch models are least biased (BERTje and RobBERT 54.82, multilingual BERT 52.43), with RoBERTa-based architectures consistently more biased than BERT-based ones. For autoregressive models, GEITje scores higher in its baseline form (85.03) than its base model Mistral-7B (59.67), and persona framing changes both dramatically, with Mistral falling to 22.21 under a good persona and rising to 94.46 under a bad persona. These results are presented as evidence that bias in language models is shaped by architecture, training corpus size and diversity, language, and prompt context.

Load-bearing premise

The Dutch, French, and English versions of CrowS-Pairs are comparable benchmarks, so that differences in bias scores reflect differences in models or languages rather than differences in dataset content.

Editorial extensions

If this is right

  • Dutch-language bias evaluation now has a public benchmark with category-level scores, so future Dutch models can be compared against BERTje, RobBERT, multilingual BERT, GEITje, and Mistral-7B on the same items.
  • The English-over-French-over-Dutch ordering, if taken at face value, implies that debiasing cannot be language-agnostic: training data and cultural context are part of what a bias score measures.
  • The persona results indicate that a model's bias score depends on prompt framing, so bias reports should state the prompt conditions used; the same model can appear strongly or weakly biased depending on the role assigned.
  • Because RoBERTa-based models outscore BERT-based models in both English and French, architecture and training-corpus scale are treated as predictors of stereotype preference, not just language.
  • Category scores should be read with caution, since the paper inherits known CrowS-Pairs quality issues such as non-minimal pairs and category mismatches despite its repair attempts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An alignment test the paper does not run would restrict all three languages to the intersection of semantically identical pairs; the reported cross-lingual ranking should be treated as provisional until that comparison is performed.
  • The persona effect could be used as an inference-time debiasing lever, but it also means bias scores are not a fixed property of a model, so claims about less-biased models should be scoped to a specified prompting setup.
  • The adaptation shows that direct translation without cultural rewrites would produce artefactual scores; one testable extension is measuring the same Dutch models on literal translations versus adapted pairs to quantify how much cultural adaptation changes the score.
  • A related resource gap is a Dutch metric for open-ended generative outputs, since the prompt-selection protocol used here is only one estimator of model bias.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Dutch CrowS-Pairs, a Dutch adaptation of the English CrowS-Pairs benchmark consisting of 1,463 sentence pairs across nine bias categories. It evaluates masked language models (BERTje, RobBERT, multilingual BERT, BERT, RoBERTa, FlauBERT, CamemBERT) using pseudo-log-likelihood scoring and evaluates two autoregressive models (GEITje, Mistral-7B) using prompt-based choice tasks, including persona manipulations. The headline findings are that English masked language models exhibit more bias than French or Dutch models, that RoBERTa-style models are generally more biased than BERT-style models, and that assigning a "bad" persona increases bias while assigning a "good" persona decreases it.

Significance. If the dataset is valid, Dutch CrowS-Pairs is a valuable resource because most bias benchmarks are English-only and the authors provide both the data and the evaluation scripts. The adaptation process is documented in useful detail, including explicit examples of repaired non-minimal pairs, double switches, and bias mismatches. The cross-lingual comparison is potentially informative but is not currently established because the models are scored on different language-specific versions of the benchmark. The persona results are consistent with prior work and are mainly useful as a demonstration that prompt framing strongly affects these measurements. Overall, the contribution is a promising dataset and an honest report of its construction, but several load-bearing claims require additional analysis or reframing.

major comments (4)
  1. [3.2, Section 4.1, Table 3] The central claim that "English models show the most bias overall, followed by French and Dutch models" is not supported by the reported experiments because each model is scored on a different version of CrowS-Pairs. Section 3.2 states that BERT and RoBERTa are evaluated on the revised English set, CamemBERT and FlauBERT on the French set, and BERTje, RobBERT, and multilingual BERT on the newly adapted Dutch set. Section 3.1 reports that the Dutch set removed 45 pairs, replaced U.S.-specific references, re-labeled categories, and repaired non-minimal pairs, double switches, and bias mismatches. Since Equation 2 is a raw count ratio over the language-specific item sets, a score of 50 is not calibrated across these different versions, and the lower Dutch scores may reflect dataset content rather than model or language. The cross-lingual comparison should either be restricted to within-dataset interpretations, or the authors should construct a matched subset of comparable items across the three languages and re-run the evaluation.
  2. [3.1, Table 1] The validity of the Dutch dataset as a bias benchmark is not sufficiently established. The translation and adaptation process relies on review by two native speakers, but no inter-annotator agreement, adjudication protocol, or quantitative reliability check is reported. This is load-bearing because the authors changed category labels and stereotype direction (e.g., the "Mexican" to "Moroccan" substitution and the "White" to "rich" substitution), so the stereotype orientation of individual pairs is a matter of subjective judgment. The paper's own limitation statement in Section 5 concedes that sentence quality and category consistency are inherited issues; without reliability evidence, the dataset should be described as a candidate resource rather than a validated benchmark.
  3. [3.3, Section 4.2, Table 4] The autoregressive language model evaluation lacks the experimental detail needed to assess robustness. The authors do not report decoding parameters (temperature, top-p, max tokens), seeds, number of repetitions, how invalid or unparseable model outputs were handled, or whether the order of "Zin A" and "Zin B" was counterbalanced. Because the persona effects are very large (e.g., Mistral moving from 59.67 to 22.21 under a "good" persona) and the text uses the word "significantly" without any statistical test, it is impossible to tell whether these results reflect stable model behavior, prompt-order artifacts, or decoding variance. The authors should provide the full generation protocol and, ideally, confidence intervals or repeated-run summaries.
  4. [Equation 2, Section 4.1] The bias scores are reported as point estimates without uncertainty quantification. Each score is a proportion over at most 1,463 pairs, and differences of a few percentage points (e.g., BERTje 54.82 vs. multilingual BERT 52.43) are treated as meaningful in the discussion. The claim that "a model that does not incorporate any stereotypes should achieve a bias score of 50" also assumes that the item set is balanced in difficulty, which is not demonstrated. Adding bootstrap confidence intervals or McNemar-style tests over the sentence pairs would clarify which cross-model and cross-category differences are actually reliable.
minor comments (5)
  1. [3.2, Equation 1] The sentence defining M says "M represents the unmodified tokens," but the surrounding text and equation indicate M should be the modified tokens; this typo should be corrected.
  2. [4.2] The statement that "Both models are most biased in Gender, Socioeconomic status, and Sexual orientation" does not match Table 4 for the baseline condition, where GEITje is highest in Physical appearance (88.89) and Mistral is highest in Socioeconomic status (66.08); the categorical summary should be reconciled with the table.
  3. [4.2] The phrase "scores dropping significantly" is used without a significance test; replace it with a quantitative description such as "dropping to X" or add statistical support.
  4. [Table 3] The caption says the highest score in each model language group is in bold and the overall highest in each row is underlined, but these visual cues are lost in the plain-text rendering; also clarify whether "Race/Color" and "Socioeconomic status" are the official category names or whether they should be hyphenated consistently with the dataset files.
  5. [3.2] The notation in Equation 1 is unclear: "|C|X" appears to be a typographical artifact for a summation over the token count, and the sentence "one modified token at a time is masked until all ui are masked" should be rewritten to state precisely which tokens are masked and how the pseudo-log-likelihood is accumulated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: bias scores are external benchmark measurements; the cross-lingual comparison is a validity confound, not a construction-level circularity.

full rationale

The derivation chain is an empirical measurement pipeline: the authors construct/adapt a benchmark (Dutch CrowS-Pairs), define the bias score as Nmore/Ntotal (Equation 2), and apply off-the-shelf pretrained models to obtain scores. No parameter is fitted to the benchmark outputs and then reported as a prediction; no equation defines the dataset labels in terms of the model scores. The main weakness is the cross-lingual comparison in Table 3: Section 3.2 states that BERT/RoBERTa are evaluated on revised English CrowS-Pairs, CamemBERT/FlauBERT on French CrowS-Pairs, and BERTje/RobBERT/mBERT on the newly adapted Dutch CrowS-Pairs, while Section 3.1 reports that 45 pairs were removed, U.S.-specific references were culturally substituted, and categories were relabelled or repaired. This makes the 'English most, Dutch least' headline dependent on dataset comparability, but that is a construct-validity/confound issue, not circularity: the scores would be computed the same way regardless of the paper's claims, and no step reduces to its own input by definition. The only self-citation (Chávez Mulsa and Spanakis, 2020) appears in related work as background and is not load-bearing. The conclusion's acknowledged limitation—'issues with sentence quality and category consistency inherited from the original CrowS-Pairs design'—is an explicit validity caveat, not a circular step. Accordingly, no circularity is present.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new theoretical entities or fitted parameters. Its claims rest on methodological assumptions inherited from prior bias benchmarks and on the comparability of the three language-specific datasets, the latter being the most fragile.

assumptions (5)
  • domain assumption The pseudo-log-likelihood score in Equation 1 is a valid estimate of sentence likelihood for bias measurement.
    Adopted from Nangia et al. (2020) without re-validation; the entire MLM analysis depends on it.
  • domain assumption A bias score of 50 indicates neutrality, and deviations indicate stereotype preference.
    Implicit in the use of Equation 2; this definition from the original CrowS-Pairs is taken as given.
  • domain assumption The Dutch translations and localizations preserve the intended stereotype direction and minimal-pair structure of the original pairs.
    Section 3.1; the authors used two native speakers but report no inter-annotator agreement or validation study, so translation quality is assumed.
  • domain assumption The prompt-based choice task for autoregressive models measures how likely the model considers each scenario.
    Section 3.3; adopted from the BLOOM evaluation without evidence that the models' choices align with their likelihood estimates.
  • domain assumption The English, French, and Dutch versions of CrowS-Pairs are comparable enough to support cross-lingual bias comparisons.
    Section 3.2 and 4.1; this is questionable because the datasets differ in content (45 removed, many modified, categories re-assigned).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dutch CrowS-Pairs: Adapting a Challenge Dataset for Measuring Social Biases in Language Models for Dutch." pith.science (2026). https://pith.science/paper/7KCH5VN2

@misc{pith2026250716442,
  author       = {Pith},
  title        = {Pith review of: Dutch CrowS-Pairs: Adapting a Challenge Dataset for Measuring Social Biases in Language Models for Dutch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7KCH5VN2}},
  note         = {Machine review of arXiv:2507.16442}
}
read the original abstract

Warning: This paper contains explicit statements of offensive stereotypes which might be upsetting. Language models are prone to exhibiting biases, further amplifying unfair and harmful stereotypes. Given the fast-growing popularity and wide application of these models, it is necessary to ensure safe and fair language models. As of recent considerable attention has been paid to measuring bias in language models, yet the majority of studies have focused only on English language. A Dutch version of the US-specific CrowS-Pairs dataset for measuring bias in Dutch language models is introduced. The resulting dataset consists of 1463 sentence pairs that cover bias in 9 categories, such as Sexual orientation, Gender and Disability. The sentence pairs are composed of contrasting sentences, where one of the sentences concerns disadvantaged groups and the other advantaged groups. Using the Dutch CrowS-Pairs dataset, we show that various language models, BERTje, RobBERT, multilingual BERT, GEITje and Mistral-7B exhibit substantial bias across the various bias categories. Using the English and French versions of the CrowS-Pairs dataset, bias was evaluated in English (BERT and RoBERTa) and French (FlauBERT and CamemBERT) language models, and it was shown that English models exhibit the most bias, whereas Dutch models the least amount of bias. Additionally, results also indicate that assigning a persona to a language model changes the level of bias it exhibits. These findings highlight the variability of bias across languages and contexts, suggesting that cultural and linguistic factors play a significant role in shaping model biases.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 5 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Mukul Bhutani, Kevin Robinson, Vinodkumar Prabhakaran, Shachi Dave, and Sunipa Dev. 2024. https://doi.org/10.18653/v1/2024.acl-short.75 S ee GULL multilingual: a dataset of geo-culturally situated stereotypes . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 842--854, Bangkok, Thai...

    doi:10.18653/v1/2024.acl-short.75 2024

  4. [4]

    Christopher Akiki Ellie Pavlick BigScience Workshop: Teven Le Scao, Angela Fan et al. 2023. http://arxiv.org/abs/2211.05100 BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

    arXiv 2023

  5. [5]

    Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. https://doi.org/10.18653/v1/2021.acl-long.81 Stereotyping N orwegian salmon: An inventory of pitfalls in fairness benchmark datasets . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc...

    doi:10.18653/v1/2021.acl-long.81 2021

  6. [6]

    Bryson, and Arvind Narayanan

    Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. https://doi.org/10.1126/science.aal4230 Semantics derived automatically from language corpora contain human-like biases . Science, 356(6334):183–186

    doi:10.1126/science.aal4230 2017

  7. [7]

    Rodrigo Alejandro Ch \'a vez Mulsa and Gerasimos Spanakis. 2020. https://aclanthology.org/2020.gebnlp-1.6/ Evaluating bias in D utch word embeddings . In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 56--71, Barcelona, Spain (Online). Association for Computational Linguistics

    work page 2020

  8. [8]

    Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.292 R ob BERT : a D utch R o BERT a-based L anguage M odel . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3255--3265, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.findings-emnlp.292 2020

Show all 37 references
  1. [9]

    Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. http://arxiv.org/abs/2304.05335 Toxicity in chatgpt: Analyzing persona-assigned language models

    2023 arXiv

  2. [10]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. http://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . CoRR, abs/1810.04805

    2018 arXiv

  3. [11]

    Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. https://doi.org/10.1145/3442188.3445924 Bold: Dataset and metrics for measuring biases in open-ended language generation . In Proceedings of the 2021 ACM Confere...

    2021

  4. [13]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020 b . http://arxiv.org/abs/2009.11462 Realtoxicityprompts: Evaluating neural toxic degeneration in language models . CoRR, abs/2009.11462

    2020 arXiv

  5. [14]

    Greenwald, Debbie E

    Anthony G. Greenwald, Debbie E. McGhee, and Jordan L. Schwartz. 1998. https://doi.org/10.1037//0022-3514.74.6.1464 Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74(6):1464–1480

    1998 doi

  6. [15]

    Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders S gaard. 2022. https://doi.org/10.186...

    2022 doi

  7. [16]

    Masahiro Kaneko and Danushka Bollegala. 2021. http://arxiv.org/abs/2104.07496 Unmasking the mask - evaluating social biases in masked language models . CoRR, abs/2104.07496

    2021 arXiv

  8. [17]

    Jinqi Lai, Wensheng Gan, Jiayang Wu, Zhenlian Qi, and Philip S. Yu. 2024. https://doi.org/https://doi.org/10.1016/j.aiopen.2024.09.002 Large language models in law: A survey . AI Open, 5:181--196

    2024 doi

  9. [18]

    Hang Le, Lo\" i c Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Beno\^ i t Crabb\' e , Laurent Besacier, and Didier Schwab. 2020. https://www.aclweb.org/anthology/2020.lrec-1.302 Flaubert: Unsupervised language model pre-training ...

    2020

  10. [19]

    Yang Liu. 2024. http://arxiv.org/abs/2401.11601 Robust evaluation measures for evaluating social biases in masked language models

    2024 arXiv

  11. [20]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach

    2019 arXiv

  12. [21]

    Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. https://doi.org/10.18653/v1/2020.acl-main.645 Camembert: a tasty french language model . In Proceedings of the 58th Annual Meetin...

    2020 doi

  13. [22]

    Bowman, and Rachel Rudinger

    Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. https://doi.org/10.18653/v1/N19-1063 On measuring social biases in sentence encoders . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational...

    2019 doi

  14. [23]

    Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. http://arxiv.org/abs/2004.09456 Stereoset: Measuring stereotypical bias in pretrained language models

    2020 arXiv

  15. [24]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.154 C row S -pairs: A challenge dataset for measuring social biases in masked language models . In Proceedings of the 2020 Conference on Empirical Methods in Na...

    2020 doi

  16. [25]

    Roberto Navigli, Simone Conia, and Bj \"o rn Ross. 2023. Biases in large language models: origins, inventory, and discussion. ACM Journal of Data and Information Quality, 15(2):1--21

    2023

  17. [26]

    Aur \'e lie N \'e v \'e ol, Yoann Dupont, Julien Bezan c on, and Kar \"e n Fort. 2022. https://doi.org/10.18653/v1/2022.acl-long.583 F rench C row S -pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than E nglish . In...

    2022 doi

  18. [27]

    Irene Pagliai, Goya van Boven, Tosin Adewumi, Lama Alkhaled, Namrata Gurung, Isabella Södergren, and Elisa Barney. 2024. http://arxiv.org/abs/2404.04838 Data bias according to bipol: Men are naturally right and it is the role of women to follow their lead

    2024 arXiv

  19. [28]

    Edwin Rijgersberg and Bob Lucassen. 2023. https://github.com/Rijgersberg/GEITje GEITje: een groot open Nederlands taalmodel

    2023

  20. [29]

    Robinson, Perez Ogayo, David R

    Nathaniel R. Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023. http://arxiv.org/abs/2309.07423 Chatgpt mt: Competitive for high- (but not low-) resource languages

    2023 arXiv

  21. [30]

    Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. http://arxiv.org/abs/1804.09301 Gender bias in coreference resolution

    2018 arXiv

  22. [31]

    Leonard Salewski, Stephan Alaniz, Isabel Rio-Torto, Eric Schulz, and Zeynep Akata. 2023. http://arxiv.org/abs/2305.14930 In-context impersonation reveals large language models' strengths and biases

    2023 arXiv

  23. [32]

    Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. http://arxiv.org/abs/1909.01326 The woman worked as a babysitter: On biases in language generation

    2019 arXiv

  24. [33]

    Shah, Katherine Morley, David Osborn, Joseph Hayes, Robert Stewart, Johnny Downs, and et al

    Sumithra Velupillai, Hanna Suominen, Maria Liakata, Angus Roberts, Anoop D. Shah, Katherine Morley, David Osborn, Joseph Hayes, Robert Stewart, Johnny Downs, and et al. 2018. https://doi.org/10.1016/j.jbi.2018.10.005 Using clinical natural language processing for health outcom...

    2018 doi

  25. [34]

    Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. http://arxiv.org/abs/1912.09582 BERTje : A Dutch BERT Model . arXiv:1912.09582

    2019 arXiv

  26. [35]

    Jiaxu Zhao, Meng Fang, Shirui Pan, Wenpeng Yin, and Mykola Pechenizkiy. 2023. http://arxiv.org/abs/2312.06315 Gptbias: A comprehensive framework for evaluating bias in large language models

    2023 arXiv

  27. [36]

    Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. https://doi.org/10.18653/v1/N18-2003 Gender bias in coreference resolution: Evaluation and debiasing methods . In Proceedings of the 2018 Conference of the North A merican Chapter of the Associati...

    2018 doi

  28. [37]

    Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. http://arxiv.org/abs/2304.04675 Multilingual machine translation with large language models: Empirical results and analysis

    2023 arXiv

  29. [38]

    Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. http://arxiv.org/abs/2305.03514 Can large language models transform computational social science?

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.