Pith. sign in

REVIEW 4 major objections 4 minor 35 references

Mind the Language Gap: Automated and Augmented Evaluation of Bias in LLMs for High- and Low-Resource Languages

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Low-resource languages show more LLM bias in translated tests

desk verdict Solid engineering with an honest write-up, but the RQ2 low-resource bias gap is confounded by machine-translated oracles and discarded failures; treat that claim as not yet established. read the letter →

arxiv 2504.18560 v1 pith:ER22SDHX submitted 2025-04-19 cs.CL cs.AI

Alessio Buscemi , Cédric Lothritz , Sergio Morales , Marcos Gomez-Vazquez , Robert Clarisó , Jordi Cabot , German Castignani This is my paper · ORCID
classification cs.CLcs.AI
keywords multilingualbiastestinglow-resourcelanguagesmachinetranslationaugmentationparaphrasingtemplatessocialinLLMsLangBiTefairnessevaluationcross-lingual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MLA-BiTe, a framework that uses a large language model to translate and paraphrase bias-testing prompt templates so the same ethical tests can run in many languages. The authors claim this automation is reliable: GPT-4o's translations and paraphrases preserved semantic meaning well enough to serve as test instruments, and the order of translation versus paraphrasing mattered little. Using the augmented templates, they test four LLMs in six languages, including two low-resource ones, and find that models pass fewer bias tests — that is, exhibit more detectable bias — in Catalan and especially Luxembourgish than in English. If correct, the result means organizations with multilingual obligations can audit AI bias without hiring native-speaking test authors, and it warns that models behave worse in languages with less training data.

What carries the argument

The carrying mechanism is the MLA-BiTe pipeline: each prompt template is translated into a target language and then paraphrased a chosen number of times at the template level, before community placeholders are filled in, so that all generated test prompts stay syntactically uniform. The paraphraser first identifies the template's grammatical number, the singular or plural form of the sentence, to avoid pluralization errors such as 'Is a {GENDER1} better than a {GENDER2}?'. Test outcomes are measured by LangBiTe's oracle mechanism: every template carries an expected non-biased response, and a test is counted as passed when the model's answer matches that expectation, yielding a pass rate per language and bias category.

What would settle it

Translate the same template library by hand into French, German, and Luxembourgish with human-written expected answers, run the identical test set, and compare pass rates with the machine-translated versions; if the low-resource gap shrinks or disappears, the reported bias difference is an artifact of translation rather than a property of the models.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that LLM-based translation and paraphrasing can automatically expand a curated bias-test template library into new languages while preserving the semantic meaning that makes the test valid, and that doing so reveals higher social bias in low-resource languages. In the main evaluation, English and Spanish yielded the highest and most stable pass rates across all four tested models, while Luxembourgish showed the highest discrimination rates overall and Catalan varied widely by category; Llama3 405B was the least fair across languages. A correlation analysis adds that models' bias patterns do not track language families: Luxembourgish results correlate more closely with Spanish and Catalan than with German or English.

Load-bearing premise

The claim that lower pass rates in French, German, and Luxembourgish reflect model bias assumes the machine-translated templates and their machine-translated expected answers are as valid as human-authored ones, but human ground truth exists only for English, Spanish, and Catalan, so translation quality may be partly driving the language gap.

Editorial extensions

If this is right

  • Organizations can run bias audits in co-official or low-resource languages such as Luxembourgish and Catalan without recruiting native-speaking prompt authors for every language.
  • Model selection for fairness should be made per language and per bias category, since rankings shift: Gemini 1.5 Flash excels in religion and LGBTIQ+ tests while GPT-4o leads in ageism, sexism, and xenophobia.
  • Low-resource languages are where bias mitigation efforts will matter most, because all four tested models showed more detectable bias there.
  • Bias patterns do not transfer along language-family lines, so treating German and Luxembourgish as similar for fairness testing would be a mistake; the data show Luxembourgish behaving more like Spanish and Catalan.
  • Politics is the least reliable category: the source templates are US-centric and expect neutrality, so cross-lingual political-bias results should be read with caution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because GPT-4o generated the translations and paraphrases, its own later test scores may be inflated relative to other models; the paper flags this as a possible confound, and a fair comparison would use per-model translators or human translations.
  • The unvalidated French, German, and Luxembourgish templates raise the possibility that some of the measured 'bias gap' is translation artifact; testing with human-translated templates would isolate the model-bias component.
  • The same template-level augmentation approach could extend to non-Indo-European low-resource languages, but the method's reliance on grammatical-number identification would need to handle noun classes and richer morphology first.
  • A practical follow-up would test whether fine-tuning on low-resource-language data reduces the pass-rate gap, which would confirm the training-data-resource explanation suggested by the results.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MLA-BiTe, a framework that uses LLM-based translation and paraphrasing to expand LangBiTe bias-testing templates into six languages (English, Spanish, Catalan, French, German, Luxembourgish) and applies the augmented templates to four LLMs across seven sensitive categories. RQ1 asks whether automated translation and paraphrasing can reliably augment multilingual bias-test templates and which ordering of the two steps works best; RQ2 asks whether low-resource languages exhibit more detected bias than high-resource languages. The paper reports that the augmentation is viable, that the paraphrasing-then-translation order is marginally better, and that low-resource languages, especially Luxembourgish, show higher apparent bias. It also reports unprocessable-response rates per model, concern, and language, and a language-level correlation analysis. The central claims are practical and timely, but the evidence for RQ2 is weakened by validation gaps for French, German, and Luxembourgish and by the handling of unprocessable responses.

Significance. If the results hold, the framework would be a useful step toward scalable multilingual bias evaluation, lowering the need for native-speaker teams and extending single-language bias-testing tools to low-resource and co-official languages. The paper also provides a useful empirical comparison of four LLMs on seven bias categories across six languages, and it is transparent about processing failures and about the circularity of using GPT-4o as both generator and test subject. However, the central RQ2 conclusion—that low-resource languages show more bias—is not yet established, because the translation/oracle instrument is validated only for English, Spanish, and Catalan, unprocessable responses are discarded at unequal rates across languages, and no statistical tests accompany the headline means. These are load-bearing gaps that can, in principle, be fixed with additional validation and analysis.

major comments (4)
  1. [Section 5.2, Tables 5 and 6] The reported pass rates are conditional on successful response processing, and unprocessable responses are discarded at rates that differ strongly across languages and batches: Table 5 reports 3.3% unprocessable for English but 9.7% for Luxembourgish and 9.2% for Spanish, and Table 6 lists per-batch fault rates as high as 63.3–66.7% for several sexism batches and 51.6% for GPT-4o on Luxembourgish racism. Because the RQ2 comparison in Figure 5 uses only the processed responses, the lower mean for Luxembourgish may reflect a systematically noisier measurement instrument rather than higher model bias. The paper should report results with and without discarded responses, or model the missingness, and provide per-language and per-batch denominators and confidence intervals.
  2. [Section 4.3 and Section 5.1] Human-reference validation of translation quality is reported only for English, Spanish, and Catalan (Figure 1 and Section 4.3), while the French, German, and Luxembourgish templates and oracle expected values are machine-generated by GPT-4o without human ground truth. Since Algorithm 2 (lines 7–8) also machine-translates the expected values used for keyword matching, an unbiased or correct model response in a language with non-idiomatic or awkwardly translated oracle text can fail the keyword match and be counted as biased. Consequently, the Figure 5 gap (mean 61.1 for Luxembourgish versus 75.6 for English) is not separable from translation and oracle validity. The authors should add human validation or a second independent translation source for FR/DE/LB and perform a sensitivity analysis of the RQ2 result to oracle wording.
  3. [Figure 5 and Section 5.2] The cross-language comparison that answers RQ2 is presented only as aggregate means with no confidence intervals, effect sizes, or significance tests. Given that there are only four models and that processing-failure rates vary by language, the visual difference between, for example, Luxembourgish and English in Figure 5 could be within the range of sampling variation. The paper should report per-language distributions of pass rates and a statistical comparison, or explicitly justify why such a comparison is not appropriate given the measurement limitations.
  4. [Section 4.6 and Section 5.2] The pipeline-ordering decision (T2P versus P2T) is based only on EN–ES and ES–CA translations, and the main evaluation then applies this ordering to French, German, and Luxembourgish without validation for those languages. Similarly, the model-selection procedure does not include the two low-resource languages that are central to RQ2. If the ordering or the chosen generator behaves differently for FR/DE/LB, the conclusions about both RQ1 and RQ2 could change. The paper should either motivate the transfer of the ordering decision to these languages or test it directly.
minor comments (4)
  1. [Section 4.3] The sentence 'a lower BLEU score is actually preferred' is confusing in the context of translation evaluation, where BLEU is normally a similarity-to-reference metric; if the preference for low BLEU refers to paraphrasing, it should not be applied to the translation results in Figure 1.
  2. [Table 1 and Section 4.3] The candidate model is listed as 'Gemini Pro 1.5' in Table 1 but is referred to as 'Gemini 1.5 Flash' in Section 4.3 and in Figures 1–3; the model name should be consistent throughout.
  3. [Section 3.1 and Appendix] Cross-references to 'section 8' and to 'Section 3.2' for the prompt texts do not match the actual appendix structure; the appendices should be numbered or the references updated.
  4. [Section 3.2, Algorithm 3] The phrase 'which is reported and described in Section 3.2' in the description of Algorithm 3 appears to refer to the paraphrasing prompt, but that prompt is presented later in the appendix; please clarify the reference.

Circularity Check

1 steps flagged · score 6.0 of 10

RQ2's low-resource bias gap is built on GPT-4o's own translations and oracle values for FR/DE/LB; the paper concedes a self-advantage, making the comparison partially circular.

  1. self definitional [Section 5.2; Algorithm 2 (lines 7-8); Section 4.5]
    "GPT-4o was selected for translation and paraphrasing in the main tests presented in Section 5. ... EVT← T.expectedValueTranslator(l); t_expVal← EVT.translate(pt.expectedValue) ... English served as the source language, from which the test cases were translated into the target languages. ... because GPT-4o was chosen as the translation and paraphrasing model according to the results reported in Section 4, its output may provide GPT-4o with a slight advantage in the bias-detection task."

    The RQ2 measure of 'social bias' is the percentage of passed tests, where LangBiTe 'evaluates answers by searching for predefined, case-specific keywords' against an expected value. In the main experiment those expected values are produced by GPT-4o: Algorithm 2 lines 7-8 translate pt.expectedValue with the same translator selected in Section 4.5, and Section 5.2 states 'English served as the source language, from which the test cases were translated into the target languages.' GPT-4o is also one of the four models under test, so for one of the four systems the gold standard is its own output; the paper concedes 'its output may provide GPT-4o with a slight advantage in the bias-detection task.' Moreover, for FR/DE/LB no human reference exists (Section 4.3 validated only EN/ES/CA), so the…

full rationale

RQ1 is externally grounded: Section 4.3 compares machine translations against human-translated LangBiTe references for EN/ES/CA using cosine similarity and BLEU; this is an independent check and not circular. The use of LangBiTe via [19][20] is self-citation, but the template libraries are public and human-curated, so the self-citation is not load-bearing in the sense of an unverified uniqueness claim. The circular element is concentrated in RQ2's instrument: the paper's own Algorithm 2 uses the GPT-4o translator to create expected/oracle values, and GPT-4o is one of the tested models. Section 5.2 explicitly admits a possible GPT-4o advantage. For FR, DE, LB no human ground truth validates the translations, while Section 5.1 says ground truth is available only for ES, CA, EN; therefore the LB-vs-EN gap (61.1 vs 75.6) is partly a difference in instrument validity rather than a pure measure of model bias. The ES/CA anchor gives the central claim independent content, so the circularity is partial, not total. Uneven discarding of unprocessable responses (LB 9.7%, fault rates up to 66.7 per batch) is a serious validity threat but is not itself circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central results assume that machine-generated translations preserve test validity outside the two languages with human ground truth, and that the LangBiTe oracle keyword matching is a faithful bias detector. Both are domain assumptions that the paper partially validates but does not prove. No free parameters are fitted; experimental settings such as P=1 or temperature=1 are choices, not fitted constants.

assumptions (3)
  • domain assumption LLM-generated translations preserve semantic meaning and test intent for languages beyond ES/CA, where human ground truth exists.
    Invoked in Section 4.3, where translation quality is validated only for EN/ES/CA; then applied to FR/DE/LB in Section 5.1 without human references.
  • domain assumption The LangBiTe oracle keyword matching correctly identifies biased vs non-biased responses across translations.
    Used in Section 5.2; the paper itself notes in Section 6 that the oracle templates are U.S.-centric for politics and expect neutrality, which limits their validity.
  • domain assumption Using GPT-4o to generate and translate test materials does not invalidate its evaluation as a test subject.
    Acknowledged as a potential 'slight advantage' in Section 5.2, but not controlled or quantified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mind the Language Gap: Automated and Augmented Evaluation of Bias in LLMs for High- and Low-Resource Languages." pith.science (2026). https://pith.science/paper/ER22SDHX

@misc{pith2026250418560,
  author       = {Pith},
  title        = {Pith review of: Mind the Language Gap: Automated and Augmented Evaluation of Bias in LLMs for High- and Low-Resource Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ER22SDHX}},
  note         = {Machine review of arXiv:2504.18560}
}
read the original abstract

Large Language Models (LLMs) have exhibited impressive natural language processing capabilities but often perpetuate social biases inherent in their training data. To address this, we introduce MultiLingual Augmented Bias Testing (MLA-BiTe), a framework that improves prior bias evaluation methods by enabling systematic multilingual bias testing. MLA-BiTe leverages automated translation and paraphrasing techniques to support comprehensive assessments across diverse linguistic settings. In this study, we evaluate the effectiveness of MLA-BiTe by testing four state-of-the-art LLMs in six languages -- including two low-resource languages -- focusing on seven sensitive categories of discrimination.

Figures

Figures reproduced from arXiv: 2504.18560 by the authors.

Figure 1
Figure 1. The BLEU scores and cosine similarities for translations between each of the tested [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. BLEU and cosine similarities for paraphrasing across all the tested languages, with the [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Distribution of cosine similarity scores for selected translations at [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Each spider plot illustrates the percentage of passed tests for each LLM in one of the [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Aggregated results by language and model. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Heatmap of the Pearson Correlation of the performance achieved on the same model/con [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Paraphrasing performance by language and variations different values of [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 17 canonical work pages

  1. [1]

    Using natural sentence prompts for understanding biases in language models

    Sarah Alnegheimish, Alicia Guo, and Yi Sun. Using natural sentence prompts for understanding biases in language models. InHuman Language Technologies, pages 2824–2830. ACL, 2022

    work page 2022

  2. [2]

    Costa-Jussà, and Noe Casas

    Christine Basta, Marta R. Costa-Jussà, and Noe Casas. Evaluating the underlying gender bias in contextualized word embeddings. InGender Bias in NLP, pages 33–39. ACL, 2019

    work page 2019

  3. [3]

    On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021

    2021

  4. [4]

    Man is to computer programmer as woman is to homemaker? debiasing word embeddings

    Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in NeurIPS, 29, 2016

    work page 2016

  5. [5]

    Rae, Erich Elsen, and Laurent Sifre

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Ori...

    arXiv 2022

  6. [6]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  7. [7]

    Chatgpt vs gemini vs llama on multilingual sentiment analysis

    Alessio Buscemi and Daniele Proverbio. Chatgpt vs gemini vs llama on multilingual sentiment analysis. arXiv preprint arXiv:2402.01715, 2024. 17

    arXiv 2024

  8. [8]

    RogueGPT: dis-ethical tuning transforms ChatGPT4 into a Rogue AI in 158 Words

    Alessio Buscemi and Daniele Proverbio. Roguegpt: dis-ethical tuning transforms chatgpt4 into a rogue ai in 158 words.arXiv preprint arXiv:2407.15009, 2024

    work page Pith review arXiv 2024

Show all 35 references
  1. [9]

    Marked personas: Using natural language prompts to measure stereotypes in language models

    Myra Cheng, Esin Durmus, and Dan Jurafsky. Marked personas: Using natural language prompts to measure stereotypes in language models. In61st Annual Meeting of the Association for Computational Linguistics, pages 1504–1532. ACL, 2023

    2023

  2. [10]

    Bold: dataset and metrics for measuring biases in open-ended language generation

    Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: dataset and metrics for measuring biases in open-ended language generation. InFAccT, pages 862–872. ACM, 2021

    2021

  3. [11]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealTox- icityPrompts: Evaluating neural toxic degeneration in language models. InEMNLP, pages 3356–3369. ACL, 2020

    2020

  4. [12]

    Social bias evaluation for large language models requires prompt variations.arXiv preprint arXiv:2407.03129, 2024

    Rem Hida, Masahiro Kaneko, and Naoaki Okazaki. Social bias evaluation for large language models requires prompt variations.arXiv preprint arXiv:2407.03129, 2024

    2024 arXiv

  5. [13]

    Measuring bias in contextualized word representations

    Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. Measuring bias in contextualized word representations. In1st Workshop on Gender Bias in Natural Language Processing, pages 166–172. ACL, 2019

    2019

  6. [14]

    From zero to hero: On the limitations of zero-shot cross-lingual transfer with multilingual transformers.arXiv preprint arXiv:2005.00633, 2020

    Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. From zero to hero: On the limitations of zero-shot cross-lingual transfer with multilingual transformers.arXiv preprint arXiv:2005.00633, 2020

    2005 arXiv

  7. [15]

    The myth of left and right: How the political spectrum misleads and harms america

    Verlan Lewis and Hyrum Lewis. The myth of left and right: How the political spectrum misleads and harms america. 2022

    2022

  8. [17]

    Holistic evaluation of language models.arXiv preprint arXiv:2211.09110, 2023

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, et al. Holistic evaluation of language models.arXiv preprint arXiv:2211.09110, 2023

    2023 arXiv

  9. [18]

    Langbite, 2024

    Sergio Morales. Langbite, 2024. URLhttps://github.com/SOM-Research/LangBiTe

    2024

  10. [19]

    A DSL for testing LLMs for fairness and bias

    Sergio Morales, Robert Clarisó, and Jordi Cabot. A DSL for testing LLMs for fairness and bias. In MODELS, page 203–213. ACM, 2024

    2024

  11. [20]

    LangBiTe: A platform for testing bias in large language models.arXiv preprint arXiv:2404.18558, 2024

    Sergio Morales, Robert Clarisó, and Jordi Cabot. LangBiTe: A platform for testing bias in large language models.arXiv preprint arXiv:2404.18558, 2024

    2024 arXiv

  12. [21]

    ImageBiTe: A framework for evaluating representational harms in text-to-image models

    Sergio Morales, Robert Clarisó, and Jordi Cabot. ImageBiTe: A framework for evaluating representational harms in text-to-image models. In Proceedings of the 4th International Conference on AI Engineering – Software Engineering for AI, 2025. Pending publication

    2025

  13. [22]

    Stereoset: Measuring stereotypical bias in pretrained language models.arXiv preprint arXiv:2004.09456, 2020

    Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models.arXiv preprint arXiv:2004.09456, 2020. 18

    2004 arXiv

  14. [23]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), 2020

    2020

  15. [24]

    A comprehensive survey of bias in LLMs: Current landscape and future directions.arXiv preprint arXiv:2409.16430, 2024

    Rajesh Ranjan, Shailja Gupta, and Surya Narayan Singh. A comprehensive survey of bias in LLMs: Current landscape and future directions.arXiv preprint arXiv:2409.16430, 2024

    2024 arXiv

  16. [25]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URLhttp://arxiv.org/abs/ 1908.10084

    2019 arXiv

  17. [26]

    Recipes for building an open-domain chatbot

    Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004.13637, 2020

    2004 arXiv

  18. [27]

    Social bias in large language models for Bangla: An empirical study on gender and religious bias.arXiv preprint arXiv:2407.03536, 2024

    Jayanta Sadhu, Maneesha Rani Saha, and Rifat Shahriyar. Social bias in large language models for Bangla: An empirical study on gender and religious bias.arXiv preprint arXiv:2407.03536, 2024

    2024 arXiv

  19. [28]

    Term-weighting approaches in automatic text retrieval

    Gerard Salton and Christopher Buckley. Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5):513–523, 1988

    1988

  20. [29]

    Evaluating large language models with fmeval.arXiv preprint arXiv:2407.12872, 2024

    Pola Schwöbel, Luca Franceschi, Muhammad Bilal Zafar, et al. Evaluating large language models with fmeval.arXiv preprint arXiv:2407.12872, 2024

    2024 arXiv

  21. [30]

    The woman worked as a babysitter: On biases in language generation

    Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. InEMNLP-IJCNLP, pages 3407–3412. ACL, 2019

    2019

  22. [31]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint a...

    2022 arXiv

  23. [32]

    The artificial intelligence act, 2024

    European Union. The artificial intelligence act, 2024. URL https:// artificialintelligenceact.eu

    2024

  24. [33]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  25. [34]

    Yuxuan Wan, Wenxuan Wang, Pinjia He, Jiazhen Gu, Haonan Bai, and Michael R. Lyu. BiasAsker: Measuring the bias in conversational ai system. InESEC/FSE, page 515–527. ACM, 2023

    2023

  26. [36]

    Ethical and social risks of harm from language models

    Laura Weidinger, John Mellor, Maribeth Rauh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. 19

    2021 arXiv

  27. [37]

    {sentence}

    Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. InConference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1...

    2018

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.