Pith. sign in

REVIEW 4 major objections 5 minor 33 references

LIBRA: Measuring Bias of Large Language Model from a Local Context

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new bias score separates stereotyping from ignorance of local words.

desk verdict New local-context bias benchmark worth engaging; EiCAT metric needs validation and de-circularing. read the letter →

arxiv 2502.01679 v1 pith:LYJ2YI6L submitted 2025-02-02 cs.CY cs.CLcs.LG

Bo Pang , Tingrui Qiao , Caroline Walker , Chris Cunningham , Yun Sing Koh This is my paper · ORCID
classification cs.CYcs.CLcs.LG
keywords largelanguagemodelbiaslocalcontextknowledgeboundaryJensen-ShannondivergenceNewZealandMāorievaluationdatasetconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Bias benchmarks for large language models mostly use American English and assume the model knows the words in the test prompt. The paper argues that when a prompt contains a local word outside the model's training data, the model's bizarre completions are ignorance, not stereotype, and a fair bias test must tell these apart. It introduces LIBRA, a pipeline that builds bias test cases from local news corpora without crowdsourcing, and EiCAT, a score that combines the usual stereotype-preference measure with a 'beyond knowledge boundary' score and a Jensen–Shannon divergence between stereotyped and anti-stereotyped logit distributions. On a New Zealand dataset of over 160,000 test cases, the method reports that Llama-3-8B is the best causal model (EiCAT 10.72 versus GPT-2-xl's 1.68), mainly because it recognizes more local vocabulary. The claim matters because it implies that bias rankings change once a model's knowledge of local terms is accounted for.

What carries the argument

The central object is the EiCAT score of Eq. (3), which combines the Idealized CAT Score's language-model term with a beyond-knowledge-boundary score (bbs) and a Jensen–Shannon divergence $\mathrm{JSD}(D_A\|D_S)$. The bbs measures the fraction of local words in the test sentences that the model interprets in agreement with their formal definitions, and $\alpha$ is set equal to bbs so that models ignorant of local terms are automatically down-weighted on the bias component. The dataset machinery is the triplet-construction pipeline: sentences from local corpora are searched by keywords per social group, and the demographic token $\omega$ is replaced by another demographic term $M_p$ (anti-stereotyped) or an unrelated term $M_u$ (irrelevant), producing the three sentences $S$, $S_p$, and $S_u$ that the logit distributions are computed over.

What would settle it

Sample 200 anti-stereotyped triplets from the released New Zealand dataset and have fluent speakers rate the swapped sentence for grammaticality and counter-stereotypicality; if more than about 20% are judged unnatural or not clearly counter-stereotypical, the JSD component of EiCAT is contaminated and the reported rankings (Llama-3-8B 10.72 vs GPT-2-xl 1.68) would need to be recomputed on the natural subset.

Watch

Extended reading notes

Core claim

The central claim is that a fair local-context bias measurement must separate a model's tendency to prefer stereotyped completions from its failure to recognize local vocabulary. The paper formalizes this with the Enhanced Idealized CAT Score (EiCAT), defined as $\mathrm{EiCAT} = \mathrm{lms}\cdot[\alpha(1-\mathrm{JSD}(D_A\|D_S)) + (1-\alpha)\mathrm{bbs}]$, where lms is the language model score, $\mathrm{JSD}(D_A\|D_S)$ is the Jensen–Shannon divergence between the logit distributions for anti-stereotyped and stereotyped sentences, bbs is the fraction of out-of-vocabulary local words the model correctly understands, and $\alpha$ is set equal to bbs. A model that knows more local words gets a higher bbs, which raises $\alpha$ and therefore puts more weight on the bias term; a model that fails on local vocabulary is penalized twice, once through low bbs and once through the weighting. The authors report that on their New Zealand dataset, all tested models understand a small fraction of local words, and Llama-3-8B achieves the highest EiCAT among causal models because it handles Māori vocabulary better than GPT-2 or the BERT family.

Load-bearing premise

The load-bearing premise is that replacing the demographic word with another group's term yields a fluent, genuinely counter-stereotypical sentence; when the swap creates odd phrasing, the JSD term will measure grammaticality rather than bias.

Editorial extensions

If this is right

  • Bias rankings across models change when local vocabulary knowledge is scored: in the New Zealand context Llama-3-8B outperforms GPT-2-xl on EiCAT despite showing a larger stereotype divergence, because its higher bbs outweighs the higher JSD.
  • Existing U.S.-centric benchmarks may misclassify a model as biased when it is simply unfamiliar with the local terms, so bias reports should be accompanied by a knowledge-boundary score.
  • The pipeline lets researchers build culture-specific bias datasets from local news and broadcast corpora without paying crowd workers, lowering the barrier to testing LLMs in under-resourced languages.
  • Test cases containing local words the model misunderstands can be identified and either filtered or separately scored, making the bias measurement less noisy.
  • More linguistically competent models tend to show higher JSD, so a model that improves at language modeling may appear more biased unless the knowledge-boundary correction is applied.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the assumption of fluent anti-stereotyped sentences fails, the JSD term would partly measure grammaticality; one could test this by adding a human-fluency rating layer to a sample of the dataset and recomputing EiCAT on the subset rated fluent.
  • Editorial extension: the beyond-knowledge-boundary idea generalizes to any jargon the model rarely sees, such as medical or legal terms, where hallucination could also contaminate bias measurements.
  • Editorial extension: the framework predicts that fine-tuning a model on local text should raise its EiCAT mainly through bbs even if stereotype divergence stays flat, which could be checked with a before/after fine-tuning experiment.
  • Editorial extension: because the definition-verification step uses an LLM to judge alignment with the formal definition, that verifier model could itself hallucinate; a human-annotated sample of word definitions would bound this error.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LIBRA, an automated framework for building local-context bias datasets from local news corpora without crowdsourcing, together with a new metric, the Enhanced Idealized CAT Score (EiCAT). EiCAT combines the StereoSet language model score (lms), a Jensen-Shannon divergence (JSD) between log-likelihood distributions of stereotyped and anti-stereotyped sentences, and a Beyond Knowledge Boundary Score (bbs), as defined in Eq. (3). The authors construct a New Zealand dataset of 167,712 triplets from 367,384 local articles and evaluate several open-source LLMs (BERT family, RoBERTa, ALBERT, GPT-2, Llama-3), reporting that Llama-3-8B has the highest EiCAT among causal language models in the New Zealand context.

Significance. If the proposed measure is valid, LIBRA addresses a genuine gap: most bias benchmarks are US-centric, while local cultural contexts introduce both unfamiliar vocabulary and different stereotype structures. The dataset is large, the pipeline is automated and reproducible in principle, and the code/data are publicly released, which are concrete strengths. The attempt to separate knowledge-boundary effects from stereotypical bias is also a valuable direction. However, the central claim that EiCAT quantitatively measures bias is not currently established: the construction of anti-stereotyped sentences is not validated, the bbs verification procedure is underspecified, the choice of alpha is circular, and no uncertainty estimates are reported. These issues are load-bearing because every EiCAT ranking in Table 1 depends on them.

major comments (4)
  1. [Section 3, 'Compile Dataset'; Section 4, paragraph beginning 'Our dataset recognises noise'] The anti-stereotyped sentence S_p is generated by replacing the demographic token with another term from the same keyword set (e.g., a distant embedding neighbor when no antonym exists), without checking whether the resulting sentence is fluent, grammatical, or genuinely counter-stereotypical. The paper acknowledges 'noise' in the dataset and relies on 'collective distribution' to mitigate it, but that does not address the confound: if a large fraction of S_p sentences are ungrammatical or implausible, L(S_p) will be systematically lower than L(S), shifting D_a relative to D_s and inflating JSD in Eq. (3) regardless of any stereotype. The statement that 'consistent response to poorly flowing sentences confirms that our metric accurately captures model biases' is not a substitute for validation. The authors should quantify the fraction of triplets with valid S_p (e.g., via human or LLM annotation on a sample), report inter-annotator agreement, and show that the JSD-based ranking is stable when restricted to validated triplets.
  2. [Section 3, Eq. (3) and the paragraph following it] Setting alpha equal to bbs makes EiCAT a function of the very quantity it is supposed to balance: EiCAT = lms * [bbs*(1-JSD) + (1-bbs)*bbs] (with appropriate scaling). When bbs = 0, EiCAT becomes 0 regardless of lms and JSD, so models with perfect language scores and zero measured bias are ranked the same as random models (see Table 1: RandomLM, IdealLM, OV BERT-large, and OV GPT-2-xl all receive EiCAT = 0 solely because bbs = 0). This conflates knowledge-boundary failure with bias and makes the metric's ranking largely a reordered bbs ranking. The paper needs a principled justification for this choice, a sensitivity analysis over alpha, or a different weighting scheme.
  3. [Section 3, Eq. (2) and the bbs verification procedure] The bbs computation depends on a binary function f(W, S_w) that compares an LLM-generated definition D1 with an official definition D2, but the paper never specifies the prompt template P2, the matching threshold, or which LLM is used as the verifier. The statement 'if D1 and D2 align' is undefined: exact string matching, semantic similarity, and human judgment would all give different bbs values. Because EiCAT is multiplied by bbs and alpha, the entire ranking in Table 1 is sensitive to this unspecified step. The authors should provide the full verification prompt, the similarity measure and threshold, the verifier model, and, ideally, a small human agreement study.
  4. [Table 1 and Section 5, 'LLM Bias in New Zealand Context'] No uncertainty estimates are reported for any metric. Many reported differences are small on the 0-100 scale (e.g., BERT-base EiCAT 5.73 vs. BERT-large 5.91; GPT-2-large 1.61 vs. GPT-2-xl 1.68), and it is not clear whether these differences are meaningful given sampling variability in lms, JSD, and bbs. The paper should report bootstrap confidence intervals or statistical tests for the key comparisons, and it should state clearly which cross-model comparisons are permitted, given the admission that JSD values 'cannot be directly compared across masked and causal models.'
minor comments (5)
  1. [Abstract and Section 1] There are several typos: 'ch llenge' in the abstract, 'ocal bias' in Section 1, and inconsistent caption formatting in Fig. 1. These should be corrected in revision.
  2. [References] Reference [31] is cited for Llama-3 in the Introduction and Results, but the reference list entry [31] points to Vidit et al., 'CLIP the Gap: A Single Domain Generalization Approach for Object Detection,' which is not the Llama model paper. The authors should cite the correct Llama-3 reference.
  3. [Table 1 caption] The caption states that iCAT is computed 'using the StereoSet dataset for comparison,' which explains why the iCAT column is identical across the NZ, OV, and Malay panels. This should be stated more explicitly in the table itself or in a footnote, because as presented the repeated values may confuse readers.
  4. [Section 3, Eq. (3)] The text says EiCAT ranges from 0 to 1, but Table 1 presents all metrics scaled to 0-100 and EiCAT values are on the 0-100 scale there. The scaling convention should be stated explicitly when Eq. (3) is introduced, and the definitions of lms and bbs as either probabilities or percentages should be consistent.
  5. [Section 2, Related Work] The claim that the automatic pipeline 'ensuring grammatical diversity' is stronger than the evidence supports, given that grammaticality of the generated S_p sentences is not verified. A more cautious wording would be appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EiCAT is an explicit composite metric, and the alpha=bbs weighting is a stated design choice rather than a hidden reduction of a prediction to its inputs.

full rationale

The derivation chain is self-contained and transparent. EiCAT in Eq. (3) is an explicit composite of three measured quantities, lms, JSD, and bbs, with a stated weighting rule alpha=bbs. Substituting gives EiCAT = lms * bbs * (2 - JSD - bbs), so bbs enters both as a factor and as the weight; this is a deliberate design choice that penalizes models that do not understand local vocabulary, as the paper states in Section 3, rather than a hidden reduction of a predicted quantity to a fitted input. The conclusion that Llama-3-8B has the highest EiCAT among causal models follows directly from its higher measured bbs and lms; the paper does not claim to derive bias independently of these measurements. The construction of anti-stereotyped sentences by token replacement is a validity concern because the fluency of S_p is not quantified, but the paper explicitly acknowledges noise and relies on collective distributions; this is a measurement-quality issue, not circularity. No load-bearing self-citations or imported uniqueness theorems appear: StereoSet and iCAT are cited as external prior work, and the Our Voices acknowledgement is not used as evidence. Therefore no step in the paper's derivation reduces to its own inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The metric combines existing iCAT with new bbs and JSD terms. The main dependencies are the validity of automatic triplet generation, the dictionary-based definition of knowledge boundaries, and the assumption that JSD of marginal logit distributions captures per-case bias. None of these is empirically validated in the paper.

free parameters (3)
  • alpha weighting parameter in EiCAT = alpha = bbs
    In Eq. 3, alpha is set equal to the bbs value without a principled derivation; it is a hand-chosen dynamic balance between the JSD and bbs terms.
  • Definition-matching threshold in verification prompt P2 = Not specified
    The binary function f(W, Sw) requires comparing the model-generated definition D1 with the official definition D2, but no similarity threshold or scoring rule is given.
  • Clustering and keyword expansion hyperparameters = Not reported
    The dataset pipeline depends on UMAP dimensionality, HDBSCAN cluster sizes, and association rule support, but exact values are not stated in the paper.
assumptions (4)
  • domain assumption Local news corpora are a valid source of stereotypical associations for NZ social groups.
    Pipeline builds triplets from media sentences; no external validation shows these reflect stereotypes rather than journalistic collocations.
  • domain assumption Words in V' = V \ Vw are beyond an LLM's knowledge boundary, and an LLM-generated definition matching a dictionary definition establishes understanding.
    Section 3 'Effect of Beyond Knowledge Boundary Words' uses English dictionary exclusion and definition comparison; both steps assume dictionary membership and self-reported definitions are ground truth.
  • ad hoc to paper Sentence swapping preserves language-model score comparability.
    The framework assumes replacing the demographic token with another demographic or unrelated term creates a comparable sentence, but no fluency or validity checks are reported.
  • ad hoc to paper Jensen-Shannon divergence between marginal logit distributions is a valid per-case bias measure.
    JSD on aggregate distributions can be near zero even when a model strongly prefers stereotypes in half of the cases and anti-stereotypes in the other half, so it is not obviously a per-trip.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LIBRA: Measuring Bias of Large Language Model from a Local Context." pith.science (2026). https://pith.science/paper/LYJ2YI6L

@misc{pith2026250201679,
  author       = {Pith},
  title        = {Pith review of: LIBRA: Measuring Bias of Large Language Model from a Local Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LYJ2YI6L}},
  note         = {Machine review of arXiv:2502.01679}
}
read the original abstract

Large Language Models (LLMs) have significantly advanced natural language processing applications, yet their widespread use raises concerns regarding inherent biases that may reduce utility or harm for particular social groups. Despite the advancement in addressing LLM bias, existing research has two major limitations. First, existing LLM bias evaluation focuses on the U.S. cultural context, making it challenging to reveal stereotypical biases of LLMs toward other cultures, leading to unfair development and use of LLMs. Second, current bias evaluation often assumes models are familiar with the target social groups. When LLMs encounter words beyond their knowledge boundaries that are unfamiliar in their training data, they produce irrelevant results in the local context due to hallucinations and overconfidence, which are not necessarily indicative of inherent bias. This research addresses these limitations with a Local Integrated Bias Recognition and Assessment Framework (LIBRA) for measuring bias using datasets sourced from local corpora without crowdsourcing. Implementing this framework, we develop a dataset comprising over 360,000 test cases in the New Zealand context. Furthermore, we propose the Enhanced Idealized CAT Score (EiCAT), integrating the iCAT score with a beyond knowledge boundary score (bbs) and a distribution divergence-based bias measurement to tackle the challenge of LLMs encountering words beyond knowledge boundaries. Our results show that the BERT family, GPT-2, and Llama-3 models seldom understand local words in different contexts. While Llama-3 exhibits larger bias, it responds better to different cultural contexts. The code and dataset are available at: https://github.com/ipangbo/LIBRA.

Figures

Figures reproduced from arXiv: 2502.01679 by the authors.

Figure 1
Figure 1. The comparative responses of different LLMs to prompts such as “My [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Build and use a dataset to test the bias of Large Language Models. A fair [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualisations of Contextual Diversity. Fig. (a) shows BERT embeddings [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of kernel density estimation (KDE) plots for the log [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 18 canonical work pages

  1. [1]

    In:FirstConferenceonLanguageModeling(2024),https://openreview.net/forum? id=IW1PR7vEBf 14 B

    BehnamGhader, P., Adlakha, V., Mosbach, M., Bahdanau, D., Chapados, N., Reddy, S.: LLM2vec: Large language models are secretly powerful text encoders. In:FirstConferenceonLanguageModeling(2024),https://openreview.net/forum? id=IW1PR7vEBf 14 B. Pang et al

    work page 2024

  2. [2]

    In: Proceedings of the 1stInternationalWorkshoponOpenSourceDataMining:FrequentPatternMining Implementations

    Borgelt, C.: An implementation of the fp-growth algorithm. In: Proceedings of the 1stInternationalWorkshoponOpenSourceDataMining:FrequentPatternMining Implementations. p. 1–5. OSDM ’05, Association for Computing Machinery, New York, NY, USA (2005). https://doi.org/10.1145/1133905.1133907

    work page arXiv 2005

  3. [3]

    In: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency

    Cabello, L., Jørgensen, A.K., Søgaard, A.: On the independence of association bias and empirical fairness in language models. In: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. p. 370–378. FAccT ’23, Association for Computing Machinery, New York, NY, USA (2023). https://doi. org/10.1145/3593013.3594004

    work page arXiv 2023

  4. [4]

    Science 356(6334), 183– 186 (2017)

    Caliskan, A., Bryson, J.J., Narayanan, A.: Semantics derived automatically from language corpora contain human-like biases. Science 356(6334), 183– 186 (2017). https://doi.org/10.1126/science.aal4230, https://www.science.org/ doi/abs/10.1126/science.aal4230

    doi:10.1126/science.aal4230 2017

  5. [5]

    Language Learning65(2), 298–325 (2015)

    Chan, H., Verspoor, M., Vahtrick, L.: Dynamic development in speaking versus writing in identical twins. Language Learning65(2), 298–325 (2015). https://doi. org/https://doi.org/10.1111/lang.12107, https://onlinelibrary.wiley.com/doi/abs/ 10.1111/lang.12107

    work page doi:10.1111/lang.12107 2015

  6. [6]

    In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum? id=7Ttk3RzDeu

    Chang, Y., Lo, K., Goyal, T., Iyyer, M.: Booookscore: A systematic exploration of book-length summarization in the era of LLMs. In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum? id=7Ttk3RzDeu

    work page 2024

  7. [7]

    Association for Computing Machinery Transactions on Intelligent Systems and Technology15(3) (March 2024)

    Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P.S., Yang, Q., Xie, X.: A survey on evaluation of large language models. Association for Computing Machinery Transactions on Intelligent Systems and Technology15(3) (March 2024). https: //doi.org/10.1145/3641289

    doi:10.1145/3641289 2024

  8. [8]

    In: Calzo- lari, N., Kan, M.Y., Hoste, V., Lenci, A., Sakti, S., Xue, N

    Chanthran, M., Soon, L.K., Ong, H.F., Selvaretnam, B.: Malaysian English news decoded: A linguistic resource for named entity and relation extraction. In: Calzo- lari, N., Kan, M.Y., Hoste, V., Lenci, A., Sakti, S., Xue, N. (eds.) Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. pp. 10...

    work page 2024

Show all 33 references
  1. [9]

    arXiv preprint arXiv:1810.04805 (2018)

    Devlin, J.: Bert: Pre-training of deep bidirectional transformers for language un- derstanding. arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  2. [10]

    Computational Linguistics pp

    Gallegos, I.O., Rossi, R.A., Barrow, J., Tanjim, M.M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., Ahmed, N.K.: Bias and Fairness in Large Language Models: A Survey. Computational Linguistics pp. 1–83 (08 2024). https://doi.org/10.1162/ coli_a_00524

    2024

  3. [11]

    In: International Conference on Learning Representations (2020), https://openreview.net/forum? id=H1eA7AEtvS

    Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: Albert: A lite bert for self-supervised learning of language representations. In: International Conference on Learning Representations (2020), https://openreview.net/forum? id=H1eA7AEtvS

    2020

  4. [12]

    In: Rogers, A., Boyd-Graber, J., Okazaki, N

    Li, H., Schlegel, V., Batista-Navarro, R., Nenadic, G.: Do you hear the people sing? key point analysis via iterative clustering and abstractive summarisation. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual MeetingoftheAssociationforComputat...

    2023 doi

  5. [13]

    IEEE Transactions on Knowledge and Data Engineering34(1), 50–70 (2022)

    Li, J., Sun, A., Han, J., Li, C.: A survey on deep learning for named entity recogni- tion. IEEE Transactions on Knowledge and Data Engineering34(1), 50–70 (2022). https://doi.org/10.1109/TKDE.2020.2981314

    2022

  6. [14]

    arXiv preprint arXiv:2308.10149 (2023)

    Li,Y.,Du,M.,Song,R.,Wang,X.,Wang,Y.:Asurveyonfairnessinlargelanguage models. arXiv preprint arXiv:2308.10149 (2023)

    2023 arXiv

  7. [15]

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: A robustly optimized BERT pretraining approach (2020), https://openreview.net/forum?id=SyxS0T4tvS

    2020

  8. [16]

    In: Burstein, J., Doran, C., Solorio, T

    May, C., Wang, A., Bordia, S., Bowman, S.R., Rudinger, R.: On measuring so- cial biases in sentence encoders. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Hu...

    2019 doi

  9. [17]

    Journal of Open Source Software2(11), 205 (2017)

    McInnes, L., Healy, J., Astels, S.: HDBSCAN: Hierarchical density based cluster- ing. Journal of Open Source Software2(11), 205 (2017). https://doi.org/10.21105/ joss.00205, https://doi.org/10.21105/joss.00205

    2017 doi

  10. [18]

    The Journal of Open Source Software 3(29), 861 (2018)

    McInnes, L., Healy, J., Saul, N., Grossberger, L.: UMAP: Uniform manifold ap- proximation and projection. The Journal of Open Source Software 3(29), 861 (2018)

    2018

  11. [19]

    In: Muresan, S., Nakov, P., Villavicencio, A

    Meade, N., Poole-Dayan, E., Reddy, S.: An empirical survey of the effectiveness of debiasing techniques for pre-trained language models. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics...

    2022 doi

  12. [20]

    In: Zong, C., Xia, F., Li, W., Navigli, R

    Nadeem, M., Bethke, A., Reddy, S.: StereoSet: Measuring stereotypical bias in pretrained language models. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.) Pro- ceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Jo...

    2021 doi

  13. [21]

    In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing

    Nangia, N., Vania, C., Bhalerao, R., Bowman, S.R.: CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Onl...

    2020

  14. [22]

    Navigli, R., Conia, S., Ross, B.: Biases in large language models: Origins, inventory, and discussion. J. Data and Information Quality15(2) (June 2023). https://doi. org/10.1145/3597307

    2023 doi

  15. [23]

    OurVoices: Our Voices Home, https://ourvoices.auckland.ac.nz/

  16. [24]

    In: Muresan, S., Nakov, P., Villavicencio, A

    Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P.M., Bowman, S.: BBQ: A hand-built bias benchmark for question answer- ing. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Findings of the Association for Computational Linguistics: 60th A...

    2022 doi

  17. [25]

    In: The 2023 Conference on Empirical Methods in Natural Language Processing (2023), https://openreview.net/forum? id=Y6w2prqvjM

    Pozzobon, L.A., Ermis, B., Lewis, P., Hooker, S.: On the challenges of using black- box APIs for toxicity evaluation in research. In: The 2023 Conference on Empirical Methods in Natural Language Processing (2023), https://openreview.net/forum? id=Y6w2prqvjM

    2023

  18. [26]

    In: Goldberg, Y., Kozareva, Z., Zhang, Y

    Qian, R., Ross, C., Fernandes, J., Smith, E.M., Kiela, D., Williams, A.: Perturba- tion augmentation for fairer NLP. In: Goldberg, Y., Kozareva, Z., Zhang, Y. (eds.) Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 9496–9521. Associat...

    2022 doi

  19. [27]

    OpenAI blog1(8), 9 (2019)

    Radford,A.,Wu,J.,Child,R.,Luan,D.,Amodei,D.,Sutskever,I.,etal.:Language models are unsupervised multitask learners. OpenAI blog1(8), 9 (2019)

    2019

  20. [28]

    arXiv preprint arXiv:2307.11019 (2023)

    Ren, R., Wang, Y., Qu, Y., Zhao, W.X., Liu, J., Tian, H., Wu, H., Wen, J.R., Wang, H.: Investigating the factual knowledge boundary of large language models with retrieval augmentation. arXiv preprint arXiv:2307.11019 (2023)

    2023 arXiv

  21. [29]

    Transactions of the Association for Com- putational Linguistics 9, 1408–1424 (12 2021)

    Schick, T., Udupa, S., Schütze, H.: Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP. Transactions of the Association for Com- putational Linguistics 9, 1408–1424 (12 2021). https://doi.org/10.1162/tacl_a_ 00434

    2021 doi

  22. [30]

    New Zealand journal of psychology40(2), 25–36 (October 2011), https://kar.kent.ac.uk/84622/

    Sibley, C.G., Stewart, K., Houkamau, C., Manuela, S., Perry, R., Wootton, L.W., Harding, J.F., Zhang, Y., Sengupta, N., Robertson, A.: Ethnic group stereotypes in new zealand. New Zealand journal of psychology40(2), 25–36 (October 2011), https://kar.kent.ac.uk/84622/

    2011

  23. [31]

    In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Vidit, V., Engilberge, M., Salzmann, M.: Clip the gap: A single domain gen- eralization approach for object detection. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3219–3229 (2023). https: //doi.org/10.1109/CVPR52729.2023.00314

    2023

  24. [32]

    arXiv preprint arXiv:1910.03771 (2019)

    Wolf, T.: Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019)

    2019 arXiv

  25. [33]

    arXiv preprint arXiv:2312.01509 (2023)

    Yogarajan, V., Dobbie, G., Keegan, T.T., Neuwirth, R.J.: Tackling bias in pre- trained language models: Current trends and under-represented societies. arXiv preprint arXiv:2312.01509 (2023)

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.