Pith. sign in

REVIEW 3 major objections 5 minor 14 references

Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces Filipino CrowS-Pairs and Filipino WinoQueer, the first non-English benchmark for anti-queer bias, and shows that multilingual models, especially those trained on more Filipino text, systematically prefer sexist and…

desk verdict Useful new Filipino bias benchmarks, but the bias scores are over-interpreted given untested frequency confounds. read the letter →

arxiv 2412.07303 v2 pith:BLYVCXNW submitted 2024-12-10 cs.CL

Lance Calvin Lim Gamboa , Mark Lee This is my paper · ORCID
classification cs.CL
keywords FilipinobiasbenchmarkCrowS-PairsWinoQueermultilinguallanguagemodelshomophobicsexistculturaladaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that sexist and homophobic bias in multilingual language models can be measured in a low-resource language, Filipino, and that such models indeed exhibit considerable bias. To do this, the authors culturally adapt two English bias benchmarks, CrowS-Pairs and WinoQueer, into Filipino, producing 7,074 new sentence pairs and documenting the translation challenges in detail. They then evaluate eight off-the-shelf models and find average bias scores near 59 to 60 percent, meaning models are roughly 1.5 times more likely to prefer biased statements over their less biased counterparts. A further claim is that models pretrained with more Filipino text show stronger bias, and that English-derived queer labels such as tomboy and lesbiyana pick up additional bias through English as an intermediary language.

What carries the argument

The load-bearing measurement is the pseudo-log-likelihood bias score, following Nangia et al. (2020) and Felkner et al. (2023): for each minimally different sentence pair, every unmodified token is masked and its log-probability is summed conditional on the identity-defining tokens, and the bias score is the percentage of pairs where the biased sentence receives a higher score. The cultural adaptation procedure is equally load-bearing: because Filipino is largely gender-neutral (siya covers he and she, asawa covers spouse), the authors insert lalaki and babae as explicit gender markers, and because indigenous queer concepts do not map to Western identity categories, they replace the identity taxonomy with bakla, bading, tomboy, and lesbiyana while discarding stereotypes they deem culturally meaningless. These choices define what the benchmarks can detect and where their measurements can go wrong.

What would settle it

A control experiment would settle the core measurement claim: construct matched pairs where the identity label is held constant across both sentences (comparing, for example, two sentences about bakla that differ only in whether the stereotype wording is positive or negative), or replace the identity term in one member with a frequency-matched neutral noun, then recompute the bias score. If the score does not move substantially toward 50 percent, the metric is tracking lexical frequency or structural naturalness rather than stereotype association.

Watch

Extended reading notes

Core claim

The central claim is that Filipino CrowS-Pairs and Filipino WinoQueer provide a valid measurement of sexist and anti-queer bias in multilingual models processing Filipino, and that these models are measurably biased. On the paper's own terms, the discovery is threefold: first, the benchmarks are faithful cultural adaptations of their English sources, with the translation process itself revealing structural differences between English and Filipino gender and queerness concepts; second, all eight tested models exhibit bias, on average preferring biased statements about 1.5 times more often than unbiased ones; third, bias magnitude tracks the proportion of Filipino data in a model's pretraining corpus, as the models trained on the most Filipino text score highest on both benchmarks.

Load-bearing premise

The measurement assumes that the probability differences the bias score detects come from learned social stereotypes rather than from the fact that the distinguishing identity words, rare queer labels like bakla and bading versus very common words like lalaki and babae, plus inserted gender markers, differ in naturalness and frequency in Filipino.

Editorial extensions

If this is right

  • Filipino CrowS-Pairs and Filipino WinoQueer give researchers a reusable, documented template for building bias benchmarks in other low-resource languages by adapting English materials rather than starting from scratch.
  • If the results hold, multilingual models deployed in the Philippines and Southeast Asia carry measurable sexist and anti-queer stereotypes, and the extent of that bias depends on how much local-language data the models were trained on.
  • The finding about tomboy and lesbiyana suggests that multilingual models can transfer English-language bias onto local identity words that share English morphology, implying that debiasing efforts should account for lexical overlap with English.
  • The qualitative themes identified, such as queer men portrayed as pedophiles or mentally ill and queer women as promiscuous or sinful, give concrete, testable targets for bias mitigation.
  • Because masked and causal models with very different sizes all showed bias, the paper argues that model architecture and parameter count matter less than pretraining data composition for low-resource languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to apply the same adaptation template to other Southeast Asian languages, such as Indonesian or Malay, to see whether English-derived identity words produce the same bias-transfer pattern as tomboy and lesbiyana.
  • The authors' exposure-bias finding implies that debiasing multilingual models for low-resource languages may require rebalancing or filtering pretraining corpora rather than fine-tuning on small curated datasets, an implication the paper states only implicitly.
  • Because the paper does not directly test whether frequency differences between rare queer labels and common gendered labels drive the scores, a control experiment with frequency-matched neutral nouns could strengthen or weaken the central measurement claim.
  • The documented cultural adaptation strategy, particularly the decision to exclude stereotypes that are 'culturally meaningless,' suggests that future benchmarks will need to supplement translated items with locally crowdsourced stereotypes to capture biases unique to each region.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Filipino CrowS-Pairs and Filipino WinoQueer, two culturally adapted bias evaluation benchmarks derived from English sources, and uses them to measure sexist and anti-queer bias in eight multilingual language models, including masked and causal models and models pretrained specifically on Southeast Asian data. The authors report that the models exhibit considerable bias on both benchmarks, with average scores near 59–64%, and argue that the degree of bias correlates with the amount of Filipino pretraining data. The paper also documents a detailed cultural adaptation process, including linguistic gender differences, concepts of non-heterosexuality, and culturally irrelevant stereotypes, which is offered as a template for future benchmark adaptations.

Significance. If the measurements are valid, the paper delivers the first non-English benchmark specifically targeting anti-queer bias, a reusable cultural adaptation methodology, and baseline bias scores for multilingual models deployed in Southeast Asia. The dataset release of 7,074 prompt pairs is a concrete resource for a low-resource language, and the documentation of translation challenges (Section 3) is a valuable contribution to multilingual fairness research. The tabulated results are internally consistent, and the evaluation protocol follows established CrowS-Pairs and WinoQueer methodology. However, the central interpretation of the bias scores is threatened by a measurement confound related to the lexical and structural properties of the distinguishing tokens, and the secondary claim about pretraining exposure is confounded by model differences.

major comments (3)
  1. [§3.2.1, §3.2.2, Appendix B] The pseudo-log-likelihood scores that form the basis of the paper's bias measurements conflate stereotype associations with properties of the distinguishing tokens themselves. In CrowS-Pairs, gender is introduced by inserting 'ang babae' or 'ang lalaki' into sentences that would naturally use the gender-neutral pronoun 'siya' (Section 3.2.1), and in WinoQueer the queer labels 'bakla', 'bading', 'tomboy', and 'lesbiyana' are likely far less frequent in pretraining corpora than 'lalaki' and 'babae' (Section 3.2.2). The scoring formula in Appendix B sums log-probabilities of the shared tokens conditional on the distinguishing token, so any model that assigns lower probability to a rare or structurally unexpected label will produce lower scores for sentences containing that label, regardless of stereotype content. The paper cites Blodgett et al. (2021) in Section 3.2.4 but does not run a control condition that isolates label frequency or naturalness from stereotype content. Without such a control, the reported 58–65% averages cannot be uniquely attributed to learned social bias, and the abstract's claim of 'considerable amounts of bias' is not supported.
  2. [Table 8] The below-chance category scores in Table 8 are left uninterpreted and are difficult to reconcile with the paper's framing. For example, gpt2 scores 17.41 for 'bading' and xlm-roberta-base scores 43.48 for 'bakla', meaning these models preferred the less-biased sentence in the large majority of pairs. If the score were a pure measure of social bias, such extreme values would indicate strong counter-stereotypical preferences; a more plausible explanation is that the rare labels are being systematically penalized by the language model, which would confirm the confound raised above. The paper should either explain these outliers or acknowledge that the score is not calibrated to a neutral baseline.
  3. [§4.2, Table 10] The claim that the extent of bias is influenced by how much pretraining data in that language a model was exposed to is not supported by the evidence presented. The comparison across models in Table 10 confounds pretraining corpus size with architecture (causal vs. masked), parameter count (110M to 8B), instruction tuning, and model family. SEALION models are larger and causal, while RoBERTa-Tagalog is a small masked model; the observed higher bias scores could reflect any of these factors. A controlled comparison (e.g., models from the same family with different Filipino data proportions, or a regression that includes model size and architecture) or a clearly correlational framing with stated limitations is needed.
minor comments (5)
  1. [Table 6] The example sentences in Table 6 contain typos: 'ma STD' should likely be 'may STD' and 'makaasalanan' should be 'makasalanan'; these visible examples should be corrected.
  2. [Section 1] The text says 'sexist and homophonic bias' where 'homophobic bias' is intended.
  3. [References] The reference 'Cardozo, 2104' has an incorrect year; it should be 2014.
  4. [Table 8] The column headers 'Bakla Bading Tomboy Lesbiyana' are ambiguous; they should indicate that these are subcategories of WinoQueer scores.
  5. [Section 4.1] The paper does not report variance or significance tests for the bias scores; including confidence intervals or a bootstrap analysis would aid interpretation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark scores are computed against external off-the-shelf models with no fitted parameters, and the adaptation process is documented rather than derived from the results.

full rationale

I reviewed the paper's derivation chain end to end. The central contribution is the construction of two Filipino bias benchmarks by culturally adapting English CrowS-Pairs and WinoQueer, followed by an evaluation of eight external pretrained language models. No parameter in the bias score is fitted to the benchmark data or to the model outputs; the pseudo-log-likelihood metric in Appendix B is taken from the cited external methods of Nangia et al. (2020) and Felkner et al. (2023), and the bias score is simply the percentage of pairs for which the stereotyped sentence receives a higher pseudo-log-likelihood than its partner. The benchmark sentences themselves are not defined in terms of the resulting scores, and the scores are not fed back into the benchmark construction. I checked for the enumerated circularity patterns: there is no self-definitional relation between the benchmarks and the bias score; there is no fitted parameter later renamed as a prediction; the self-citations to Gamboa and Estuar (2023a,b) are contextual related work and are not load-bearing; no uniqueness theorem is imported from the authors' prior work; no ansatz is smuggled in via citation; and the adapted benchmarks are not a renaming of a known result because the Filipino language pairs and the anti-queer benchmark are genuinely new artifacts. The skeptical concern that rare identity labels such as bading or tomboy may drive pseudo-log-likelihood differences is a question of construct validity and confound control, not circularity: the paper's claim of 'considerable bias' may be challenged on measurement grounds, but the measurement is not equivalent to its own inputs by construction. The paper even acknowledges limitations regarding stereotype provenance and metric interpretation in Section 6. Therefore the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on four domain assumptions rather than on fitted parameters: the validity of pseudo-log-likelihood as a bias measure, the naturalness of forced gender marking, the adequacy of local identity labels, and the transferability of American-sourced stereotypes. These are all inherited from or consciously chosen in the adaptation process, and the paper documents the choices transparently, which is the main source of their credibility.

assumptions (5)
  • domain assumption The pseudo-log-likelihood bias score, defined as the summed log-probability of the unmodified tokens given the identity token, measures learned social bias rather than lexical artifact.
    This is the entire measurement apparatus, taken from Nangia et al. (2020) and Felkner et al. (2023) and restated in Appendix B; the paper relies on it without validating it on Filipino data.
  • domain assumption Inserting the gender markers 'lalaki' and 'babae' preserves meaning and naturalness well enough that the resulting pairs are still minimal and the 50 percent baseline still holds.
    Section 3.2.1 introduces the marker-insertion strategy for Filipino's gender-neutral grammar; if marker insertion changes fluency asymmetrically, the baseline shifts.
  • domain assumption The four labels 'bakla', 'bading', 'tomboy', and 'lesbiyana' adequately represent Filipino non-heterosexual identities, and 'lalaki'/'babae' serve as their heterosexual counterparts.
    Section 3.2.2 replaces the 13 English identity labels with these four local labels and excludes bisexual, asexual, pansexual, and straight as untranslatable concepts, an expert decision by the native-speaker first author rather than an empirically validated mapping.
  • domain assumption American-sourced stereotypes, after replacement or deletion of culturally meaningless items, remain valid probes of bias in Filipino society.
    Section 3.2.3 and the Limitations section state that only stereotypes present in the English benchmarks are included; Philippine-specific stereotypes are explicitly left to future work, so the benchmarks measure transferable rather than indigenous stereotypes.
  • domain assumption The reported bias scores are approximately independent across prompt pairs, so category percentages carry their nominal precision.
    WinoQueer pairs are generated as crosses of 11 templates with 140 stereotypes and 40 names, so pairs share construction components; the paper reports percentages without variance or pair-level statistics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia." pith.science (2026). https://pith.science/paper/BLYVCXNW

@misc{pith2026241207303,
  author       = {Pith},
  title        = {Pith review of: Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLYVCXNW}},
  note         = {Machine review of arXiv:2412.07303}
}
read the original abstract

Bias studies on multilingual models confirm the presence of gender-related stereotypes in masked models processing languages with high NLP resources. We expand on this line of research by introducing Filipino CrowS-Pairs and Filipino WinoQueer: benchmarks that assess both sexist and anti-queer biases in pretrained language models (PLMs) handling texts in Filipino, a low-resource language from the Philippines. The benchmarks consist of 7,074 new challenge pairs resulting from our cultural adaptation of English bias evaluation datasets, a process that we document in detail to guide similar forthcoming efforts. We apply the Filipino benchmarks on masked and causal multilingual models, including those pretrained on Southeast Asian data, and find that they contain considerable amounts of bias. We also find that for multilingual models, the extent of bias learned for a particular language is influenced by how much pretraining data in that language a model was exposed to. Our benchmarks and insights can serve as a foundation for future work analyzing and mitigating bias in multilingual models.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 10 canonical work pages

  1. [5]

    In Findings of the Association for Computational Linguistics: ACL 2023 , pages 2209–2225, Toronto, Canada

    This prompt is measuring <MASK>: Evaluating bias evaluation in language models. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 2209–2225, Toronto, Canada. Association for Com- putational Linguistics. Dirk Hovy and Shannon L. Spruit

    work page 2023

  2. [6]

    Preprint, arXiv:2310.06825

    Mistral 7B. Preprint, arXiv:2310.06825. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury

    arXiv

  3. [8]

    In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online

    CrowS-pairs: A chal- lenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. As- sociation for Computational Linguistics. Rodrigo Navarro

    work page 2020

  4. [9]

    arXiv preprint arXiv:2303.08774

    GPT-4 technical re- port. arXiv preprint arXiv:2303.08774. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman

    arXiv

  5. [10]

    In Findings of the Association for Computational Linguistics: ACL 2022 , pages 2086–2105, Dublin, Ireland

    BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 2086–2105, Dublin, Ireland. Association for Computational Linguistics. Philippine Statistics Authority. Philippines’ most com- mon baby names of 2022 [online]

    work page 2022

  6. [11]

    In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2887–2896, Singapore

    Investi- gating bias in multilingual language models: Cross- lingual transfer of debiasing techniques. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2887–2896, Singapore. Association for Computational Linguis- tics. Alfonso O. Santiago and Norma G. Tiangco

    work page 2023

  7. [12]

    In Find- ings of the Association for Computational Linguistics: NAACL 2022, pages 921–932, Seattle, United States

    An information-theoretic ap- proach and dataset for probing gender stereotypes in multilingual masked language models. In Find- ings of the Association for Computational Linguistics: NAACL 2022, pages 921–932, Seattle, United States. Association for Computational Linguistics. Yarden Tal, Inbal Magar, and Roy Schwartz

    work page 2022

  8. [13]

    In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , AIES ’21, page 254–265, New York, NY , USA

    Fairness for unobserved characteristics: Insights from technological impacts on queer communities. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , AIES ’21, page 254–265, New York, NY , USA. As- sociation for Computing Machinery. Gina Velasco

    work page 2021

Show all 14 references
  1. [14]

    How do large language models handle multilingualism? arXiv preprint arXiv:2402.18815. Model Training Paradigm Language GPU Used Runtime bert-base- multilingual-uncased masked languages worldwide NVIDIA A30 03:08:27 xlm-roberta-base masked languages worldwide NVIDIA A30 04:26:4...

    2020 arXiv

  2. [2020]

    In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online

    RealToxi- cityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguistics. Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir,...

    2020

  3. [2021]

    In Proceedings of the 2021 ACM Conference on Fair- ness, Accountability, and Transparency, FAccT ’21, page 862–872, New York, NY , USA

    BOLD: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fair- ness, Accountability, and Transparency, FAccT ’21, page 862–872, New York, NY , USA. Association for Computing Machinery. David M. Eberhard, Gar...

    2021

  4. [2022]

    arXiv preprint arXiv:2211.05100

    BLOOM: A 176B- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach

    arXiv

  5. [2023]

    In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9126– 9140, Toronto, Canada

    WinoQueer: A community- in-the-loop benchmark for anti-LGBTQ+ bias in large language models. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9126– 9140, Toronto, Canada. Association for Computa- tional...

    2023

  6. [2024]

    arXiv preprint arXiv:2404.09220

    Compass: Large multilingual lan- guage model for South-east Asia. arXiv preprint arXiv:2404.09220. Meta. Introducing Meta Llama 3: The most capable openly available LLM to date [online]

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.