{"id":"d40a84aa-bf00-46fc-b140-0b622b611eef","arxiv_id":"2412.07303","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"New Filipino benchmark pairs show multilingual models choose sexist and anti-queer sentences measurably more often than neutral alternatives, with models trained on more Filipino text showing the strongest bias.","lead":"Researchers built two Filipino-language test sets, Filipino CrowS-Pairs and Filipino WinoQueer, with 7,074 sentence pairs that reveal sexist and anti-queer stereotypes in language models. These benchmarks give model builders a way to measure and later reduce harmful bias in AI systems serving Filipino speakers and Southeast Asia.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-log-likelihood scores may reflect identity-label frequency and structural unnaturalness rather than stereotype associations; without a neutral control, the 'considerable bias' conclusion is unsupported.","rationale":"The reader's weakest_assumption correctly identifies the pseudo-log-likelihood confound: the Filipino adaptation's identity markers are not matched for frequency or structural naturalness, and the metric's conditioning on those markers makes this directly consequential. My attack sharpens the point by noting that below-chance category scores (e.g., gpt2 at 17.41 for bading in Table 8) suggest label identity, not stereotype direction, can dominate the score, since a pure stereotype effect should not systematically favor the less-biased sentence unless the rare label is independently penalized. The proposed neutral control directly tests whether the benchmark measures social bias or lexical prior. The secondary claim about pretraining-data exposure is also confounded by model family, size, and training objective, but the measurement validity issue is more load-bearing because it determines whether the benchmark itself is a valid bias probe. I therefore do not move the verdict: the paper should remain CONDITIONAL pending the control analysis or an explicit caveat that the scores may reflect label-frequency artifacts.","tokens_in":14287,"tokens_out":5935,"duration_ms":65584,"concrete_test":"Construct a neutral control set using the same WinoQueer label sets and templates but replace all stereotype clauses with semantically neutral clauses matched for length and function words (e.g., 'Bakla siya at kumakain.' / 'Lalaki siya at kumakain.'; 'Tomboy siya at mahilig magbasa.' / 'Babae siya at mahilig magbasa.'). Run the exact Appendix B pseudo-log-likelihood procedure on all eight models and compute the mean percentage preferring the queer-labeled sentence, with per-label breakdowns. If the neutral-control percentages are near 50, the WinoQueer bias scores are not label-frequency artifacts; if they deviate substantially, the bias scores are confounded and the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central measurement premise is that a higher pseudo-log-likelihood for the 'more biased' sentence indicates learned stereotype association, but the Filipino adaptation makes the distinguishing tokens lexically and structurally asymmetric. In WinoQueer, the queer labels bakla, bading, tomboy, and lesbiyana are low-frequency loan/cultural terms, while the comparison labels lalaki and babae are among the most common gender words in Filipino (Section 3.2.2). In CrowS-Pairs, gender is introduced by inserting 'ang babae' or 'ang lalaki' into sentences that would naturally use the gender-neutral pronoun siya (Section 3.2.1). Appendix B computes score(S) = sum over unmodified tokens of log P(unmodified token | all other unmodified tokens, modified tokens), so any tendency of the model to assign different probabilities to the shared tokens after a rare or unfamiliar label is counted as 'bias.' The paper cites minimal-pair pitfalls in prior work (Section 3.2.4) but provides no control condition isolating label frequency from stereotype content. The below-chance category scores in Table 8 (e.g., gpt2 at 17.41 for bading) are a warning sign: a pure stereotype signal should not push scores that far below 50 unless the rare label itself is being penalized. Without a neutral-template control, the reported 58-65% scores cannot be uniquely attributed to social bias, so the abstract's 'considerable amounts of bias' claim rests on an untested confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Filipino CrowS-Pairs and Filipino WinoQueer, two culturally adapted bias evaluation benchmarks derived from English sources, and uses them to measure sexist and anti-queer bias in eight multilingual language models, including masked and causal models and models pretrained specifically on Southeast Asian data. The authors report that the models exhibit considerable bias on both benchmarks, with average scores near 59–64%, and argue that the degree of bias correlates with the amount of Filipino pretraining data. The paper also documents a detailed cultural adaptation process, including linguistic gender differences, concepts of non-heterosexuality, and culturally irrelevant stereotypes, which is offered as a template for future benchmark adaptations.","tokens_in":14416,"tokens_out":5027,"duration_ms":48965,"significance":"If the measurements are valid, the paper delivers the first non-English benchmark specifically targeting anti-queer bias, a reusable cultural adaptation methodology, and baseline bias scores for multilingual models deployed in Southeast Asia. The dataset release of 7,074 prompt pairs is a concrete resource for a low-resource language, and the documentation of translation challenges (Section 3) is a valuable contribution to multilingual fairness research. The tabulated results are internally consistent, and the evaluation protocol follows established CrowS-Pairs and WinoQueer methodology. However, the central interpretation of the bias scores is threatened by a measurement confound related to the lexical and structural properties of the distinguishing tokens, and the secondary claim about pretraining exposure is confounded by model differences.","major_comments":[{"comment":"The pseudo-log-likelihood scores that form the basis of the paper's bias measurements conflate stereotype associations with properties of the distinguishing tokens themselves. In CrowS-Pairs, gender is introduced by inserting 'ang babae' or 'ang lalaki' into sentences that would naturally use the gender-neutral pronoun 'siya' (Section 3.2.1), and in WinoQueer the queer labels 'bakla', 'bading', 'tomboy', and 'lesbiyana' are likely far less frequent in pretraining corpora than 'lalaki' and 'babae' (Section 3.2.2). The scoring formula in Appendix B sums log-probabilities of the shared tokens conditional on the distinguishing token, so any model that assigns lower probability to a rare or structurally unexpected label will produce lower scores for sentences containing that label, regardless of stereotype content. The paper cites Blodgett et al. (2021) in Section 3.2.4 but does not run a control condition that isolates label frequency or naturalness from stereotype content. Without such a control, the reported 58–65% averages cannot be uniquely attributed to learned social bias, and the abstract's claim of 'considerable amounts of bias' is not supported.","section":"§3.2.1, §3.2.2, Appendix B"},{"comment":"The below-chance category scores in Table 8 are left uninterpreted and are difficult to reconcile with the paper's framing. For example, gpt2 scores 17.41 for 'bading' and xlm-roberta-base scores 43.48 for 'bakla', meaning these models preferred the less-biased sentence in the large majority of pairs. If the score were a pure measure of social bias, such extreme values would indicate strong counter-stereotypical preferences; a more plausible explanation is that the rare labels are being systematically penalized by the language model, which would confirm the confound raised above. The paper should either explain these outliers or acknowledge that the score is not calibrated to a neutral baseline.","section":"Table 8"},{"comment":"The claim that the extent of bias is influenced by how much pretraining data in that language a model was exposed to is not supported by the evidence presented. The comparison across models in Table 10 confounds pretraining corpus size with architecture (causal vs. masked), parameter count (110M to 8B), instruction tuning, and model family. SEALION models are larger and causal, while RoBERTa-Tagalog is a small masked model; the observed higher bias scores could reflect any of these factors. A controlled comparison (e.g., models from the same family with different Filipino data proportions, or a regression that includes model size and architecture) or a clearly correlational framing with stated limitations is needed.","section":"§4.2, Table 10"}],"minor_comments":[{"comment":"The example sentences in Table 6 contain typos: 'ma STD' should likely be 'may STD' and 'makaasalanan' should be 'makasalanan'; these visible examples should be corrected.","section":"Table 6"},{"comment":"The text says 'sexist and homophonic bias' where 'homophobic bias' is intended.","section":"Section 1"},{"comment":"The reference 'Cardozo, 2104' has an incorrect year; it should be 2014.","section":"References"},{"comment":"The column headers 'Bakla Bading Tomboy Lesbiyana' are ambiguous; they should indicate that these are subcategories of WinoQueer scores.","section":"Table 8"},{"comment":"The paper does not report variance or significance tests for the bias scores; including confidence intervals or a bootstrap analysis would aid interpretation.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The dataset and cultural adaptation documentation are valuable, and the confound raised in the major comments could be addressed with an additional control experiment comparing neutral templates with the same identity labels. I would encourage the editor to ask for such an experiment or a clear re-scoping of the claims before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: the benchmark resource is genuinely useful, and the cultural adaptation write-up is the best part. The bias measurements and the 'pretraining data' explanation need to be taken with a grain of salt.\n\nWhat's new: Filipino CrowS-Pairs and Filipino WinoQueer, 7,074 pairs, the first non-English anti-queer bias benchmark. The authors carefully adapted gender-neutral Filipino by inserting lalaki/babae, mapped queer identities onto bakla/bading/tomboy/lesbiyana, and dropped stereotypes that don't travel. That documentation alone is worth publishing—it's a template for low-resource language adaptation.\n\nThe evaluation runs the standard pseudo-log-likelihood metric on eight models. I checked their arithmetic; the totals match the category weighted averages. That's solid.\n\nThe soft spots: The stress-test concern about lexical frequency is real. The queer labels are lower-frequency than lalaki/babae, and the CrowS-Pairs adaptation inserts explicit gender markers where natural Filipino would use siya. The metric holds the label fixed and sums probabilities over shared tokens, so any effect of label frequency on the shared tokens counts as 'bias.' The paper cites minimal-pair pitfalls but never runs a neutral-template control. The below-chance scores (gpt2 at 17.41 for bading) are a red flag—they suggest the rare label is penalized, not that the model has anti-queer beliefs. Without a control, saying the models 'contain considerable bias' is too strong.\n\nThe pretraining-exposure claim is also correlational. SEALION and RoBERTa-Tagalog have more Filipino tokens and higher bias, but n=2 groups with confounded architecture and size. The body hedges, the abstract doesn't. That should be fixed.\n\nMinor: no uncertainty estimates on the per-category scores, and there are typos in Table 6 (e.g., 'ma STD', 'makaasalanan').\n\nBottom line: this is a solid resource paper that deserves review. The benchmarks will be cited and used. The authors need to either add a neutral control, or at minimum temper the abstract and discuss the frequency confound and below-chance scores.\n\nI'd say send it to a serious venue. Reading group: yes.","headline":"Useful new Filipino bias benchmarks, but the bias scores are over-interpreted given untested frequency confounds.","tokens_in":15120,"tokens_out":3259,"would_cite":true,"duration_ms":32520,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces Filipino CrowS-Pairs and Filipino WinoQueer, the first non-English benchmark for anti-queer bias, and shows that multilingual models, especially those trained on more Filipino text, systematically prefer sexist and…","keywords":["Filipino","bias benchmark","CrowS-Pairs","WinoQueer","multilingual language models","homophobic bias","sexist bias","cultural adaptation"],"falsifier":"A control experiment would settle the core measurement claim: construct matched pairs where the identity label is held constant across both sentences (comparing, for example, two sentences about bakla that differ only in whether the stereotype wording is positive or negative), or replace the identity term in one member with a frequency-matched neutral noun, then recompute the bias score. If the score does not move substantially toward 50 percent, the metric is tracking lexical frequency or structural naturalness rather than stereotype association.","tokens_in":1716,"feed_emoji":"⚖️","tokens_out":2844,"duration_ms":131183,"temperature":0.7,"pith_summary":"This paper tries to establish that sexist and homophobic bias in multilingual language models can be measured in a low-resource language, Filipino, and that such models indeed exhibit considerable bias. To do this, the authors culturally adapt two English bias benchmarks, CrowS-Pairs and WinoQueer, into Filipino, producing 7,074 new sentence pairs and documenting the translation challenges in detail. They then evaluate eight off-the-shelf models and find average bias scores near 59 to 60 percent, meaning models are roughly 1.5 times more likely to prefer biased statements over their less biased counterparts. A further claim is that models pretrained with more Filipino text show stronger bias, and that English-derived queer labels such as tomboy and lesbiyana pick up additional bias through English as an intermediary language.","feed_headline":"AI prefers biased Filipino sentences, new benchmarks show","feed_subtitle":"The first non-English anti-queer benchmark finds AI models favor biased statements.","key_machinery":"The load-bearing measurement is the pseudo-log-likelihood bias score, following Nangia et al. (2020) and Felkner et al. (2023): for each minimally different sentence pair, every unmodified token is masked and its log-probability is summed conditional on the identity-defining tokens, and the bias score is the percentage of pairs where the biased sentence receives a higher score. The cultural adaptation procedure is equally load-bearing: because Filipino is largely gender-neutral (siya covers he and she, asawa covers spouse), the authors insert lalaki and babae as explicit gender markers, and because indigenous queer concepts do not map to Western identity categories, they replace the identity taxonomy with bakla, bading, tomboy, and lesbiyana while discarding stereotypes they deem culturally meaningless. These choices define what the benchmarks can detect and where their measurements can go wrong.","core_discovery":"The central claim is that Filipino CrowS-Pairs and Filipino WinoQueer provide a valid measurement of sexist and anti-queer bias in multilingual models processing Filipino, and that these models are measurably biased. On the paper's own terms, the discovery is threefold: first, the benchmarks are faithful cultural adaptations of their English sources, with the translation process itself revealing structural differences between English and Filipino gender and queerness concepts; second, all eight tested models exhibit bias, on average preferring biased statements about 1.5 times more often than unbiased ones; third, bias magnitude tracks the proportion of Filipino data in a model's pretraining corpus, as the models trained on the most Filipino text score highest on both benchmarks.","pith_inferences":["A natural testable extension is to apply the same adaptation template to other Southeast Asian languages, such as Indonesian or Malay, to see whether English-derived identity words produce the same bias-transfer pattern as tomboy and lesbiyana.","The authors' exposure-bias finding implies that debiasing multilingual models for low-resource languages may require rebalancing or filtering pretraining corpora rather than fine-tuning on small curated datasets, an implication the paper states only implicitly.","Because the paper does not directly test whether frequency differences between rare queer labels and common gendered labels drive the scores, a control experiment with frequency-matched neutral nouns could strengthen or weaken the central measurement claim.","The documented cultural adaptation strategy, particularly the decision to exclude stereotypes that are 'culturally meaningless,' suggests that future benchmarks will need to supplement translated items with locally crowdsourced stereotypes to capture biases unique to each region."],"forward_implications":["Filipino CrowS-Pairs and Filipino WinoQueer give researchers a reusable, documented template for building bias benchmarks in other low-resource languages by adapting English materials rather than starting from scratch.","If the results hold, multilingual models deployed in the Philippines and Southeast Asia carry measurable sexist and anti-queer stereotypes, and the extent of that bias depends on how much local-language data the models were trained on.","The finding about tomboy and lesbiyana suggests that multilingual models can transfer English-language bias onto local identity words that share English morphology, implying that debiasing efforts should account for lexical overlap with English.","The qualitative themes identified, such as queer men portrayed as pedophiles or mentally ill and queer women as promiscuous or sinful, give concrete, testable targets for bias mitigation.","Because masked and causal models with very different sizes all showed bias, the paper argues that model architecture and parameter count matter less than pretraining data composition for low-resource languages."],"supporting_citations":[{"why":"Supplies the CrowS-Pairs benchmark structure and the pseudo-log-likelihood bias score metric that the Filipino benchmark adapts and inherits.","marker":"(Nangia et al., 2020)"},{"why":"Supplies the WinoQueer benchmark and the extension of the bias score procedure to causal language models.","marker":"(Felkner et al., 2023)"},{"why":"Provides the prior model of careful cultural adaptation of CrowS-Pairs into a non-English language, which the authors follow and document.","marker":"(Névéol et al., 2022)"},{"why":"Catalogues benchmark pitfalls such as non-minimal pairs and incorrect bias labels, which the authors say they replicate solutions for during adaptation.","marker":"(Blodgett et al., 2021)"},{"why":"Establishes the Filipino queer identity concepts (bakla, bading, tomboy, lesbiyana) that replace the Western identity labels in the adapted WinoQueer.","marker":"(Garcia, 1996)"},{"why":"Represents prior multilingual gender-bias probing that is limited to binary gender, the gap this paper addresses with an anti-queer benchmark.","marker":"(Steinborn et al., 2022)"},{"why":"Provides the theory that multilingual models use English as an intermediary language, which the paper invokes to explain stronger bias for English-derived labels like tomboy and lesbiyana.","marker":"(Zhao et al., 2024)"}],"fun_headline_variants":["New Filipino benchmarks expose sexist, homophobic AI bias","AI bias in Filipino: models favor stereotypes in new tests","Filipino benchmarks reveal sexist and anti-queer bias in multilingual AI","Training data size predicts bias in Filipino-language AI models"],"cache_read_input_tokens":17024,"weakest_assumption_plain":"The measurement assumes that the probability differences the bias score detects come from learned social stereotypes rather than from the fact that the distinguishing identity words, rare queer labels like bakla and bading versus very common words like lalaki and babae, plus inserted gender markers, differ in naturalness and frequency in Filipino.","fun_headline_variants_meta":{"raw":{"variants":["New Filipino benchmarks expose sexist, homophobic AI bias","AI bias in Filipino: models favor stereotypes in new tests","Filipino benchmarks reveal sexist and anti-queer bias in multilingual AI","Training data size predicts bias in Filipino-language AI models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1407,"prompt_tokens":858,"completion_tokens":549,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":477}},"tokens_in":474,"tokens_out":549,"duration_ms":6541,"temperature":1.0,"reasoning_tokens":477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:56:45.677857+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A control experiment would settle the core measurement claim: construct matched pairs where the identity label is held constant across both sentences (comparing, for example, two sentences about bakla that differ only in whether the stereotype wording is positive or negative), or replace the identity term in one member with a frequency-matched neutral noun, then recompute the bias score. If the score does not move substantially toward 50 percent, the metric is tracking lexical frequency or structural naturalness rather than stereotype association.","supporting_citations":[],"review_version":1}