{"id":"63015c6e-f61c-4589-9a91-ba8554cdcb7d","arxiv_id":"2504.18560","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM-based translation and paraphrasing can effectively augment multilingual bias testing, and low-resource languages tend to show worse bias-detection scores than high-resource ones.","lead":"This paper introduces MLA-BiTe, a framework that uses LLM translation and paraphrasing to expand bias-test templates across languages. It compares four models in six languages, including low-resource ones, and finds lower-resource languages tend to show more detectable bias.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ2's low-resource bias gap is confounded by unvalidated machine-translated templates and expected values for FR/DE/LB, plus unevenly discarded unprocessable responses; the test instrument itself may be what is 'more biased' in low-resource languages.","rationale":"The paper's most valuable contribution is RQ1's demonstration that LLM-based translation and paraphrasing can be operationalized with high semantic similarity for EN/ES/CA (Section 4.3, Figures 1-3); that part has independent support in the reported cosine-similarity and BLEU checks, and the paper is honest about the GPT-4o advantage and the political-template limitations. The weak point is the inference from Section 5 pass rates to the claim that low-resource languages exhibit more bias. For FR, DE, and LB, neither the templates nor the oracle expected values are validated against human ground truth, and Algorithm 2 explicitly translates expected values along with templates. Because LangBiTe scoring is keyword-based, any lexical divergence between the translated oracle and the phrasing an LLM naturally produces will register as a failed, i.e., biased, response. LB's 9.7% unprocessable rate versus 3.3% for EN, plus per-batch faults as high as 66.7%, mean the reported means are computed after uneven exclusion, and no significance testing is reported. The proposed human-reference experiment cleanly separates the instrument effect from the model-bias effect: if replacing machine translations with human translations removes the LB/FR/DE deficit, the central claim is an artifact; if the deficit persists, the claim is strengthened. This matches the reader's weakest_assumption, so no verdict change is recommended.","tokens_in":16520,"tokens_out":5372,"duration_ms":51064,"concrete_test":"Have professional translators (two per language, reconciled) translate the full English LangBiTe template set and expected values into French, German, and Luxembourgish. Replace the GPT-4o-generated templates and oracles in the Section 5 evaluation with these human references and rerun all four LLMs under identical settings, reporting confidence intervals for language means before and after replacement. If the LB/FR/DE pass rates remain significantly below EN/ES/CA, RQ2 survives; if the gaps narrow to within noise, the original low-resource result was an artifact of unvalidated machine translation rather than model bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for RQ2 is that the augmented templates and, crucially, the oracle expected values are equally valid measurement instruments in all six languages. This is only tested for EN/ES/CA. Section 4.3 validates translation quality against human references for ES and CA only; Section 5.1 introduces FR, DE, and LB without any human ground truth, and Algorithm 2 (lines 7-8) machine-translates the expected values used to score responses. If those oracle strings are semantically inexact or contain non-idiomatic keywords in LB/FR/DE, a model can give an unbiased answer that fails keyword matching and is counted as 'more biased.' The reported gap (mean 61.1 for LB vs 75.6 for EN in Figure 5) is therefore not separable from translation quality. This is compounded by Section 5.2's decision to discard unprocessable responses: Table 5 shows LB has 9.7% unprocessable tests vs 3.3% for EN, and Table 6 lists per-batch faults as high as 51.6% (GPT-4o LB racism) and 63.3-66.7% for several sexism batches. Dropping these systematically can bias pass-rate estimates, and no significance testing or confidence intervals are reported. The paper's own Section 5.2 concedes that 14.3% of test batches exceed 10% failure rates. Thus the central cross-language comparison does not yet establish that models are more biased in low-resource languages; it may only establish that machine-translated instruments are less valid there.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MLA-BiTe, a framework that uses LLM-based translation and paraphrasing to expand LangBiTe bias-testing templates into six languages (English, Spanish, Catalan, French, German, Luxembourgish) and applies the augmented templates to four LLMs across seven sensitive categories. RQ1 asks whether automated translation and paraphrasing can reliably augment multilingual bias-test templates and which ordering of the two steps works best; RQ2 asks whether low-resource languages exhibit more detected bias than high-resource languages. The paper reports that the augmentation is viable, that the paraphrasing-then-translation order is marginally better, and that low-resource languages, especially Luxembourgish, show higher apparent bias. It also reports unprocessable-response rates per model, concern, and language, and a language-level correlation analysis. The central claims are practical and timely, but the evidence for RQ2 is weakened by validation gaps for French, German, and Luxembourgish and by the handling of unprocessable responses.","tokens_in":16823,"tokens_out":4550,"duration_ms":41728,"significance":"If the results hold, the framework would be a useful step toward scalable multilingual bias evaluation, lowering the need for native-speaker teams and extending single-language bias-testing tools to low-resource and co-official languages. The paper also provides a useful empirical comparison of four LLMs on seven bias categories across six languages, and it is transparent about processing failures and about the circularity of using GPT-4o as both generator and test subject. However, the central RQ2 conclusion—that low-resource languages show more bias—is not yet established, because the translation/oracle instrument is validated only for English, Spanish, and Catalan, unprocessable responses are discarded at unequal rates across languages, and no statistical tests accompany the headline means. These are load-bearing gaps that can, in principle, be fixed with additional validation and analysis.","major_comments":[{"comment":"The reported pass rates are conditional on successful response processing, and unprocessable responses are discarded at rates that differ strongly across languages and batches: Table 5 reports 3.3% unprocessable for English but 9.7% for Luxembourgish and 9.2% for Spanish, and Table 6 lists per-batch fault rates as high as 63.3–66.7% for several sexism batches and 51.6% for GPT-4o on Luxembourgish racism. Because the RQ2 comparison in Figure 5 uses only the processed responses, the lower mean for Luxembourgish may reflect a systematically noisier measurement instrument rather than higher model bias. The paper should report results with and without discarded responses, or model the missingness, and provide per-language and per-batch denominators and confidence intervals.","section":"Section 5.2, Tables 5 and 6"},{"comment":"Human-reference validation of translation quality is reported only for English, Spanish, and Catalan (Figure 1 and Section 4.3), while the French, German, and Luxembourgish templates and oracle expected values are machine-generated by GPT-4o without human ground truth. Since Algorithm 2 (lines 7–8) also machine-translates the expected values used for keyword matching, an unbiased or correct model response in a language with non-idiomatic or awkwardly translated oracle text can fail the keyword match and be counted as biased. Consequently, the Figure 5 gap (mean 61.1 for Luxembourgish versus 75.6 for English) is not separable from translation and oracle validity. The authors should add human validation or a second independent translation source for FR/DE/LB and perform a sensitivity analysis of the RQ2 result to oracle wording.","section":"Section 4.3 and Section 5.1"},{"comment":"The cross-language comparison that answers RQ2 is presented only as aggregate means with no confidence intervals, effect sizes, or significance tests. Given that there are only four models and that processing-failure rates vary by language, the visual difference between, for example, Luxembourgish and English in Figure 5 could be within the range of sampling variation. The paper should report per-language distributions of pass rates and a statistical comparison, or explicitly justify why such a comparison is not appropriate given the measurement limitations.","section":"Figure 5 and Section 5.2"},{"comment":"The pipeline-ordering decision (T2P versus P2T) is based only on EN–ES and ES–CA translations, and the main evaluation then applies this ordering to French, German, and Luxembourgish without validation for those languages. Similarly, the model-selection procedure does not include the two low-resource languages that are central to RQ2. If the ordering or the chosen generator behaves differently for FR/DE/LB, the conclusions about both RQ1 and RQ2 could change. The paper should either motivate the transfer of the ordering decision to these languages or test it directly.","section":"Section 4.6 and Section 5.2"}],"minor_comments":[{"comment":"The sentence 'a lower BLEU score is actually preferred' is confusing in the context of translation evaluation, where BLEU is normally a similarity-to-reference metric; if the preference for low BLEU refers to paraphrasing, it should not be applied to the translation results in Figure 1.","section":"Section 4.3"},{"comment":"The candidate model is listed as 'Gemini Pro 1.5' in Table 1 but is referred to as 'Gemini 1.5 Flash' in Section 4.3 and in Figures 1–3; the model name should be consistent throughout.","section":"Table 1 and Section 4.3"},{"comment":"Cross-references to 'section 8' and to 'Section 3.2' for the prompt texts do not match the actual appendix structure; the appendices should be numbered or the references updated.","section":"Section 3.1 and Appendix"},{"comment":"The phrase 'which is reported and described in Section 3.2' in the description of Algorithm 3 appears to refer to the paraphrasing prompt, but that prompt is presented later in the appendix; please clarify the reference.","section":"Section 3.2, Algorithm 3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the framework is useful, but the RQ2 claim requires stronger validation of the measurement instrument for French, German, and Luxembourgish and a more careful treatment of discarded responses. I would ask the authors to add human or independent translation validation for all six languages, report missing-data analyses, and include statistical comparisons of Figure 5 before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the Buscemi et al. paper on MLA-BiTe. The one-line take: it is a useful framework paper, but the headline empirical result about low-resource languages being more biased does not hold up yet. The problem is not the pipeline—it is that the scoring oracle is part of what the pipeline translates, and that part is only validated for Spanish and Catalan.\n\nWhat is actually new and good: the integrated translation-plus-paraphrasing pipeline for LangBiTe templates, the T2P vs P2T ordering comparison, and the six-language, four-model evaluation. The methods are described clearly (Algorithms 1-3), the underlying tool is open source, and the RQ1 result—that LLM translation/paraphrasing can produce semantically acceptable templates—is reasonably supported. Also, credit for including Table 6, which lists per-batch unprocessable rates; that kind of disclosure is rare and helpful.\n\nSoft spots, in order of severity. First, the RQ2 claim is structurally confounded. For French, German, and Luxembourgish there is no human ground truth for either the templates or the oracle expected values. Algorithm 2 translates the expected values using the same LLM, GPT-4o. If those oracle strings are semantically off in LB/FR/DE, a model can give a perfectly reasonable answer and fail keyword matching, so it gets scored as biased. The LB vs EN gap in Figure 5 could be an instrument artifact, not model bias. The paper acknowledges the GPT-4o advantage, but does not attempt to quantify it. Second, unprocessable responses are discarded without correction or statistical care. Table 6 shows batches with 63-67% faults for sexism, and overall 14.3% of batches exceed 10% failure. No tests, no confidence intervals. Third, the means in Figure 5 are just means; with only four models and high attrition, the differences are well within noise.\n\nNone of this undermines the RQ1 contribution, and the framework is clearly worth having on the shelf. But the RQ2 conclusion should be flagged as speculative in the paper. I would send it to peer review because the tooling and the ordering analysis deserve scrutiny. The revision needs human validation for the new language oracles, or a statistical treatment of attrition, or a softened claim. I would not cite the low-resource conclusion as evidence.\n\nYours,\n[Name]","headline":"Solid engineering with an honest write-up, but the RQ2 low-resource bias gap is confounded by machine-translated oracles and discarded failures; treat that claim as not yet established.","tokens_in":17390,"tokens_out":3835,"would_cite":false,"duration_ms":33576,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Low-resource languages show more LLM bias in translated tests","keywords":["multilingual bias testing","low-resource languages","machine translation augmentation","paraphrasing templates","social bias in LLMs","LangBiTe","fairness evaluation","cross-lingual bias"],"falsifier":"Translate the same template library by hand into French, German, and Luxembourgish with human-written expected answers, run the identical test set, and compare pass rates with the machine-translated versions; if the low-resource gap shrinks or disappears, the reported bias difference is an artifact of translation rather than a property of the models.","tokens_in":16316,"feed_emoji":"⚖️","tokens_out":5056,"duration_ms":41785,"temperature":0.7,"pith_summary":"The paper introduces MLA-BiTe, a framework that uses a large language model to translate and paraphrase bias-testing prompt templates so the same ethical tests can run in many languages. The authors claim this automation is reliable: GPT-4o's translations and paraphrases preserved semantic meaning well enough to serve as test instruments, and the order of translation versus paraphrasing mattered little. Using the augmented templates, they test four LLMs in six languages, including two low-resource ones, and find that models pass fewer bias tests — that is, exhibit more detectable bias — in Catalan and especially Luxembourgish than in English. If correct, the result means organizations with multilingual obligations can audit AI bias without hiring native-speaking test authors, and it warns that models behave worse in languages with less training data.","feed_headline":"Low-resource languages show more LLM bias in translated tests","feed_subtitle":"Four models, six languages: Luxembourgish and Catalan score worst on AI bias tests built by machine translation.","key_machinery":"The carrying mechanism is the MLA-BiTe pipeline: each prompt template is translated into a target language and then paraphrased a chosen number of times at the template level, before community placeholders are filled in, so that all generated test prompts stay syntactically uniform. The paraphraser first identifies the template's grammatical number, the singular or plural form of the sentence, to avoid pluralization errors such as 'Is a {GENDER1} better than a {GENDER2}?'. Test outcomes are measured by LangBiTe's oracle mechanism: every template carries an expected non-biased response, and a test is counted as passed when the model's answer matches that expectation, yielding a pass rate per language and bias category.","core_discovery":"On its own terms, the paper's central claim is that LLM-based translation and paraphrasing can automatically expand a curated bias-test template library into new languages while preserving the semantic meaning that makes the test valid, and that doing so reveals higher social bias in low-resource languages. In the main evaluation, English and Spanish yielded the highest and most stable pass rates across all four tested models, while Luxembourgish showed the highest discrimination rates overall and Catalan varied widely by category; Llama3 405B was the least fair across languages. A correlation analysis adds that models' bias patterns do not track language families: Luxembourgish results correlate more closely with Spanish and Catalan than with German or English.","pith_inferences":["Because GPT-4o generated the translations and paraphrases, its own later test scores may be inflated relative to other models; the paper flags this as a possible confound, and a fair comparison would use per-model translators or human translations.","The unvalidated French, German, and Luxembourgish templates raise the possibility that some of the measured 'bias gap' is translation artifact; testing with human-translated templates would isolate the model-bias component.","The same template-level augmentation approach could extend to non-Indo-European low-resource languages, but the method's reliance on grammatical-number identification would need to handle noun classes and richer morphology first.","A practical follow-up would test whether fine-tuning on low-resource-language data reduces the pass-rate gap, which would confirm the training-data-resource explanation suggested by the results."],"forward_implications":["Organizations can run bias audits in co-official or low-resource languages such as Luxembourgish and Catalan without recruiting native-speaking prompt authors for every language.","Model selection for fairness should be made per language and per bias category, since rankings shift: Gemini 1.5 Flash excels in religion and LGBTIQ+ tests while GPT-4o leads in ageism, sexism, and xenophobia.","Low-resource languages are where bias mitigation efforts will matter most, because all four tested models showed more detectable bias there.","Bias patterns do not transfer along language-family lines, so treating German and Luxembourgish as similar for fairness testing would be a mistake; the data show Luxembourgish behaving more like Spanish and Catalan.","Politics is the least reliable category: the source templates are US-centric and expect neutrality, so cross-lingual political-bias results should be read with caution."],"supporting_citations":[{"why":"Supplies the LangBiTe template design, oracle concept, and evaluation method that MLA-BiTe builds on.","marker":"[19]"},{"why":"Provides the concrete test-case library in English, Spanish, and Catalan that MLA-BiTe translates and paraphrases.","marker":"[18]"},{"why":"Offers the multilingual sentence-embedding model used to compute cosine similarity between machine and human translations.","marker":"[25]"},{"why":"Motivates the augmentation step by showing that social bias evaluation requires prompt variations.","marker":"[12]"},{"why":"Supports the expectation that low-resource languages perform worse in cross-lingual transfer settings.","marker":"[14]"},{"why":"Describes the LangBiTe platform that serves as the target framework adapted in the study.","marker":"[20]"}],"fun_headline_variants":["LLM bias tests go multilingual: low-resource languages fail hardest","Luxembourgish torments LLMs: bias tests across six languages","Machine translation exposes hidden bias in low-resource LLMs","Language gap: LLM bias higher in low-resource tongues","MLA-BiTe: auto-translated bias tests expose low-resource LLM gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that lower pass rates in French, German, and Luxembourgish reflect model bias assumes the machine-translated templates and their machine-translated expected answers are as valid as human-authored ones, but human ground truth exists only for English, Spanish, and Catalan, so translation quality may be partly driving the language gap.","fun_headline_variants_meta":{"raw":{"variants":["LLM bias tests go multilingual: low-resource languages fail hardest","Luxembourgish torments LLMs: bias tests across six languages","Machine translation exposes hidden bias in low-resource LLMs","Language gap: LLM bias higher in low-resource tongues","MLA-BiTe: auto-translated bias tests expose low-resource LLM gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00074,"raw_usage":{"total_tokens":3219,"prompt_tokens":779,"completion_tokens":2440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":395,"completion_tokens_details":{"reasoning_tokens":2346}},"tokens_in":395,"tokens_out":2440,"duration_ms":15631,"temperature":1.0,"reasoning_tokens":2346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:50:11.117007+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Translate the same template library by hand into French, German, and Luxembourgish with human-written expected answers, run the identical test set, and compare pass rates with the machine-translated versions; if the low-resource gap shrinks or disappears, the reported bias difference is an artifact of translation rather than a property of the models.","supporting_citations":[{"cited_title":"A DSL for testing LLMs for fairness and bias","cited_arxiv_id":null,"evidence_quote":"Supplies the LangBiTe template design, oracle concept, and evaluation method that MLA-BiTe builds on."},{"cited_title":"Langbite, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the concrete test-case library in English, Spanish, and Catalan that MLA-BiTe translates and paraphrases."}],"review_version":1}