{"id":"089612ca-71da-4433-9b99-9024d23285bc","arxiv_id":"2502.01679","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A corpus-derived benchmark and metric, EiCAT, for measuring LLM bias in local cultural contexts, applied to 167,712 New Zealand test cases.","lead":"The authors introduce LIBRA, a pipeline that builds local-language bias tests for LLMs from news corpora, and EiCAT, a score that combines an existing bias metric with a measure of whether models understand local words. They apply it to New Zealand English and Maori text, finding that common LLMs frequently misunderstand local vocabulary and that Llama-3 handles it best among tested causal models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The JSD bias term in EiCAT (Eq. 3) is confounded by unvalidated anti-stereotyped sentence fluency; the paper never quantifies how many triplets have fluent, genuinely counter-stereotypical S_p.","rationale":"The reader's weakest assumption is the same as the most load-bearing concern: the anti-stereotyped sentences may be ungrammatical or not counter-stereotypical, making JSD a fluency measure. I agree with this diagnosis and locate it precisely in Eq. (3) and the dataset construction in Section 3. The paper's own Section 4 admission of 'noise' without quantifying it, and the apparent contradiction with the claim that experts review all triplets, strengthens the concern. The concrete test is a clean, feasible validation: if the EiCAT ranking is stable after filtering to validated triplets, the metric is robust; if not, the central claim is unsupported. The paper's strengths—a large automatically built local dataset, the bbs mechanism, and open code—are real, but they do not resolve the validity of S_p. Since the concern is empirically testable and the authors could address it in a revision, the conditional verdict stands. I would not escalate to reject because the paper provides a concrete framework and the flaw is an unvalidated assumption rather than a logical contradiction. UNCHANGED is appropriate.","tokens_in":12632,"tokens_out":6185,"duration_ms":60398,"concrete_test":"Sample 300 triplets from the released GitHub dataset. Have two annotators fluent in NZ English and Te Reo Maori independently rate each S_p for (i) fluency and (ii) whether it expresses a counter-stereotypical (non-stereotypical) statement about the target group. Compute inter-annotator agreement; retain only triplets where both agree S_p is fluent and counter-stereotypical. Recompute Ds/Da, JSD, and EiCAT (Eq. 3) for all models in Table 1 on this clean subset. If Llama-3-8B remains the top causal model and the relative ordering is unchanged, the concern is resolved. If rankings shift materially (e.g., GPT-2-xl's JSD increases or Llama-3's EiCAT drops below GPT-2), the JSD term is not a valid bias measure and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EiCAT (Eq. 3) measures bias, with Llama-3-8B as the best causal model in the New Zealand context. The JSD term computes divergence between pooled log-likelihood distributions of stereotyped (S) and anti-stereotyped (S_p) sentences. This is only a bias measure if S_p is a fluent, valid counter-stereotypical counterpart. The pipeline constructs S_p by replacing the demographic token with another term from the same keyword set, selecting 'distant' terms in embedding space when no antonym exists; it never checks grammaticality or whether the resulting sentence is genuinely counter-stereotypical. Section 3 ('Compile Dataset') claims experts review all triplets, but Section 4 states the dataset 'recognises noise' and relies on collective distribution, so invalid swaps remain. For CLMs, Eq. 1 computes average log-probability over the right context; for MLMs, pseudo-log-likelihood. An awkward or merely different-group S_p systematically lowers L(S_p) relative to L(S), shifting Da left of Ds and inflating JSD. Since EiCAT = lms * [alpha * (1 - JSD) + (1 - alpha) * bbs], the metric rewards models that happen to assign similar probabilities to fluent and ungrammatical sentences, not models that are unbiased. The paper offers no estimate of the fraction of the 167,712 triplets with valid S_p, so the central quantity JSD may measure fluency, not bias. This is the load-bearing weakness because every reported EiCAT ranking depends on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LIBRA, an automated framework for building local-context bias datasets from local news corpora without crowdsourcing, together with a new metric, the Enhanced Idealized CAT Score (EiCAT). EiCAT combines the StereoSet language model score (lms), a Jensen-Shannon divergence (JSD) between log-likelihood distributions of stereotyped and anti-stereotyped sentences, and a Beyond Knowledge Boundary Score (bbs), as defined in Eq. (3). The authors construct a New Zealand dataset of 167,712 triplets from 367,384 local articles and evaluate several open-source LLMs (BERT family, RoBERTa, ALBERT, GPT-2, Llama-3), reporting that Llama-3-8B has the highest EiCAT among causal language models in the New Zealand context.","tokens_in":13013,"tokens_out":4983,"duration_ms":53796,"significance":"If the proposed measure is valid, LIBRA addresses a genuine gap: most bias benchmarks are US-centric, while local cultural contexts introduce both unfamiliar vocabulary and different stereotype structures. The dataset is large, the pipeline is automated and reproducible in principle, and the code/data are publicly released, which are concrete strengths. The attempt to separate knowledge-boundary effects from stereotypical bias is also a valuable direction. However, the central claim that EiCAT quantitatively measures bias is not currently established: the construction of anti-stereotyped sentences is not validated, the bbs verification procedure is underspecified, the choice of alpha is circular, and no uncertainty estimates are reported. These issues are load-bearing because every EiCAT ranking in Table 1 depends on them.","major_comments":[{"comment":"The anti-stereotyped sentence S_p is generated by replacing the demographic token with another term from the same keyword set (e.g., a distant embedding neighbor when no antonym exists), without checking whether the resulting sentence is fluent, grammatical, or genuinely counter-stereotypical. The paper acknowledges 'noise' in the dataset and relies on 'collective distribution' to mitigate it, but that does not address the confound: if a large fraction of S_p sentences are ungrammatical or implausible, L(S_p) will be systematically lower than L(S), shifting D_a relative to D_s and inflating JSD in Eq. (3) regardless of any stereotype. The statement that 'consistent response to poorly flowing sentences confirms that our metric accurately captures model biases' is not a substitute for validation. The authors should quantify the fraction of triplets with valid S_p (e.g., via human or LLM annotation on a sample), report inter-annotator agreement, and show that the JSD-based ranking is stable when restricted to validated triplets.","section":"Section 3, 'Compile Dataset'; Section 4, paragraph beginning 'Our dataset recognises noise'"},{"comment":"Setting alpha equal to bbs makes EiCAT a function of the very quantity it is supposed to balance: EiCAT = lms * [bbs*(1-JSD) + (1-bbs)*bbs] (with appropriate scaling). When bbs = 0, EiCAT becomes 0 regardless of lms and JSD, so models with perfect language scores and zero measured bias are ranked the same as random models (see Table 1: RandomLM, IdealLM, OV BERT-large, and OV GPT-2-xl all receive EiCAT = 0 solely because bbs = 0). This conflates knowledge-boundary failure with bias and makes the metric's ranking largely a reordered bbs ranking. The paper needs a principled justification for this choice, a sensitivity analysis over alpha, or a different weighting scheme.","section":"Section 3, Eq. (3) and the paragraph following it"},{"comment":"The bbs computation depends on a binary function f(W, S_w) that compares an LLM-generated definition D1 with an official definition D2, but the paper never specifies the prompt template P2, the matching threshold, or which LLM is used as the verifier. The statement 'if D1 and D2 align' is undefined: exact string matching, semantic similarity, and human judgment would all give different bbs values. Because EiCAT is multiplied by bbs and alpha, the entire ranking in Table 1 is sensitive to this unspecified step. The authors should provide the full verification prompt, the similarity measure and threshold, the verifier model, and, ideally, a small human agreement study.","section":"Section 3, Eq. (2) and the bbs verification procedure"},{"comment":"No uncertainty estimates are reported for any metric. Many reported differences are small on the 0-100 scale (e.g., BERT-base EiCAT 5.73 vs. BERT-large 5.91; GPT-2-large 1.61 vs. GPT-2-xl 1.68), and it is not clear whether these differences are meaningful given sampling variability in lms, JSD, and bbs. The paper should report bootstrap confidence intervals or statistical tests for the key comparisons, and it should state clearly which cross-model comparisons are permitted, given the admission that JSD values 'cannot be directly compared across masked and causal models.'","section":"Table 1 and Section 5, 'LLM Bias in New Zealand Context'"}],"minor_comments":[{"comment":"There are several typos: 'ch llenge' in the abstract, 'ocal bias' in Section 1, and inconsistent caption formatting in Fig. 1. These should be corrected in revision.","section":"Abstract and Section 1"},{"comment":"Reference [31] is cited for Llama-3 in the Introduction and Results, but the reference list entry [31] points to Vidit et al., 'CLIP the Gap: A Single Domain Generalization Approach for Object Detection,' which is not the Llama model paper. The authors should cite the correct Llama-3 reference.","section":"References"},{"comment":"The caption states that iCAT is computed 'using the StereoSet dataset for comparison,' which explains why the iCAT column is identical across the NZ, OV, and Malay panels. This should be stated more explicitly in the table itself or in a footnote, because as presented the repeated values may confuse readers.","section":"Table 1 caption"},{"comment":"The text says EiCAT ranges from 0 to 1, but Table 1 presents all metrics scaled to 0-100 and EiCAT values are on the 0-100 scale there. The scaling convention should be stated explicitly when Eq. (3) is introduced, and the definitions of lms and bbs as either probabilities or percentages should be consistent.","section":"Section 3, Eq. (3)"},{"comment":"The claim that the automatic pipeline 'ensuring grammatical diversity' is stronger than the evidence supports, given that grammaticality of the generated S_p sentences is not verified. A more cautious wording would be appropriate.","section":"Section 2, Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and under-served problem, and the released dataset is a potentially valuable community resource. My main concern is that the central metric EiCAT may be measuring sentence fluency and knowledge-boundary failure rather than stereotype bias, and the current manuscript does not provide the validation needed to rule this out. I would be willing to reconsider after a revision that quantifies the validity of the anti-stereotyped sentences, specifies the bbs verification procedure, reports uncertainty, and re-examines the choice of alpha."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth your time for one reason: it ships a real new artifact. The authors build a pipeline that turns a local news corpus into a stereotype triplet benchmark without crowdsourcing, and they release a 167k-case New Zealand English dataset plus code. That is a genuine gap in bias evaluation, which is heavily U.S.-centric. The bbs idea — measuring whether a model actually knows the local words before scoring its bias — is a good instinct, and the separate reporting of lms, JSD, and bbs is the right way to keep the components visible.\n\nThe soft spots are real, though. The largest is that the anti-stereotyped sentences are never validated. The paper says experts review the triplets, but then says the dataset 'recognises noise' and relies on the collective distribution. No number is given for how many swapped sentences are actually fluent or genuinely counter-stereotypical. Since the JSD term compares log-likelihood distributions of stereotyped vs anti-stereotyped sentences, a model that treats the swaps as odd phrasing will look 'biased' when it is really just responding to fluency differences. That confound is load-bearing for every EiCAT ranking.\n\nThere is also a circularity in the metric: Eq. 3 sets alpha equal to bbs, so EiCAT includes bbs both as a weighting factor and as a scored term. That makes EiCAT partly a function of the quantity it is supposed to temper. The separate bbs results save the paper from incoherence, but the combined score needs a clearer interpretation.\n\nMinor issues: the iCAT column in Table 1 appears to be computed on StereoSet, not on this dataset, so those numbers should not sit in a table with the other columns. The abstract says 'over 360,000 test cases' while the text says 167,712 triplets; the 367k figure is the article count, not the test-case count. No error bars or significance tests, but for an exploratory benchmark that is forgivable. The citation pattern is fine; StereoSet and CrowS-Pairs are the right anchors, and local-context prior work is covered.\n\nWho should read it: anyone building local-context fairness benchmarks or comparing models for non-English markets. The dataset and pipeline are the contribution; the EiCAT metric needs another iteration. I would send it to review, but with a request for human validation of a sample of the triplets and a reworked metric or a clear defense of the alpha choice.\n\nBest.","headline":"New local-context bias benchmark worth engaging; EiCAT metric needs validation and de-circularing.","tokens_in":13493,"tokens_out":3277,"would_cite":true,"duration_ms":32063,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new bias score separates stereotyping from ignorance of local words.","keywords":["large language model bias","local context","knowledge boundary","Jensen-Shannon divergence","New Zealand","Māori language","bias evaluation","dataset construction"],"falsifier":"Sample 200 anti-stereotyped triplets from the released New Zealand dataset and have fluent speakers rate the swapped sentence for grammaticality and counter-stereotypicality; if more than about 20% are judged unnatural or not clearly counter-stereotypical, the JSD component of EiCAT is contaminated and the reported rankings (Llama-3-8B 10.72 vs GPT-2-xl 1.68) would need to be recomputed on the natural subset.","tokens_in":12452,"feed_emoji":"⚖️","tokens_out":6619,"duration_ms":60615,"temperature":0.7,"pith_summary":"Bias benchmarks for large language models mostly use American English and assume the model knows the words in the test prompt. The paper argues that when a prompt contains a local word outside the model's training data, the model's bizarre completions are ignorance, not stereotype, and a fair bias test must tell these apart. It introduces LIBRA, a pipeline that builds bias test cases from local news corpora without crowdsourcing, and EiCAT, a score that combines the usual stereotype-preference measure with a 'beyond knowledge boundary' score and a Jensen–Shannon divergence between stereotyped and anti-stereotyped logit distributions. On a New Zealand dataset of over 160,000 test cases, the method reports that Llama-3-8B is the best causal model (EiCAT 10.72 versus GPT-2-xl's 1.68), mainly because it recognizes more local vocabulary. The claim matters because it implies that bias rankings change once a model's knowledge of local terms is accounted for.","feed_headline":"New bias score separates stereotyping from local-word ignorance","feed_subtitle":"EiCAT folds stereotype divergence with vocabulary-knowledge score, changing which models look fair in New Zealand.","key_machinery":"The central object is the EiCAT score of Eq. (3), which combines the Idealized CAT Score's language-model term with a beyond-knowledge-boundary score (bbs) and a Jensen–Shannon divergence $\\mathrm{JSD}(D_A\\|D_S)$. The bbs measures the fraction of local words in the test sentences that the model interprets in agreement with their formal definitions, and $\\alpha$ is set equal to bbs so that models ignorant of local terms are automatically down-weighted on the bias component. The dataset machinery is the triplet-construction pipeline: sentences from local corpora are searched by keywords per social group, and the demographic token $\\omega$ is replaced by another demographic term $M_p$ (anti-stereotyped) or an unrelated term $M_u$ (irrelevant), producing the three sentences $S$, $S_p$, and $S_u$ that the logit distributions are computed over.","core_discovery":"The central claim is that a fair local-context bias measurement must separate a model's tendency to prefer stereotyped completions from its failure to recognize local vocabulary. The paper formalizes this with the Enhanced Idealized CAT Score (EiCAT), defined as $\\mathrm{EiCAT} = \\mathrm{lms}\\cdot[\\alpha(1-\\mathrm{JSD}(D_A\\|D_S)) + (1-\\alpha)\\mathrm{bbs}]$, where lms is the language model score, $\\mathrm{JSD}(D_A\\|D_S)$ is the Jensen–Shannon divergence between the logit distributions for anti-stereotyped and stereotyped sentences, bbs is the fraction of out-of-vocabulary local words the model correctly understands, and $\\alpha$ is set equal to bbs. A model that knows more local words gets a higher bbs, which raises $\\alpha$ and therefore puts more weight on the bias term; a model that fails on local vocabulary is penalized twice, once through low bbs and once through the weighting. The authors report that on their New Zealand dataset, all tested models understand a small fraction of local words, and Llama-3-8B achieves the highest EiCAT among causal models because it handles Māori vocabulary better than GPT-2 or the BERT family.","pith_inferences":["Editorial extension: if the assumption of fluent anti-stereotyped sentences fails, the JSD term would partly measure grammaticality; one could test this by adding a human-fluency rating layer to a sample of the dataset and recomputing EiCAT on the subset rated fluent.","Editorial extension: the beyond-knowledge-boundary idea generalizes to any jargon the model rarely sees, such as medical or legal terms, where hallucination could also contaminate bias measurements.","Editorial extension: the framework predicts that fine-tuning a model on local text should raise its EiCAT mainly through bbs even if stereotype divergence stays flat, which could be checked with a before/after fine-tuning experiment.","Editorial extension: because the definition-verification step uses an LLM to judge alignment with the formal definition, that verifier model could itself hallucinate; a human-annotated sample of word definitions would bound this error."],"forward_implications":["Bias rankings across models change when local vocabulary knowledge is scored: in the New Zealand context Llama-3-8B outperforms GPT-2-xl on EiCAT despite showing a larger stereotype divergence, because its higher bbs outweighs the higher JSD.","Existing U.S.-centric benchmarks may misclassify a model as biased when it is simply unfamiliar with the local terms, so bias reports should be accompanied by a knowledge-boundary score.","The pipeline lets researchers build culture-specific bias datasets from local news and broadcast corpora without paying crowd workers, lowering the barrier to testing LLMs in under-resourced languages.","Test cases containing local words the model misunderstands can be identified and either filtered or separately scored, making the bias measurement less noisy.","More linguistically competent models tend to show higher JSD, so a model that improves at language modeling may appear more biased unless the knowledge-boundary correction is applied."],"supporting_citations":[{"why":"Supplies the iCAT score and triplet structure (stereotyped, anti-stereotyped, irrelevant) that EiCAT extends.","marker":"[20]"},{"why":"Provides the CrowS-Pairs probability-based bias measurement approach that the paper contrasts with its distributional method.","marker":"[21]"},{"why":"Motivates the 'beyond knowledge boundary' problem, showing how models answer beyond their factual knowledge.","marker":"[28]"},{"why":"LLM2Vec is used to expand seed keywords and to embed sentences for clustering in the dataset pipeline.","marker":"[1]"},{"why":"HDBSCAN is used to cluster corpus articles into topics associated with target social groups.","marker":"[17]"},{"why":"UMAP is used to reduce embedding dimensions before clustering in the pipeline.","marker":"[18]"}],"fun_headline_variants":["Bias score that separates stereotypes from word ignorance","New metric: disentangle LLM bias from local-word gaps","EiCAT: a fairer bias test for non-US cultures","Local bias metric: when models don't know the vocabulary","Separating stereotype bias from vocab ignorance in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that replacing the demographic word with another group's term yields a fluent, genuinely counter-stereotypical sentence; when the swap creates odd phrasing, the JSD term will measure grammaticality rather than bias.","fun_headline_variants_meta":{"raw":{"variants":["Bias score that separates stereotypes from word ignorance","New metric: disentangle LLM bias from local-word gaps","EiCAT: a fairer bias test for non-US cultures","Local bias metric: when models don't know the vocabulary","Separating stereotype bias from vocab ignorance in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1592,"prompt_tokens":1088,"completion_tokens":504,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":704,"completion_tokens_details":{"reasoning_tokens":424}},"tokens_in":704,"tokens_out":504,"duration_ms":5432,"temperature":1.0,"reasoning_tokens":424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T18:09:14.418316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample 200 anti-stereotyped triplets from the released New Zealand dataset and have fluent speakers rate the swapped sentence for grammaticality and counter-stereotypicality; if more than about 20% are judged unnatural or not clearly counter-stereotypical, the JSD component of EiCAT is contaminated and the reported rankings (Llama-3-8B 10.72 vs GPT-2-xl 1.68) would need to be recomputed on the natural subset.","supporting_citations":[{"cited_title":"In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing","cited_arxiv_id":null,"evidence_quote":"Provides the CrowS-Pairs probability-based bias measurement approach that the paper contrasts with its distributional method."},{"cited_title":"In:FirstConferenceonLanguageModeling(2024),https://openreview.net/forum? id=IW1PR7vEBf 14 B","cited_arxiv_id":null,"evidence_quote":"LLM2Vec is used to expand seed keywords and to embed sentences for clustering in the dataset pipeline."},{"cited_title":"The Journal of Open Source Software 3(29), 861 (2018)","cited_arxiv_id":null,"evidence_quote":"UMAP is used to reduce embedding dimensions before clustering in the pipeline."}],"review_version":1}