{"id":"b1d0d16d-84a0-4c42-9dd9-9b441281307f","arxiv_id":"2501.04662","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Arab cultural entities that double as everyday Arabic words are harder for language models to recognize, especially when tokenized as single tokens.","lead":"This paper introduces CAMeL-2, a parallel Arabic-English benchmark of 58,086 entities, and uses it to show that language models recognize Arab cultural entities worse in Arabic than in English. The authors trace this gap to Arabic word polysemy, overlap with other Arabic-script languages, and tokenizer vocabulary size, rather than pre-training frequency alone.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal claim that tokenization independently drives the Arabic performance drop is not established: Figure 8 compares models that differ in vocabulary size, training data, and scale simultaneously.","rationale":"I read the paper as an empirical, correlational study of possible origins of entity-related cultural bias, not as a controlled causal experiment. The strongest contribution is the CAMeL-2 benchmark and the converging descriptive analyses showing that polysemous, high-frequency, one-token Arab entities are harder for LMs in Arabic than their English counterparts. The reader's weakest assumption, the mC4 proxy for pretraining data, is acknowledged by the authors and would matter for the frequency-stratification and lexical-overlap analyses. I do not dispute that it is load-bearing, and I agree the verdict should remain conditional on releasing data/code and addressing that proxy. However, I found a more internal concern: the paper's abstract states that frequency-based tokenization 'leads to' the issue and 'gets worse with larger Arabic vocabularies,' yet the support is an across-model scatter with no within-family control. Vocabulary size, training-data mix, and model scale are not independent in Figure 8, and the multi-token comparison in Figure 7 does not hold frequency or polysemy fixed. This is not an internal inconsistency, but it is a correctness risk in the causal language. The proposed matched-family test would settle whether the tokenization conclusion is real or an artifact of model family differences. Since this concern is addressable and does not invalidate the benchmark or the descriptive bias measurements, the appropriate action is to keep the paper conditional rather than accept or reject it outright.","tokens_in":23380,"tokens_out":7928,"duration_ms":81331,"concrete_test":"Run the extractive-QA evaluation of Section 4.4 on the same one-token Arab location entities using Llama-2-13B and AceGPTv1.5-13B, two models that share the Llama-2 base architecture and differ mainly in an expanded Arabic vocabulary plus continued Arabic training, with identical prompts, greedy decoding, and scoring. If AceGPTv1.5 does not underperform Llama-2-13B on these entities, the Figure 8 vocabulary-size trend is confounded by model design and the tokenization causal claim fails; if it underperforms despite additional Arabic training, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 concludes that frequency-based tokenization causes the observed struggle with Arab entities and that larger Arabic vocabularies make it worse, but the evidence is cross-model correlation, not controlled comparison. Figure 8 plots one point per model, with vocabulary size varying together with model family, pretraining corpus, and parameter count: the low-vocabulary Llama-3.3-70B is a 70B model, while the high-vocabulary ARBERT is a small Arabic-only encoder; the mid-vocabulary JAIS-13B differs in training data and size. The one-token vs multi-token comparison in Figure 7 also leaves entity frequency and polysemy uncontrolled, and one-token entities are generally the most frequent and most polysemous. Since frequency, polysemy, and tokenization are collinear, the unique contribution of tokenization to the central claim is not identified. The mC4 proxy flagged by the reader is a real limitation for the frequency analyses, but the tokenization conclusion would remain questionable even if the true pretraining distribution were known.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates why entity-related cultural biases in language models are more pronounced in Arabic than in English, and attributes the gap to pre-training frequency, Arabic word polysemy, lexical overlap with other Arabic-script languages, and subword tokenization. The authors introduce CAMeL-2, a parallel Arabic-English benchmark of 58,086 cultural entities and 367 masked contexts, and evaluate several causal and encoder LMs on text-infilling, extractive QA, and NER. They report that Arab-Western performance gaps are consistently smaller in English than in Arabic, and that high-frequency entities in the mC4 corpus are harder for LMs in Arabic, that polysemous Arabic location names remain hard even for large models, that entities frequent in other Arabic-script languages cause additional drops, and that one-token tokenization with large Arabic vocabularies is associated with lower accuracy. The paper frames these analyses as uncovering the origins of cultural bias in terms of frequency, linguistic phenomena, and tokenization.","tokens_in":23537,"tokens_out":4741,"duration_ms":48099,"significance":"If the findings hold, this is a useful contribution: it extends cultural-bias analysis beyond representational imbalance to concrete linguistic mechanisms, introduces a reusable parallel benchmark, and provides convergent evidence across multiple model families and tasks. The paper is transparent about the mC4 proxy for pre-training data in §4.1, releases the new resource, and grounds the polysemy analysis in Arabic-specific linguistic facts. The main caveat is that the tokenization section supports only an association, not the causal claim stated in the abstract and conclusion; the vocabulary-size comparison in Figure 8 lacks any controlled manipulation, and the one-token versus multi-token comparisons are confounded with frequency and polysemy. Because tokenization is one of the paper's headline explanatory factors, this overclaim is load-bearing and needs to be fixed or softened before publication.","major_comments":[{"comment":"The claim that \"frequency-based tokenization leads to this issue in LMs, which gets worse with larger Arabic vocabularies\" is not supported by the presented comparisons. Figure 8 plots one point per model, with vocabulary size varying simultaneously with model family, parameter count, architecture, training data, and tokenizer (Llama-3.3-70B, Aya23-35B, AceGPTv1.5-13B, JAIS-13B for QA; XLM-R, CAMeLBERT, AraBERT, MARBERT, ARBERT for NER), so vocabulary size is not isolated as a cause. Figure 7 and Appendix Figure 18 compare one-token versus multi-token entities within a model, but one-token entities are typically the most frequent and most polysemous entities, so token count is confounded with the variables already identified in §§4.1-4.2. To support the causal statement, the authors need a controlled comparison, such as models with the same architecture and training data but different vocabulary sizes, or matched entity sets that vary token count while holding frequency and polysemy fixed. Without such evidence, the tokenization claim should be presented as a suggestive correlation rather than a demonstrated mechanism.","section":"§4.4, Figures 7-8"},{"comment":"The frequency-stratified analysis and the Arabic-script lexical-overlap analysis depend entirely on mC4 counts as a proxy for the pre-training corpora of Llama, Qwen, Aya, and AceGPT. The paper acknowledges this approximation in §4.1, but it does not assess how sensitive the conclusions are to the choice of proxy. If the entity frequency distribution in mC4 differs from the actual training distribution of these models, the U-shaped accuracy curves and the cross-script frequency trends could reflect properties of the proxy rather than the mechanism claimed. At minimum, the authors should validate the counts against at least one other public web corpus (for example, RedPajama, SlimPajama, or Falcon RefinedWeb) and report whether the high-frequency drop and the script-overlap trend survive under that alternative proxy.","section":"§4.1, Figure 3 and §4.3, Figure 6"},{"comment":"The causal wording around frequency and polysemy is stronger than the observational evidence warrants. The high-frequency bin in Figure 3 is heavily populated by common Arabic words that also have non-entity senses, as the authors themselves note, so the performance drop cannot be attributed to frequency alone without controlling for polysemy and tokenization. The polysemy analysis in §4.2 uses the top-100 most frequent locations per country, but countries differ in their overall frequency distributions and in the prevalence of polysemous names, so the comparison between polysemous and non-polysemous entities does not hold frequency constant. A stratified analysis or a regression with frequency, polysemy, and script-overlap as covariates, together with confidence intervals, would substantially strengthen the central claim. The hand-chosen log-spaced frequency bins in §4.1 also deserve a robustness check.","section":"§4.1 and §4.2"}],"minor_comments":[{"comment":"The text reads \"88.34% of entities in CAMel-2\", but the benchmark is consistently capitalized as CAMeL-2 elsewhere; this inconsistent capitalization appears in at least one other place in the manuscript.","section":"§2.2"},{"comment":"The combined y-axis label \"QA Accuracy\" and \"NER F1\" is ambiguous because the two panels use different metrics and scales; each panel should carry its own axis label, and the small number of points should be supplemented with error bars or per-model confidence intervals.","section":"Figure 8"},{"comment":"The reported accuracy and F1 differences are not accompanied by any measure of uncertainty, even though the QA evaluation samples 1,000 entities per type and the underlying entity sets vary in size; standard errors or bootstrap intervals would help the reader judge whether the small English-language gaps are meaningful.","section":"§3.3, Tables 3 and 5"},{"comment":"The prose first says that tokenization \"likely\" makes recognition harder and then the conclusion states the mechanism as established; the hedging should be aligned with the evidence, or the controls needed for a causal statement should be added.","section":"§4.4"},{"comment":"The x-axis \"% Polysemous Words\" is used for both Arab and Western panels, but Western locations are almost uniformly non-polysemous; adding entity counts and country labels to the panels would make the relationship easier to assess.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for a computational-linguistics venue and the CAMeL-2 resource is a valuable community contribution. The main risk is the overclaim in §4.4 about tokenization as an independent cause; if the authors reframe that section as correlational or add the missing controlled comparisons, I would support publication. The mC4 proxy limitation is real but partially acknowledged; requiring a validation corpus would be a reasonable condition. I would also encourage the editor to ask for a robustness check on the frequency bins and for uncertainty estimates in the main comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nQuick take: the polysemy story is real and worth building on; the tokenization story is the weakest part of the paper and should not be cited as causal. The paper does introduce a genuinely useful resource and a set of convergent analyses that make the central claim plausible.\n\nWhat's new: CAMeL-2, a parallel Arabic-English benchmark that triples CAMeL's entity coverage and adds longer extractive-QA contexts. The finding that Arabic entities which double as common words (polysemous) cause bigger performance drops than non-polysemous ones, across multiple LMs and across countries stratified by linguistic influence, is a solid empirical contribution. The English-versus-Arabic comparison showing much smaller gaps in English is also clean and reproduced across tasks and model families. The authors are appropriately careful in the limitations section about rare cases where Western transliterations collide with Arabic words.\n\nSoft spots: the mC4 proxy for pre-training data is acknowledged but it is load-bearing for the frequency and lexical-overlap analyses. If the true training distribution differs, those stratifications could shift. That is a known limitation, not a hidden one. The bigger issue is Section 4.4. The claim that frequency-based tokenization causes the drop, and that larger Arabic vocabularies make it worse, rests on cross-model comparisons in Figure 8 where model family, scale, and training data all vary at once. The within-model token-count comparison in Figure 7 is more controlled, but one-token entities are exactly the most frequent and most polysemous ones, so tokenization is collinear with the two factors already identified. The paper does not disentangle them. The tokenization conclusion should be framed as suggestive, not demonstrated.\n\nWho this is for: people working on multilingual fairness, Arabic NLP, and tokenizer design. The benchmark alone justifies a serious look. The resource is not yet released (the GitHub link is a promise), which matters for any version of record.\n\nRecommendation: send it to review, but the tokenization section needs to be rewritten with weaker claims and ideally some controlled comparison—e.g., matching entities on frequency and polysemy before comparing token counts, or an intervention like vocabulary rebalancing. With that, it's a solid paper.","headline":"Solid benchmark and a credible polysemy finding, but the tokenization claim is over-sold and should be read as correlational.","tokens_in":24046,"tokens_out":3142,"would_cite":true,"duration_ms":27011,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that Western cultural bias in LMs in Arabic stems less from training-data imbalance alone and more from Arabic word polysemy, cross-script lexical overlap, and frequency-based tokenization.","keywords":["cultural bias","Arabic NLP","language models","named entity recognition","polysemy","tokenization","multilingual evaluation","pre-training data"],"falsifier":"A controlled comparison on one tested LM using its actual pre-training corpus: take matched sets of Arab entities that differ only in polysemy status, holding frequency bin and token count constant, and run extractive QA in Arabic; if the polysemous versus non-polysemous accuracy gap vanishes, the paper's causal chain breaks. Alternatively, re-tokenize the same Arabic text with a vocabulary that splits polysemous entity words into subwords; if performance does not improve, the tokenization claim is falsified.","tokens_in":23162,"feed_emoji":"🌍","tokens_out":5742,"duration_ms":50797,"temperature":0.7,"pith_summary":"The paper asks why language models that work in English show a stubborn preference for Western-culture entities when prompted in Arabic. It introduces CAMeL-2, a parallel Arabic–English benchmark of 58,086 cultural entities and 367 masked contexts, and finds that the performance gap between Arab and Western entities largely disappears in English but widens in Arabic. The origin, the authors argue, is not simply under-representation: Arabic entities that double as common polysemous words, that resemble high-frequency words in other Arabic-script languages, and that tokenizers merge into a single token are the ones LMs fail on. The failure gets worse as models add larger Arabic vocabularies, because frequency-based tokenization encodes polysemous entities the same way as their non-entity senses.","feed_headline":"Cultural bias in LMs pinned on Arabic polysemy and tokenization","feed_subtitle":"A new Arabic–English benchmark shows the Arab/Western performance gap shrinks to near zero in English but widens in Arabic.","key_machinery":"The load-bearing apparatus is CAMeL-2, a parallel Arabic–English benchmark of 58,086 entities across seven entity types with 367 naturally masked contexts, built on the earlier CAMeL benchmark and extended with Wikipedia category extraction and OpenStreetMap location data. Around it, the analysis instruments are log-spaced occurrence counts of each entity in the Arabic and English mC4 portions computed with Aho-Corasick; the Almaany dictionary to tag entity words as polysemous; part-of-speech entropy over sampled corpus sentences to quantify how often entity words appear as adjectives or verbs in Arabic versus nouns in English; and token counts with Arabic-vocabulary sizes to connect subword tokenization to the failure.","core_discovery":"The authors' central claim is that entity-related cultural bias in LMs is co-produced by pre-training data statistics and by structural properties of Arabic. Using CAMeL-2, they show that, when tested in Arabic, LMs (Llama-3.3, Qwen-2.5, Aya-23, AceGPTv2, JAIS, and several encoder models) consistently recognize Western entities better than Arab entities, with gaps up to 15% accuracy on extractive QA and 27 F1 points on NER; when the same entities and translated contexts are given in English, the gaps shrink to near zero. The paper attributes this to four interacting causes: Arab entities appearing at very high frequencies in Arabic pre-training data (approximated by mC4) are often Arabic words with multiple senses; the degree of polysemy tracks the regional linguistic substrate behind location names; lexical overlap with other Arabic-script languages worsens performance; and frequency-based subword tokenization, especially into single tokens for polysemous words, is the encoding mechanism that makes the confusion concrete—an effect that amplifies with larger Arabic vocabularies.","pith_inferences":["The same mechanism likely applies to other languages whose writing systems reuse the Arabic script or whose place and food names coincide with common words, such as Hebrew and Persian; extending this analysis to those languages would test whether polysemy-driven bias is a general phenomenon rather than Arabic-specific.","The vocabulary-size result suggests a trade-off for Arabic LLM development: efficient single-token encoding of frequent words competes directly with disambiguation of entities, and a tokenizer that marks named-entity boundaries or preserves context might keep both.","Because the paper's frequency proxy is mC4, its quantitative slopes, such as the drop above 1M occurrences, may shift if measured against each model's true corpus; the qualitative ordering—polysemous, cross-script-overlapping, one-token entities fail—is the more robust claim.","CAMeL-2's parallel context set could be reused for tasks beyond NER and QA, such as generation, translation, or retrieval probing, to see whether the entities that fail extraction also produce stereotyped or Western-leaning continuations."],"forward_implications":["Testing an LM in English hides the cultural bias: the same Arab entities and translated contexts show near-zero gaps in English, so cross-lingual benchmarking is necessary to expose entity-related bias.","Very frequent and very rare Arab entities are the failure zones; medium-frequency entities in the roughly 1k–100k occurrence range are handled best in both languages.","Location names in countries whose names come from non-Arabic substrate languages, such as Comoros, fare better than names that are ordinary Arabic words, so LM performance on Arab locations is predictable from the percentage of polysemous names.","High-frequency overlap with Farsi, Urdu, Kurdish, Tajik, and Pashto harms extraction of Arab locations and food entities but less so personal names.","Larger Arabic vocabularies, intended to improve Arabic support, can make entity recognition worse for one-token polysemous entities unless tokenization changes."],"supporting_citations":[{"why":"Provides the original CAMeL benchmark, the Cultural Bias Score, and the Arab-vs-Western entity distinction that CAMeL-2 extends and tests bilingually.","marker":"Naous et al., 2024"},{"why":"Source of the mC4 corpus whose Arabic and English portions stand in for LM pre-training data in the frequency analyses.","marker":"Xue et al., 2021"},{"why":"Farasa POS tagger supplies the part-of-speech tags used to quantify how often entity words appear as adjectives or verbs in Arabic versus nouns in English.","marker":"Darwish and Mubarak, 2016"},{"why":"ANERCorp provides the Arabic NER training data for fine-tuning BERT-type models on names, authors, and locations.","marker":"Benajiba et al., 2007"},{"why":"CoNLL-2003 supplies the English NER training data for the cross-lingual NER comparison.","marker":"Sang and De Meulder, 2003"},{"why":"BOND distant supervision is used to create NER fine-tuning samples for entity types that lack manually annotated corpora.","marker":"Liang et al., 2020"},{"why":"Subword regularization is cited as the tokenization background for the frequency-based merging that turns polysemous entities into single tokens.","marker":"Kudo, 2018"},{"why":"Supplies the claim that subword tokenization is suboptimal for pre-training, supporting the tokenization mechanism in Section 4.4.","marker":"Bostrom and Durrett, 2020"}],"fun_headline_variants":["Arabic polysemy and tokenization drive cultural bias in language models","LMs favor Western entities in Arabic; English tests shrink gap to near zero","CAMeL-2 reveals pre-training frequency and polysemy cause Arab cultural bias","Why LMs are less culturally biased in English: Arab entities are polysemous"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes that the mC4 corpus's Arabic entity frequencies stand in for the actual, unpublished pre-training data of the tested LMs; if their real training corpora distribute entities differently, the frequency-based explanation and the lexical-overlap results may not reflect the true cause.","fun_headline_variants_meta":{"raw":{"variants":["Arabic polysemy and tokenization drive cultural bias in language models","LMs favor Western entities in Arabic; English tests shrink gap to near zero","CAMeL-2 reveals pre-training frequency and polysemy cause Arab cultural bias","Why LMs are less culturally biased in English: Arab entities are polysemous"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2794,"prompt_tokens":986,"completion_tokens":1808,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":1726}},"tokens_in":602,"tokens_out":1808,"duration_ms":13123,"temperature":1.0,"reasoning_tokens":1726,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:26:40.886505+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled comparison on one tested LM using its actual pre-training corpus: take matched sets of Arab entities that differ only in polysemy status, holding frequency bin and token count constant, and run extractive QA in Arabic; if the polysemous versus non-polysemous accuracy gap vanishes, the paper's causal chain breaks. Alternatively, re-tokenize the same Arabic text with a vocabulary that splits polysemous entity words into subwords; if performance does not improve, the tokenization claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the original CAMeL benchmark, the Cultural Bias Score, and the Arab-vs-Western entity distinction that CAMeL-2 extends and tests bilingually."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the mC4 corpus whose Arabic and English portions stand in for LM pre-training data in the frequency analyses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Farasa POS tagger supplies the part-of-speech tags used to quantify how often entity words appear as adjectives or verbs in Arabic versus nouns in English."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ANERCorp provides the Arabic NER training data for fine-tuning BERT-type models on names, authors, and locations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CoNLL-2003 supplies the English NER training data for the cross-lingual NER comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BOND distant supervision is used to create NER fine-tuning samples for entity types that lack manually annotated corpora."}],"review_version":1}