pith. machine review for the scientific record. sign in

arxiv: 2604.21751 · v1 · submitted 2026-04-23 · 💻 cs.CL · cs.AI· cs.CY

Recognition: unknown

Why are all LLMs Obsessed with Japanese Culture? On the Hidden Cultural and Regional Biases of LLMs

Carla Perez-Almendros, Joseba Fernandez de Landa, Jose Camacho-Collados

Authors on Pith no claims yet

Pith reviewed 2026-05-09 22:12 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.CY
keywords cultural biasLLMsregional preferencesJapanCROQ datasetsupervised fine-tuningprompt languagecultural competence
0
0 comments X

The pith

LLMs show a clear preference for Japanese culture when answering open cultural questions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models consistently favor references to Japan when responding to neutral questions about culture, food, history, and daily life, even though their training data is dominated by Western sources. The authors built the CROQ taxonomy and question set to measure these regional leanings across many countries and prompt languages. Experiments reveal that answers become more varied and less tied to the prompt language when the input is English or another high-resource language. The same tests locate the first strong signs of the Japan tilt after supervised fine-tuning rather than during the initial pre-training phase.

Core claim

The paper claims that LLMs exhibit a pronounced tendency to highlight Japan in answers to Culture-Related Open Questions, contrary to earlier findings of Western or Anglocentric bias. Prompting in high-resource languages produces more diverse outputs and reduces the pull toward countries where the prompt language is official. The cultural preference first becomes detectable after supervised fine-tuning and is not visible in the pre-trained base models.

What carries the argument

The CROQ taxonomy and dataset, a collection of open-ended cultural questions organized by country and topic that serves as a probe for measuring which nations LLMs spontaneously emphasize in their responses.

If this is right

  • Models will over-represent Japanese cultural elements in responses to everyday cultural queries even when the prompt does not mention Japan.
  • Using English or other high-resource prompt languages reduces the tendency to favor countries linked to the prompt language itself.
  • Cultural skews of this kind are introduced primarily during supervised fine-tuning, so mitigation efforts should target that stage.
  • Balanced cultural coverage requires deliberate adjustments after the pre-training phase rather than relying on the raw data distribution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Widespread use of these models could quietly shape public perceptions of which cultures are central to global topics.
  • Similar hidden preferences may exist for other cultures when tested with different question formats or domains.
  • Developers could test whether targeted fine-tuning on balanced cultural data removes the Japan tilt without harming other capabilities.

Load-bearing premise

The CROQ questions measure authentic model preferences rather than artifacts created by question wording, language choice, or patterns already present in the training data.

What would settle it

Running the identical CROQ questions on the same models but with rephrased items or additional low-resource languages and finding no consistent Japan preference would falsify the main result.

Figures

Figures reproduced from arXiv: 2604.21751 by Carla Perez-Almendros, Joseba Fernandez de Landa, Jose Camacho-Collados.

Figure 1
Figure 1. Figure 1: Framework for measuring cultural biases us [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Correlation between CommonCrawl data share and output diversity (mean of models) across 24 languages (log-scale). non-responses) and lower diversity scores. This pattern suggests a link between training data vol￾ume and cultural framing: for languages with less comprehensive corpora, models default to safer, more self-referential, or less diverse outputs (more detail in Appendix D.1). Pretraining data and … view at source ↗
Figure 3
Figure 3. Figure 3: Number of outputs from Base and Instruct models of Llama-3.1, Gemma-2, Mistral and Qwen2.5 variants [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Number of outputs mentioning top-referenced [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Taxonomy coverage for Related Work datasets. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of the selected 23 languages (En [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Chosen System prompt, Middle (3), given to the Judge model You are a geographical context classifier. Identify the nation, country, or state(s) explicitly mentioned in the text. Rules: 1. Only list countries/states that appear in the text. Do not infer. 2. If multiple countries/states are mentioned, list up to 5. 3. Only list nations, countries, or states; skip regions or cities unless needed for clarity. … view at source ↗
Figure 8
Figure 8. Figure 8: System prompt given to the Judge model (top: [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Outputs (without own references) from gpt-4o-mini, gemini-2.5-flash, claude-3.5-haiku, llama-4-maverick, command-r-08-2024, magistral-small-2506, qwen3-next-80b-a3b-instruct and deepseek-v3.2-exp variants over the full dataset of 24 languages. Countries: Japan (JP), USA (US), India (IN), China (CN), France (FR), Italy (IT), UK (GB), Germany (DE), Mexico (MX), Brazil (BR), Russia (RU), Egypt (EG). Data from… view at source ↗
Figure 11
Figure 11. Figure 11: Outputs from the Base and Instruct vari [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt given to the Base models in order to [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
read the original abstract

LLMs have been showing limitations when it comes to cultural coverage and competence, and in some cases show regional biases such as amplifying Western and Anglocentric viewpoints. While there have been works analysing the cultural capabilities of LLMs, there has not been specific work on highlighting LLM regional preferences when it comes to cultural-related questions. In this work, we propose a new dataset based on a comprehensive taxonomy of Culture-Related Open Questions (CROQ). The results show that, contrary to previous cultural bias work, LLMs show a clear tendency towards countries such as Japan. Moveover, our results show that when prompting in languages such as English or other high-resource ones, LLMs tend to provide more diverse outputs and show less inclinations towards answering questions highlighting countries for which the input language is an official language. Finally, we also investigate at which point of LLM training this cultural bias emerges, with our results suggesting that the first clear signs appear after supervised fine-tuning, and not during pre-training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 3 minor

Summary. The paper introduces the CROQ taxonomy and associated dataset of culture-related open questions to probe regional biases in LLMs. It reports that, contrary to prior work on Western/Anglocentric biases, models exhibit a clear preference for Japan when answering these questions. It further claims that prompts in high-resource languages (e.g., English) yield more diverse outputs and reduced inclination toward countries where the prompt language is official, and that the Japan-centric bias first appears after supervised fine-tuning rather than during pre-training.

Significance. If the central empirical claims hold after methodological validation, the work would be significant for documenting an under-reported cultural bias in LLMs and for tracing its emergence to the SFT stage. The new CROQ probe and the training-stage analysis constitute concrete contributions that could guide bias-mitigation research, provided the dataset is shown to be a neutral elicitation instrument rather than an artifact of question phrasing or topic selection.

major comments (3)
  1. [CROQ construction / methods] Dataset construction (CROQ taxonomy and question set): No evidence is supplied that the questions were balanced across geographic regions, pre-tested for phrasing neutrality, or checked against common training-data patterns that would produce a Japan preference even under a uniform prior. This is load-bearing for the headline claim because the observed distribution could arise from topic salience (pop culture, aesthetics, technology) rather than genuine model preference.
  2. [Results / experimental setup] Quantification of 'tendency' and statistical controls: The abstract and results sections state clear directional findings but supply no dataset size, exact bias metric (e.g., country mention frequency, normalized rank), confidence intervals, or controls for prompt language and model scale. Without these, it is impossible to judge whether the Japan preference is robust or sensitive to post-hoc choices.
  3. [Training-stage investigation] Training-stage analysis: The claim that 'first clear signs appear after supervised fine-tuning' requires explicit description of the base models, checkpoints, and operational definition of 'clear signs.' This analysis is central to the paper's contribution on bias emergence and cannot be evaluated without those details.
minor comments (3)
  1. [Abstract] Abstract contains a typo: 'Moveover' should be 'Moreover'.
  2. [Abstract] The phrasing 'show less inclinations' is grammatically awkward; consider 'exhibit less inclination'.
  3. [Related work] Ensure the related-work section explicitly contrasts the new Japan-centric finding with the specific prior cultural-bias papers cited in the abstract.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive feedback. We address each major comment below and will revise the manuscript to incorporate the requested details and clarifications.

read point-by-point responses

  1. Referee: Dataset construction (CROQ taxonomy and question set): No evidence is supplied that the questions were balanced across geographic regions, pre-tested for phrasing neutrality, or checked against common training-data patterns that would produce a Japan preference even under a uniform prior. This is load-bearing for the headline claim because the observed distribution could arise from topic salience (pop culture, aesthetics, technology) rather than genuine model preference.

    Authors: We agree that the current manuscript lacks explicit documentation of these aspects of CROQ construction. In the revision we will expand the methods section with: a table showing the number of questions per geographic region to demonstrate balance; a description of pilot testing for phrasing neutrality; and an analysis comparing the Japan preference against topic salience in training data (e.g., by contrasting with other high-salience cultures). We will also add a limitations paragraph discussing whether the bias could partly reflect pop-culture salience rather than model preference alone. revision: yes

  2. Referee: Quantification of 'tendency' and statistical controls: The abstract and results sections state clear directional findings but supply no dataset size, exact bias metric (e.g., country mention frequency, normalized rank), confidence intervals, or controls for prompt language and model scale. Without these, it is impossible to judge whether the Japan preference is robust or sensitive to post-hoc choices.

    Authors: We acknowledge that the current version does not report these quantitative details. The revised manuscript will state the exact dataset size (number of questions and models evaluated), define the primary bias metric (country-mention frequency normalized by response length and rank), include 95% confidence intervals and statistical tests, and report results stratified by prompt language and model scale. These additions will allow readers to assess robustness directly. revision: yes

  3. Referee: Training-stage analysis: The claim that 'first clear signs appear after supervised fine-tuning' requires explicit description of the base models, checkpoints, and operational definition of 'clear signs.' This analysis is central to the paper's contribution on bias emergence and cannot be evaluated without those details.

    Authors: We will revise the training-stage section to specify the exact base models and checkpoints examined, the supervised fine-tuning stages analyzed, and an operational definition of 'clear signs' (e.g., when the normalized Japan preference first exceeds a pre-defined threshold with statistical support). We will also add a figure tracing bias metrics across pre-training and post-SFT checkpoints. revision: yes

Circularity Check

0 steps flagged

No significant circularity in empirical analysis

full rationale

The paper introduces a new taxonomy and dataset (CROQ) of culture-related open questions, then reports direct experimental results from querying LLMs at different training stages. No mathematical derivations, parameter fits, or self-referential definitions are used to obtain the central claims about regional biases or the emergence point after supervised fine-tuning. All load-bearing steps consist of new data collection and observation rather than reduction to prior fitted quantities or self-citation chains. The analysis is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the new CROQ dataset being a valid, unbiased instrument for detecting cultural preferences; this depends on an untested domain assumption about the taxonomy's completeness and neutrality.

axioms (1)
  • domain assumption The CROQ taxonomy comprehensively covers culture-related questions without introducing its own regional or linguistic biases.
    Invoked when interpreting model outputs as evidence of LLM bias rather than dataset artifacts.

pith-pipeline@v0.9.0 · 5490 in / 1285 out tokens · 63547 ms · 2026-05-09T22:12:33.060880+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    The Llama 3 Herd of Models

    CaLMQA: Exploring culturally specific long- form question answering across 23 languages. In Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (V olume 1: Long Papers), pages 11772–11817, Vienna, Austria. Association for Computational Linguistics. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan,...

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  2. [2]

    Gemma 2: Improving Open Language Models at a Practical Size

    Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evalua- tion. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V ol- ume 1: Long Papers), pages 18761–18799, Vienna, Austria. Association for Computational Linguistics. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe...

    work page internal anchor Pith review arXiv 2024

  3. [3]

    Values & Ethics: Core beliefs about right/wrong, community, and individual roles

    Beliefs, Values, and Identity Religion & Spirituality: Beliefs and practices about the divine, morality, and afterlife. Values & Ethics: Core beliefs about right/wrong, community, and individual roles. Symbolism: Visual or material symbols that carry cultural meaning. Mythology & Folklore: Narratives that explain origins, values, or natural phenomena. Gen...

  4. [4]

    Community Life: How people relate to neighbors and groups

    Social Structure and Daily Life Family & Social Roles: Structure of family and communal relationships. Community Life: How people relate to neighbors and groups. Manners & Etiquette: Everyday norms for respectful behavior and interaction. Work Culture: Attitudes and habits related to labor, time, and productivity. Conflict Resolution: How societies handle...

  5. [5]

    Education Systems: Ways knowledge is transmitted and valued

    Knowledge, Communication, and Education Language & Communication: Spoken, written, and non-verbal systems of communication. Education Systems: Ways knowledge is transmitted and valued. Technology & Tools: Use of tools and innovation in daily and cultural life. Knowledge Transmission: How wisdom and skills are preserved across generations. Science & Philos...

  6. [6]

    Cultural Expression and the Arts Music: Rhythmic and melodic expression of culture Dance: Movement as performance, ritual, or celebration Art: Visual cultural expression (painting, sculpture, etc.) Literature & Storytelling: Written and oral traditions conveying culture and values Theater & Performance: Live dramatic cultural expression Film & Cinema: Sto...

  7. [7]

    Food, Drink, and Leisure Food & Cuisine: Traditional dishes, cooking styles, and dietary customs Traditional Beverages: Culturally significant drinks Recreation & Leisure: Activities for enjoyment and relaxation Sports & Games: Physical activities and competitions enjoyed culturally Life-Cycle Rituals: Ceremonies marking stages of life Festivals & Celebra...

  8. [8]

    Geographic Aspects Climate & Environment: How weather and landscape shape cultural practices, housing, and food Topography & Land Use: How people interact with physical geography like mountains, rivers, or plains Rural vs Urban Culture: Differences in lifestyle, work, and values between city and countryside Regional Identity: Sub-national cultural traits ...

  9. [9]

    Political Aspects Government Systems: Political structures influencing law, rights, and daily life Law & Legal Traditions: Cultural expectations of justice, rights, and punishment Civic Values & Participation: How people engage with politics, voting, and national identity Freedom of Expression: Cultural and political limits on speech or art Colonial Histo...

  10. [10]

    Health & Wellness Health Practices: Traditional, spiritual, or modern approaches to healing and care Public Health: Collective strategies for health and safety Wellbeing & Lifestyle: Balancing mental, physical, and social health Birth & Reproductive Health: Fertility, childbirth, and parenting norms and practices

  11. [11]

    Media & Entertainment Local Media: Print, radio, and TV shaping cultural narratives Digital Media: Online platforms influencing communication, identity, and activism Entertainment Industries: Mass production of cultural content News & Information: Flow of journalism and narratives in society Popular Culture: Icons, trends, and shared cultural references G...

  12. [12]

    History Historical Events: Major turning points shaping collective identity Historical Figures: Leaders, thinkers, and cultural icons Cultural Memory: How societies remember, teach, and reinterpret their past Colonialism & Resistance: Struggles of domination and liberation National Narratives: Shared stories of origin, destiny, and identity Heritage & Pre...

  13. [13]

    datasets address only a limited subset of the gen- eral topics defined in our taxonomy

    Economy & Industry Key Industries: Dominant sectors shaping work and identity Trade & Exchange: Movement of goods, ideas, and culture Labor & Employment: Work systems, professions, and class structures Wealth & Inequality: How resources and opportunities are distributed Economic Growth & Development: Paths to modernization and sustainability Globalization...

    2024

  14. [14]

    Do not infer nations, countries or states based on general knowledge, stereotypes, or assumptions beyond what is stated in the text

  15. [15]

    Not more than 5

    If multiple nations, countries or states are mentioned, list all of them. Not more than 5. Keep your answers concise and limited to the nation, country or state names. Translate the final answers into English. Figure 7: Chosen System prompt,Middle(3), given to the Judge model You are a geographical context classifier. Identify the nation, country, or stat...

  16. [16]

    Do not infer

    Only list countries/states that appear in the text. Do not infer

  17. [18]

    Answer with country names only, in English, separated by commas

    Only list nations, countries, or states; skip regions or cities unless needed for clarity. Answer with country names only, in English, separated by commas. You are a geographical context classifier. Identify the nation, country, or state(s) the text refers to. Rules:

  18. [19]

    If no country is mentioned, infer the most likely one from context

  19. [20]

    If multiple countries/states are mentioned, list up to 5

  20. [21]

    Non-Answered,

    Only list nations, countries, or states; skip regions or cities unless needed for clarity. Answer with country names only, in English, separated by commas. Figure 8: System prompt given to the Judge model (top:Explicit (1); bottom:Infer (2)) Models Language English (en) Chinese (zh) Spanish (es) Arabic (ar) French (fr) Russian (ru) Japanese (ja) Table 8: ...

    2000