REVIEW 13 cited by
Towards Measuring and Modeling "Culture" in LLMs: A Survey
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We present a survey of more than 90 recent papers that aim to study cultural representation and inclusion in large language models (LLMs). We observe that none of the studies explicitly define "culture, which is a complex, multifaceted concept; instead, they probe the models on some specially designed datasets which represent certain aspects of "culture". We call these aspects the proxies of culture, and organize them across two dimensions of demographic and semantic proxies. We also categorize the probing methods employed. Our analysis indicates that only certain aspects of ``culture,'' such as values and objectives, have been studied, leaving several other interesting and important facets, especially the multitude of semantic domains (Thompson et al., 2020) and aboutness (Hershcovich et al., 2022), unexplored. Two other crucial gaps are the lack of robustness of probing techniques and situated studies on the impact of cultural mis- and under-representation in LLM-based applications.
Forward citations
Cited by 13 Pith papers
-
Toward a Theory of Value in AI Alignment
A systematic annotation of 94 AI alignment papers shows the field largely equates human values with measurable preferences, rarely defines values, and is increasingly removing humans from alignment evaluation.
-
People Are Not Just Their Countries. Disentangling Social Determinants of LLM Value Alignment Across Europe
Across 10 LLMs and the European Social Survey, AI alignment favors wealthier, more educated, less religious, and more politically interested groups, with country of residence explaining as much variance as all sociode...
-
PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture
PalmX 2025 introduces a two-subtask MCQA benchmark for Arabic and Islamic cultural knowledge and shows that task-specific fine-tuning, especially LoRA, improves LLM accuracy.
-
A Dual-Layered Evaluation of Geopolitical and Cultural Bias in LLMs
A multilingual two-phase evaluation shows LLMs lean on query language for factual questions and on training-country perspective for territorial and historical disputes.
-
On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena
Arab cultural entities that double as everyday Arabic words are harder for language models to recognize, especially when tokenized as single tokens.
-
CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
CultureVerse is a 188-country, 19k-concept visual QA benchmark, and fine-tuning open VLMs on it improves cultural accuracy, but the main evaluation shares concepts between training and test sets.
-
All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages
ALM-bench is a 100-language, 19-domain cultural visual QA benchmark on which GPT-4o reaches 78.8% and the best open model, GLM-4V-9B, reaches 51.9%.
-
Obscured but Not Erased: Evaluating Nationality Bias in LLMs via Name-Based Bias Benchmarks
A name-substituted variant of the BBQ benchmark shows that LLMs retain nationality stereotypes even when explicit labels are removed, with smaller models showing more bias and lower accuracy.
-
Against 'softmaxing' culture
A position paper arguing that AI evaluations should shift from defining culture to understanding when culture becomes relationally valid.
-
Normative Evaluation of Large Language Models with Everyday Moral Dilemmas
Seven LLMs give different moral verdicts on AITA dilemmas, differ from Redditors, and only in an ensemble approximate human consensus.
-
The Human Labour of Data Work: Capturing Cultural Diversity through World Wide Dishes
A design retrospective of World Wide Dishes identifies three dimensions of community ambassador labor, trust building, accessibility, and cultural contextualization, as essential to participatory dataset creation.
-
Musical ethnocentrism in Large Language Models
LLMs like ChatGPT and Mixtral show a strong Western bias when asked to name top musical contributors and to rate the musical cultures of countries.
-
A Framework for LLM-powered Design Assistants
LLMs are organized into a three-modality framework for design assistance, but no evidence is provided that the framework works.
Discussion (0). Continue with ORCID to comment.