REVIEW 5 cited by
Towards Measuring and Modeling "Culture" in LLMs: A Survey
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present a survey of more than 90 recent papers that aim to study cultural representation and inclusion in large language models (LLMs). We observe that none of the studies explicitly define "culture, which is a complex, multifaceted concept; instead, they probe the models on some specially designed datasets which represent certain aspects of "culture". We call these aspects the proxies of culture, and organize them across two dimensions of demographic and semantic proxies. We also categorize the probing methods employed. Our analysis indicates that only certain aspects of ``culture,'' such as values and objectives, have been studied, leaving several other interesting and important facets, especially the multitude of semantic domains (Thompson et al., 2020) and aboutness (Hershcovich et al., 2022), unexplored. Two other crucial gaps are the lack of robustness of probing techniques and situated studies on the impact of cultural mis- and under-representation in LLM-based applications.
Forward citations
Cited by 5 Pith papers
-
A Dual-Layered Evaluation of Geopolitical and Cultural Bias in LLMs
A multilingual two-phase evaluation shows LLMs lean on query language for factual questions and on training-country perspective for territorial and historical disputes.
-
Obscured but Not Erased: Evaluating Nationality Bias in LLMs via Name-Based Bias Benchmarks
A name-substituted variant of the BBQ benchmark shows that LLMs retain nationality stereotypes even when explicit labels are removed, with smaller models showing more bias and lower accuracy.
-
Against 'softmaxing' culture
A position paper arguing that AI evaluations should shift from defining culture to understanding when culture becomes relationally valid.
-
The Human Labour of Data Work: Capturing Cultural Diversity through World Wide Dishes
A design retrospective of World Wide Dishes identifies three dimensions of community ambassador labor, trust building, accessibility, and cultural contextualization, as essential to participatory dataset creation.
-
A Framework for LLM-powered Design Assistants
LLMs are organized into a three-modality framework for design assistance, but no evidence is provided that the framework works.
Discussion (0). Continue with ORCID to comment.