Pith. sign in

REVIEW 3 major objections 6 minor 43 references

CulFiT: A Fine-grained Cultural-aware LLM Training Paradigm via Multilingual Critique Data Synthesis

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CulFiT claims a synthetic critique-and-reward loop makes open-weight LLMs culturally aligned without costing reasoning ability.

desk verdict A genuinely useful cultural fine-tuning recipe with believable gains on external benchmarks, but the headline SOTA claim on GlobalCultureQA is undermined by a same-generator, same-judge evaluation loop. read the letter →

arxiv 2505.19484 v2 pith:3UCT6KDS submitted 2025-05-26 cs.CL

Ruixiang Feng , Shen Gao , Xiuying Chen , Lisi Chen , Shuo Shang This is my paper
classification cs.CL
keywords culturalbiasalignmentmultilingualdatasynthesiscritiquegenerationfine-grainedrewardmodelingknowledgeunitsdirectpreferenceoptimizationGlobalCultureQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CulFiT is a training recipe designed to make open-weight language models culturally knowledgeable instead of Western-centric. The pipeline synthesizes cultural question-answer pairs from existing cultural knowledge collections, generates critiques that compare the target model's answers against golden answers decomposed into atomic knowledge units, translates and back-translation-verifies this material into culturally relevant languages, then fine-tunes with supervised learning plus Direct Preference Optimization using a fine-grained cultural precision, recall, and F1 reward. On GlobalCultureQA, a new 1,104-question open-ended multilingual benchmark introduced here, the best CulFiT model reaches a cultural F1 of 72.94 and precision of 74.73, surpassing all open-source baselines and roughly matching a leading closed-source model. The paper also reports gains of up to 8.8 points on CANDLE500 and 6.6 on CulturalBench, larger gains in low-resource languages on BLEnD, and no loss, with small gains, on CSQA, Hellaswag, and MMLU-pro. The central claim is that decomposing cultural answers into checkable knowledge units and rewarding only unit-level matches teaches cultural alignment that transfers to general reasoning.

What carries the argument

The load-bearing mechanism is knowledge-unit decomposition with three-way meta-critique. A generator splits golden and model answers into atomic knowledge units, then judges each golden unit as semantically matched, unaddressed, or contradicted in the model answer; these judgments are summarized into a targeted critique that tells the model exactly which cultural facts it missed or got wrong. Around that loop sit multilingual synthesis with back-translation verification and a fine-grained reward: cultural precision scores how many model units match golden units, cultural recall scores how many golden units are covered, and the cultural F1 combines them as $S_{f1} = 2S_pS_r/(S_p+S_r)$. The paper uses that same $S_{f1}$ both to choose DPO preference pairs, keeping pairs with $S_{f1}<0.7$, and to score answers on the new GlobalCultureQA benchmark.

What would settle it

Have native speakers from the target cultures independently write golden answers for a sample of GlobalCultureQA questions, then blind-rate CulFiT, its base model, and a leading closed model against those human references; if the CulFiT margin over its base shrinks or disappears under human scoring, the reported F1 reflects agreement with the synthetic generator, not cultural correctness.

Watch

Extended reading notes

Core claim

The paper's central claim is that a fully synthetic, target-aware critique loop can inject cultural knowledge into open-weight models more effectively than feeding them descriptive cultural text alone. For each synthesized question, the generator produces a golden answer and the target model produces its own answer; both are decomposed into atomic knowledge units, and a meta-critique labels each golden unit as semantically covered, unaddressed, or contradicted. That critique is included in supervised fine-tuning, and a fine-grained reward, cultural precision minus unmatched units and cultural recall over golden units, combined into a cultural F1, selects preference pairs for DPO. The authors state this achieves state-of-the-art open-source cultural alignment, competitive with closed-source systems, while preserving or improving general reasoning, and they interpret the improvements as evidence that the method corrects the target model's specific cultural blind spots rather than merely memorizing new text.

Load-bearing premise

The claimed cultural gains assume that the same generator that writes the golden answers and critiques also provides a fair judge of whether a trained model's answers contain the right cultural knowledge units, rather than rewarding answers that merely mimic the generator's phrasing.

Editorial extensions

If this is right

  • Open-weight models can be brought to roughly GPT-4o-level open-ended cultural QA, so cultural alignment does not inherently require a larger or closed model.
  • Multilingual training data is not a side effect: it is what lifts low-resource-language performance on BLEnD, with the largest gains in Sundanese, Hausa, and Amharic.
  • Including target-aware critique in SFT is the single most important component; dropping it costs more than dropping multilingual data in the ablations.
  • Fine-tuning on cultural critique data improves, rather than degrades, scores on CSQA, Hellaswag, and MMLU-pro, so the authors claim no catastrophic forgetting and some transfer to reasoning.
  • The reward threshold matters: DPO pairs selected below a cultural F1 of 0.7 help most, and pairs at 0.9 hurt, consistent with preferring pairs with a clear correctness gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the unit-decomposition loop is sound, the same critique-then-reward recipe could be applied to other alignment targets, such as etiquette, taboo, religious norms, or political values, wherever source texts can be broken into checkable atomic claims.
  • The open-ended benchmark numbers should be read with the loop in mind: the golden answers, the critiques, the DPO selector, and the evaluation judge all come from the same generator, so a human-written golden reference set on GlobalCultureQA would be the decisive check of how much of the GPT-4o-level margin is genuine cultural knowledge rather than stylistic conformity.
  • The multiple-choice gains on CANDLE500 and CulturalBench are independent of that loop and are the cleanest evidence that cultural knowledge itself improved; an extension of this work could therefore track whether open-ended gains track the same independent benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CulFiT, a cultural-awareness training paradigm with three components: target-aware critique data generation over existing cultural knowledge sources, multilingual expansion with back-translation verification, and fine-grained reward modeling based on decomposing answers into verifiable knowledge units. The authors introduce GlobalCultureQA, a multilingual open-ended QA benchmark, and train Llama3.1-8B and Qwen2.5-7B with SFT and DPO using the proposed cultural F1 reward. They report improvements over the base models on CANDLE500, CulturalBench, BLEnD, and GlobalCultureQA, reduced Hofstede distance, and preserved general reasoning on CSQA, Hellaswag, and MMLU-Pro.

Significance. The paper's external multiple-choice results are a genuine, if modest, contribution: CANDLE500 and CulturalBench gains of up to 8.8% and 6.6% over the base models, plus broad BLEnD improvements, are believable and do not depend on the paper's new benchmark. The release of code and the use of existing benchmarks are strengths. If the GlobalCultureQA result were independently validated, the fine-grained reward and multilingual critique pipeline would be a useful addition to cultural alignment research. However, the self-referential evaluation loop makes the headline state-of-the-art claim unverified; the paper currently demonstrates a promising training method with independent evidence on existing benchmarks, not an unbiased SOTA result on a new benchmark.

major comments (3)
  1. [§3.2, §3.4.1, §4.3, §5.3; Eq. (12)] The headline result on GlobalCultureQA is load-bearing for the claimed state of the art, but the evaluation loop is not independent of the training loop. The Sf1 metric (Eq. 12) is used both to filter DPO preference pairs (Sf1 < 0.7, §3.4.1 and §5.3) and to score the benchmark (§4.3), and the golden answers in GlobalCultureQA are generated by the same model G that generates the training golden answers and critiques (§3.2). Because the judge in §7.9 accepts near-paraphrases as 'Yes' matches, a model that learns to reproduce G's knowledge-unit phrasing can inflate its F1 without any real gain in independent cultural knowledge. The external multiple-choice benchmarks in Table 2 do support a weaker claim, but no table reports GlobalCultureQA under an independent judge or with human-validated unit matching. I recommend re-scoring GlobalCultureQA with a judge from a different model family, or with human unit-level annotation, and separating that result from the main claim.
  2. [§7.6, Table 8] The human study in §7.6 is too limited to resolve the circularity concern: it evaluates 100 questions with whole-answer preference, does not validate the Sf1 scores, and the annotators are only described as proficient in Chinese and English even though GlobalCultureQA spans 23 languages. In particular, it does not confirm the 0.13 F1 margin over GPT-4o in Table 1. A small unit-level human agreement study on a stratified sample of GlobalCultureQA would be needed to establish that the fine-grained F1 measures cultural correctness rather than surface similarity.
  3. [Table 1] The reported margins on GlobalCultureQA are small (CulFiT (Llama3.1) F1 72.94 vs. GPT-4o 72.81, and CulFiT (Qwen2.5) F1 68.81 vs. Aya 68.66), but the paper gives no confidence intervals, significance tests, or per-region breakdowns for this benchmark. Given the shared metric and generator, these margins should be treated as provisional until evaluated independently.
minor comments (6)
  1. [Abstract] The phrase 'a specific cultural biases' should be corrected to 'specific cultural biases'.
  2. [Eq. (4)] Equation (4) uses 'LLM(P, ...)' but should identify the model as G, so that the notation is consistent with Eqs. (1)-(3).
  3. [Table 3] The color shading is defined relative to the base model, but the text says CulFiT outperforms open-source models such as Aya and Mistral in 12 of 16 countries; please clarify which comparison the 12-of-16 count refers to, since CulFiT (Llama3.1) regresses on South Korea and CulFiT (Qwen2.5) regresses on Algeria relative to its base.
  4. [Figure 3] The threshold analysis reports only precision on CulturalBench when varying Sf1; reporting F1 and results on at least one additional benchmark would make the threshold analysis more convincing.
  5. [§7.4] The Hofstede experiment is said to cover 9 cultures, but the list contains a typo ('Chinese Germany') and no per-culture distances are shown; please provide the per-culture results or a supplementary table.
  6. [§3.3, §7.1] The multilingual data synthesis reports total counts for English and other languages but not the distribution across the 24 target languages; this distribution is important for evaluating the multilingual robustness claim.

Circularity Check

1 steps flagged · score 6.0 of 10

GlobalCultureQA's SOTA margin is substantially an in-distribution score: the same Sf1 reward (Eq. 12) selects DPO training pairs and scores the benchmark, and both the training and benchmark golden answers come from generator G. Independent benchmarks support only a weaker claim.

  1. fitted input called prediction [§3.2, §3.4.1, §4.3, §4.1 (Eqs. 8-12)]
    "Golden Answer ( Ag): Produced by data generation LLM G through knowledge-aware synthesis. ... It is notable that we select our DPO training data using cultural F1 metric with Sf1 < 0.7 ... For GlobalCultureQA benchmark, we evaluate cultural precision score Sp, cultural recall score Sr and then calculate cultural f1 score Sf1 described in § 3.4.1."

    Eq. (12) is used both to select DPO training pairs (Sf1 < 0.7) and to score GlobalCultureQA. The golden answers that define Sf1 are generated by the same model G in §3.2, and GlobalCultureQA is introduced as 'based on the proposed data construction method' (§1). So the headline Table 1 numbers partly measure how closely CulFiT reproduces G's knowledge-unit phrasing, not an externally validated cultural-correctness score. The model is trained on pairs chosen for low Sf1 and then evaluated with Sf1, so its SOTA margin over GPT-4o on GlobalCultureQA is not independent evidence. External benchmarks (CANDLE500, CulturalBench, BLEnD) are non-circular but support a weaker claim than the abstract's unqualified SOTA statement.

full rationale

Most of CulFiT's pipeline is not circular: CANDLE500, CulturalBench, and BLEnD are fixed external benchmarks with human or independent gold answers, and the gains there (Tables 2-3) plus the Hofstede-distance analysis (Figure 4) provide genuinely independent evidence that the method helps. The circularity is concentrated in the newly introduced GlobalCultureQA loop. Eq. (12) is simultaneously the DPO data-selection reward and the benchmark metric; the golden answers behind both come from generator G, and GlobalCultureQA reuses the same data-construction method. Thus the central new-benchmark result (72.94 vs. 72.81 cultural F1 over GPT-4o) largely reflects optimizing the model against its own generator rather than measuring cultural correctness against an external standard. The 100-question human study (§7.6) compares whole answers and does not validate Sf1 or the 0.13-F1 margin. The score of 6 reflects this partial reduction-by-construction, while acknowledging that the external benchmarks carry independent, non-circular content.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central claim rests on a chain of trust in the generator G: G writes the cultural facts (golden answers), G decomposes answers into units, G writes critiques, G judges matches for the Sf1 metric, and G translates and back-translates. The only external anchors are the source datasets (CANDLE, CultureAtlas, CultureBank), the fixed multiple-choice benchmarks, and the 100-sample human study. One threshold (Sf1 < 0.7) is tuned against CulturalBench. No physical entities are involved; the invented pieces are evaluation scaffolding (Ac, As, Al units).

free parameters (2)
  • DPO data selection threshold Sf1 = 0.7
    Preference pairs with cultural F1 below 0.7 are chosen for DPO; Figure 3 sweeps thresholds 0.5 to 1.0 on CulturalBench and 0.7 is selected as best, so reported results depend on this tuned value.
  • knowledge-unit decomposition granularity (prompt to G) = not specified (LLM-chosen)
    The number and granularity of atomic units into which answers are split is determined by prompt instructions to G (§3.2, §7.7), not by a fixed rule; this choice controls both critique quality and every Sf1 score.
assumptions (5)
  • domain assumption Cultural correctness is reducible to atomic knowledge units, and LLM-based semantic matching of units is a valid measure of it.
    Invoked in §3.2 and §3.4.1: golden and target answers are decomposed into units and matched by the generation model G (Eqs. 8-11); no human validation of the matching itself is provided (the human study in §7.6 uses 100 samples and does not directly validate Sf1).
  • domain assumption The generator G produces factually reliable cultural golden answers and critiques.
    G writes the golden answers (§3.2), the critiques, and the evaluation judge's Yes/No matches (§7.9); if G's cultural knowledge is biased, the errors propagate into the benchmark and the training signal. G is not named or ablated.
  • domain assumption Back-translation plus semantic alignment preserves cultural meaning across languages.
    §3.3 assumes that translating to a target language and back, then checking semantic alignment, guarantees that cultural content survives; no human check of translated critique quality is reported.
  • domain assumption Source datasets CANDLE, CultureAtlas, and CultureBank are accurate cultural knowledge.
    All training questions and golden answers are synthesized from these three external datasets (§3.2); errors in the sources propagate directly into training and evaluation.
  • domain assumption Hofstede VSM13 formulas remain valid when items are converted from Likert scale to multiple choice.
    §7.4 converts the 24 VSM13 survey questions to multiple choice and applies Eqs. 13-18 with the original constants; the conversion is not validated, yet Figure 4's large-distance claims (e.g., GPT-4o at 321.9) rest on it.
invented entities (1)
  • Contextual evaluation units (cultural group Ac, topic As, language Al)
    purpose: Appended to each decomposed answer (Eq. 7) to make evaluation targets verifiable and reduce scoring variance.
    These are new constructs introduced by the paper's evaluation framework; their validity is internal to the pipeline and no external benchmark or human study confirms that requiring models to output them measures cultural awareness better than direct assessment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CulFiT: A Fine-grained Cultural-aware LLM Training Paradigm via Multilingual Critique Data Synthesis." pith.science (2026). https://pith.science/paper/3UCT6KDS

@misc{pith2026250519484,
  author       = {Pith},
  title        = {Pith review of: CulFiT: A Fine-grained Cultural-aware LLM Training Paradigm via Multilingual Critique Data Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3UCT6KDS}},
  note         = {Machine review of arXiv:2505.19484}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks, yet they often exhibit a specific cultural biases, neglecting the values and linguistic diversity of low-resource regions. This cultural bias not only undermines universal equality, but also risks reinforcing stereotypes and perpetuating discrimination. To address this, we propose CulFiT, a novel culturally-aware training paradigm that leverages multilingual data and fine-grained reward modeling to enhance cultural sensitivity and inclusivity. Our approach synthesizes diverse cultural-related questions, constructs critique data in culturally relevant languages, and employs fine-grained rewards to decompose cultural texts into verifiable knowledge units for interpretable evaluation. We also introduce GlobalCultureQA, a multilingual open-ended question-answering dataset designed to evaluate culturally-aware responses in a global context. Extensive experiments on three existing benchmarks and our GlobalCultureQA demonstrate that CulFiT achieves state-of-the-art open-source model performance in cultural alignment and general reasoning.

Figures

Figures reproduced from arXiv: 2505.19484 by the authors.

Figure 1
Figure 1. An example of language inconsistency. When [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of our proposed CulFiT. as a triple: Ti = {A i g , Aj t , Cr}, (3) where Cr denotes the meta-critique described above. Finally, we summarize all meta critiques (T1, T2, · · · , Tk) for corresponding answer into a comprehensive critique C, and it will be used to serve as target-aware cultural error reminder in the supervised fine-tuning stage. C = LLM(P,(T1, T2, · · · , Tk)), (4) where P denotes the crit… view at source ↗
Figure 3
Figure 3. Results of the precision on different reward [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison in terms of Hofstede distance. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 7
Figure 7. Figure 7: Distribution of overall topics. We classify the [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 6
Figure 6. Figure 6: The continent level distribution on our Glob [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 8
Figure 8. Figure 8: A case study on the results of GlobalCultureQA. We use [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 9 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157

    arXiv 2024

  4. [4]

    Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    work page 2024

  5. [5]

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954

    arXiv 2024

  6. [6]

    Charles S Carver and Michael F Scheier. 1982. Control theory: A useful conceptual framework for personality--social, clinical, and health psychology. Psychological bulletin, 92(1):111

    work page 1982

  7. [7]

    Yu Ying Chiu, Liwei Jiang, Maria Antoniak, Chan Young Park, Shuyue Stella Li, Mehar Bhatia, Sahithya Ravi, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2024 a . Culturalteaming: Ai-assisted interactive red-teaming for challenging llms'(lack of) multicultural knowledge. arXiv preprint arXiv:2404.06664

    arXiv 2024

  8. [8]

    Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, et al. 2024 b . Culturalbench: a robust, diverse and challenging benchmark on measuring the (lack of) cultural knowledge of llms. arXiv preprint arXiv:2410.02677

    arXiv 2024

Show all 43 references
  1. [9]

    Rochelle Choenni and Ekaterina Shutova. 2024. Self-alignment: Improving alignment of cultural values in llms via in-context learning. arXiv preprint arXiv:2408.16482

    2024 arXiv

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [11]

    Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. 2024. Massively multi-cultural knowledge acquisition & lm benchmarking. arXiv preprint arXiv:2402.09369

    2024 arXiv

  4. [12]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738

    2023 arXiv

  5. [13]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  6. [14]

    Geert Hofstede and Michael Minkov. 2013. Vsm 2013. Values survey module

    2013

  7. [15]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798

    2023 arXiv

  8. [16]

    Jing Huang and Diyi Yang. 2023. Culturally aware natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7591--7609

    2023

  9. [17]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  10. [18]

    Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When can llms actually correct their own mistakes? a critical survey of self-correction of llms. Transactions of the Association for Computational Linguistics, 12:1417--1440

    2024

  11. [19]

    Nayeon Lee, Chani Jung, Junho Myung, Jiho Jin, Jose Camacho-Collados, Juho Kim, and Alice Oh. 2024. Exploring cross-cultural differences in english hate speech annotations: From dataset construction to analysis. In Proceedings of the 2024 Conference of the North American Chapt...

    2024

  12. [20]

    Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024 a . Culturellm: Incorporating cultural differences into large language models. arXiv preprint arXiv:2402.10946

    2024 arXiv

  13. [21]

    Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang. 2024 b . Culturepark: Boosting cross-cultural understanding in large language models. arXiv preprint arXiv:2405.15145

    2024 arXiv

  14. [22]

    Huihan Li, Liwei Jiang, Jena D Hwang, Hyunwoo Kim, Sebastin Santy, Taylor Sorensen, Bill Yuchen Lin, Nouha Dziri, Xiang Ren, and Yejin Choi. 2024 c . Culture-gen: Revealing global cultural perception in language models through natural language prompting. arXiv preprint arXiv:2...

    2024 arXiv

  15. [23]

    Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computation...

    2024

  16. [24]

    Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, et al. 2024. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. arXiv preprint arXiv:2406.09948

    2024 arXiv

  17. [25]

    Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    2024

  18. [26]

    Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. 2023. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, pages 1907--1917

    2023

  19. [27]

    Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. 2024. Survey of cultural awareness in language models: Text and beyond. arXiv preprint arXiv:2411.00860

    2024 arXiv

  20. [28]

    Rifki Afina Putri, Faiz Ghifari Haznitrama, Dea Adhista, and Alice Oh. 2024. Can llm generate culturally relevant commonsense qa data? case study in indonesian and sundanese. arXiv preprint arXiv:2402.17302

    2024 arXiv

  21. [29]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  22. [30]

    Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. Normad: A benchmark for measuring the cultural adaptability of large language models. arXiv preprint arXiv:2404.12464

    2024 arXiv

  23. [31]

    Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Soujanya Poria, and Rada Mihalcea. 2024. Understanding the capabilities and limitations of large language models for cultural commonsense. In Proceedings of the 2024 Conference of the North American Chapter of the Asso...

    2024

  24. [32]

    Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Raya Horesh, Rog \'e rio Abreu de Paula, Diyi Yang, et al. 2024. Culturebank: An online community-driven knowledge base towards culturally aware language technologies. arXiv preprint arXiv:2404.15238

    2024 arXiv

  25. [33]

    KaShun Shum, Shizhe Diao, and Tong Zhang. 2023. Automatic prompt augmentation and selection with chain-of-thought from labeled data. arXiv preprint arXiv:2302.12822

    2023 arXiv

  26. [34]

    Jiaxing Sun, Weiquan Huang, Jiang Wu, Chenya Gu, Wei Li, Songyang Zhang, Hang Yan, and Conghui He. 2024. Benchmarking chinese commonsense reasoning of llms: From chinese-specifics to reasoning-memorization correlations. arXiv preprint arXiv:2403.14112

    2024 arXiv

  27. [35]

    World Values Survey. 2022. World values survey. https://www.worldvaluessurvey.org/wvs.jsp

    2022

  28. [36]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937

    2018 arXiv

  29. [37]

    Yan Tao, Olga Viberg, Ryan S Baker, and Ren \'e F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS nexus, 3(9):pgae346

    2024

  30. [38]

    Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, AiTi Aw, and Nancy F Chen. 2023 a . Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning. arXiv preprint arXiv:2309.04766

    2023 arXiv

  31. [39]

    Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael R Lyu. 2023 b . Not all countries celebrate thanksgiving: On the cultural dominance in large language models. arXiv preprint arXiv:2310.12481

    2023 arXiv

  32. [40]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574

    2024 arXiv

  33. [41]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  34. [42]

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2401.10020

    2024 arXiv

  35. [43]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.