Pith. sign in

REVIEW 4 major objections 5 minor 43 references

PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PerCul shows the best LLM answers 81.7% of Persian cultural stories; humans score 93%.

desk verdict PerCul is a genuinely useful Persian cultural benchmark, but the headline human-model gap and the cultural-sensitivity claim rest on a human baseline drawn from the same small annotator pool and a task design that may reward clue-following as much as cultural knowledge. read the letter →

arxiv 2502.07459 v1 pith:JUWQBN7N submitted 2025-02-11 cs.CL cs.AIcs.CY

Erfan Moosavi Monazzah , Vahid Rahimzadeh , Yadollah Yaghoobzadeh , Azadeh Shakery , Mohammad Taher Pilehvar This is my paper · ORCID
classification cs.CLcs.AIcs.CY
keywords PersiancultureculturalevaluationlargelanguagemodelsbenchmarkdatasetstorycomprehensioncompetencemultilingualNLPtranslationresistance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PerCul is a new benchmark that tests whether large language models can recognize Persian cultural concepts when those concepts are implied, not named, in short everyday stories. The dataset contains 592 multiple-choice questions built by native Persian speakers: humans chose seed topics and wrote metadata clues, while LLMs drafted storylines and distractor options that humans then edited and filtered. The paper's central finding is that current models fall measurably short of human cultural reading: the best closed model answers 81.7% correctly against a 93% human baseline, and the best open-weight model reaches only 71.7%. Translating the stories into English makes the best models worse by 6.6 to 14.5 points, which the authors read as evidence that PerCul measures Persian-specific cultural knowledge rather than translation-solvable facts. If the benchmark is valid, it gives the field a concrete way to track progress toward non-Western cultural competence in LLMs.

What carries the argument

The load-bearing machinery is the PerCul item itself: a short story written so that a Persian cultural object or practice is implied but never named, paired with four options and a question asking which concept the story evokes. Three components make it work. First, human-selected seed topics and metadata facets under each of the 11 cultural-iceberg-derived categories supply factual grounding and indirect clues, so stories are authentic rather than hallucinated. Second, human editing removes direct references to the seed topic, enforcing the 'implied, not named' constraint. Third, six distractor-generation rules—partial correctness, misinterpretation, unrelated fact, plausible unsupported, noun confusion, and overgeneralization—produce options that separate surface lexical overlap from genuine inference. The translation experiment is the mechanism that validates the design: when a story is translated into English, culturally specific vocabulary flattens into generic terms and accuracy falls by 6.6 to 14.5 points, showing that solving PerCul requires Persian-specific cultural knowledge rather than English paraphrase.

What would settle it

A control experiment would rewrite PerCul stories with all surface cultural cues removed while preserving the underlying human action (for example, describing the physical motions of burning seeds and circling a person without naming espand, samovar, or the evil eye). If a top model's accuracy stays near 81.7% on such cue-free paraphrases, the benchmark rewards contextual synthesis; if accuracy collapses toward chance, the current scores are inflated by lexical clue-following. A second check gives the English translations to non-Persian readers: if they approach the model scores, translation alone carries the answer.

Watch

Extended reading notes

Core claim

The central claim is that PerCul is a valid, translation-resistant measure of LLM sensitivity to Persian culture, organized by the cultural-iceberg triad (technical, formal, and informal levels) extended with Iconic Figures and Objects into 11 categories: Institutions, Music, Dress, Objects, Visible Behavior, Art, Iconic Figures, Appropriacy, Rituals, Architecture, and Foods. Each question presents a short narrative that indirectly reflects a seed topic—burning espand, cardamom in tea, Bandari coastal music, Khoresht Mast—with four answer choices generated under six heuristic rules and filtered through three human selection stages. The evaluation of the open-weight and closed-source models shows that within a model family, accuracy rises with parameter count, but across families size is not decisive; the best closed model (Claude 3.5 Sonnet at 0.817) trails the 0.930 layperson baseline, and the best open-weight model (Llama 3.1 405B at 0.717) trails by more. The Persian-specialized models perform worse than their base Llama counterparts, and translating the dataset into English reduces accuracy by 6.6 to 14.5 points. The authors conclude that LLMs rely on surface-level story details rather than synthesizing cultural context, and that PerCul exposes a real, quantifiable gap in Persian cultural competence.

Load-bearing premise

The benchmark assumes that picking the right implied cultural concept from a multiple-choice story is a faithful proxy for cultural sensitivity; if models can score high by matching words and story details to option phrasing, the gap to humans would measure puzzle-solving rather than cultural competence.

Editorial extensions

If this is right

  • Any model claiming Persian cultural competence can now be benchmarked against a human baseline of 93%, and the best current models fall short by at least 11.3 points.
  • Machine translation is not a viable shortcut for cultural evaluation: translating PerCul to English lowers the strongest models by 6.6 to 14.5 points, so multilingual models must acquire Persian-specific cultural knowledge directly.
  • The gap between closed and open-weight models (81.7% vs. 71.7% best) means open models need more than scale; architecture and training data quality matter as much as parameter count.
  • Persian-specialized fine-tunes currently underperform their base Llama models, indicating that existing Persian instruction data may be too small or noisy to transfer cultural knowledge.
  • Error analysis with the six distractor rules shows that partial-correctness options are the most effective decoys, pointing to surface-level matching as the main failure mode of current LLMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is a generation-based version of PerCul that asks the model to produce or explain the implied cultural element rather than choosing among options; such a test would likely show an even larger model-human gap because it removes the clues embedded in the distractor set.
  • The construction pipeline is transferable to other underrepresented cultures, but the translation-degradation result warns that translating an existing cultural benchmark into a new language is not a substitute for building the benchmark natively with human speakers.
  • A testable consequence of the Persian fine-tuning result is that adding culturally rich, human-written Persian narratives to instruction-tuning data should improve PerCul scores; if it does not, the bottleneck is not data volume but the evaluation task itself.
  • Because PerCul covers only the technical and formal levels of the cultural-iceberg triad, the informal level (unconscious norms such as face-saving, indirect refusal, and modesty rituals) remains untested; a simulation-based probe where models act out social scenarios could complement PerCul.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PerCul, a multiple-choice story-based benchmark for evaluating LLM knowledge of Persian culture. The construction pipeline starts from Hall's Triad of Culture, uses native Persian annotators to produce seed topics, facets, and metadata, and then uses GPT-4o and Claude Sonnet 3.5 to draft stories and distractors, followed by extensive human editing and selection. The final dataset contains 592 questions across 11 cultural categories. The authors evaluate a wide range of closed-source, open-weight, and Persian fine-tuned models, reporting a best closed-source accuracy of 81.7% (Claude 3.5 Sonnet), a best open-weight accuracy of 71.7% (Llama-3.1-405B), and a human baseline of 93.0%. They also report a performance drop when the dataset is translated to English and an analysis of which distractor rules most often mislead models.

Significance. If the construct-validity concerns are addressed, PerCul would be a valuable and timely resource: it is, to my knowledge, the first story-based cultural benchmark specifically for Persian, it is publicly released, and it is built with a human-in-the-loop process rather than purely automatic generation. The paper also makes a useful empirical contribution by benchmarking a broad set of models and by documenting the effect of translation. The translation experiment and the distractor-rule analysis are thoughtful and go beyond a simple accuracy table. The strengths of the work are the public dataset, the detailed construction protocol, and the explicit native-annotator involvement; the main risks are whether the task measures cultural sensitivity rather than clue-following, and whether the reported human baseline is adequately grounded.

major comments (4)
  1. [§3.3, §4.3] The construction protocol in §3.3 explicitly requires metadata facets to “provide sufficient clues and factual information, allowing inference of the seed topic from an indirectly reflecting narrative.” Consequently, each story is deliberately built as a clue set, and the evaluation task asks models to identify the implied seed topic. The reported model–human gap may therefore measure lexical association and attribute matching rather than the “sensitivity toward Persian culture” claimed in the abstract. The error analysis in §4.3 (e.g., the Bandari music example) shows models selecting partial surface-level options, but it does not establish that a human who knows the clues but not Persian culture would fail. I ask the authors to provide a control or validation experiment (for example, testing items with distinctive clues removed, or evaluating a non-Persian-culture human control group), or to reframe the central claim as measuring knowledge of Persian cultural concepts rather than cultural sensitivity.
  2. [Table 3, §3.6, Appendix B.5] The human baseline of 93.0% is load-bearing for the headline 11.3% and 21.3% gaps, but the paper does not report the number of baseline participants, their recruitment, or any measure of variance. Appendix A describes only eight annotators, mostly university students, who were involved in dataset construction, and it is unclear whether the same individuals provided the baseline. If the baseline comes from the same small, educated, and dataset-familiar pool, the 93% figure may reflect familiarity with the construction process and its editorial decisions rather than lay cultural competence. Please report the full human-baseline protocol, including n, demographics, recruitment, per-participant accuracy, and, if possible, a separate evaluation by participants not involved in dataset creation.
  3. [§4.2] The claim that translation does not help models is weakened by the fact that the stories were designed to encode Persian-specific facets as clues. The manual categorization of P−E (19% nuance loss, 27% translation error, 54% correctly translated but answered incorrectly) is based on a single model (Claude 3.5 Sonnet) with no reported inter-annotator agreement. Moreover, the E−P examples given — “Tombak (a type of Persian drum)” and “Abgoosht (Persian lamb stew)” — show that the English translation added explanatory glosses that act as extra clues; the same mechanism may explain part of the P−E drop when Persian-specific lexical items are generalized (the sugar-bowl example). The authors should control for clue availability, for example by translating only the story text without explanatory glosses, or should temper the conclusion that the drop reflects loss of cultural meaning rather than loss of surface clues.
  4. [§4, Figure 6] The conclusion that Persian fine-tuned models “exhibited lower performance compared to their corresponding base models” is based on only two models. PersianMind v1.0 refuses most questions and is not directly comparable to a base model in a meaningful way; Dorna-Llama3-8B-Instruct (0.440) is statistically indistinguishable from Llama-3.1-8B-Instruct (0.444) given the reported results and the absence of significance testing. The statement in the conclusion that Persian-specific LLMs “fall short and even degrade in performance” is therefore not supported by the evidence presented. I recommend either adding more Persian fine-tuned models or rephrasing the claim to state that the two Persian fine-tuned models tested did not improve over their base models.
minor comments (5)
  1. [Table 1] The column header “Edition (%)” appears to mean “Edit (%)” (the percentage of tokens changed during human editing); please rename it and clarify the caption.
  2. [§4.3] The first sentence of Section 4.3 is garbled: “The effectiveness, distribution of heuristic rules within each category, of distractor options created by different heuristic rules in misleading models over different cultural categories.” Please rewrite it into a complete, grammatical sentence.
  3. [§3.3] There is a typo in Section 3.3: “ground the generated storylines in next steps in factual infromation” should be “factual information.”
  4. [Abstract] The abstract says “a 11.3% gap”; this should be “an 11.3% gap.”
  5. [Throughout] The name is spelled inconsistently as both “PerCul” (title) and “PERCUL” (abstract and body); please standardize.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: PerCul's model results are empirical measurements on a human-curated dataset, not derivations from fitted parameters or self-citational premises.

full rationale

PerCul is an empirical benchmark paper: the central result is a set of accuracy measurements of LLMs on 592 human-curated multiple-choice story questions against a human baseline. There is no derivation chain in which a predicted quantity is algebraically or definitionally identical to an input. The dataset construction pipeline uses human-annotated seed topics and metadata as prompts for LLM story generation, and those same metadata are later hidden inside the stories; however, this is a design property of the benchmark, not a circular reduction of the reported model scores. The model accuracies are measured on a fixed answer key, with the human baseline obtained separately through annotation guidelines. The use of GPT-4o and Claude to draft stories and distractors while also being evaluated is a contamination risk, not circularity, especially because the paper documents human editing, rewriting, and selection at multiple stages. The concern that metadata 'provide sufficient clues' makes the task solvable by lexical association is a construct-validity critique, not evidence that the evaluation outcome is forced by construction. Translation-degradation and baseline-composition issues similarly bear on measurement validity and fairness, not on circularity. The only notable self-citation is BLEnD (Myung et al., 2024), which includes a co-author of PerCul, but it is used in related work as a comparative limitation and is not load-bearing for any claim in the paper. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is presented under new coordinates. The paper's claims stand or fall on data quality and task validity, which are not circularity defects.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. Its central claims rest on four domain assumptions about cultural taxonomy, task validity, annotator representativeness, and label stability.

assumptions (4)
  • domain assumption Hall's Triad of Culture is an adequate taxonomy for operationalizing Persian culture for LLM evaluation.
    Section 3.1 selects technical and formal levels and adds two category extensions; if the taxonomy omits important Persian cultural dimensions, benchmark coverage is incomplete.
  • domain assumption Story-based multiple-choice questions with implied concepts measure cultural competence rather than general reading or clue-following ability.
    Section 3.3 instructs annotators to include clues that allow inference of the seed topic, so the task conflates cultural knowledge with story reasoning.
  • domain assumption A pool of eight native Persian university students is representative enough to define ground truth for Persian culture.
    Appendix A lists demographic skew with ages 18-28 and urban universities; the Limitations section acknowledges this bias.
  • domain assumption Human-annotated correct answers are unambiguous and stable across raters.
    Inter-agreement is reported for seed topic selection, but not for the final correctness of the 592 stories or for the human baseline measurement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian." pith.science (2026). https://pith.science/paper/JUWQBN7N

@misc{pith2026250207459,
  author       = {Pith},
  title        = {Pith review of: PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JUWQBN7N}},
  note         = {Machine review of arXiv:2502.07459}
}
read the original abstract

Large language models predominantly reflect Western cultures, largely due to the dominance of English-centric training data. This imbalance presents a significant challenge, as LLMs are increasingly used across diverse contexts without adequate evaluation of their cultural competence in non-English languages, including Persian. To address this gap, we introduce PerCul, a carefully constructed dataset designed to assess the sensitivity of LLMs toward Persian culture. PerCul features story-based, multiple-choice questions that capture culturally nuanced scenarios. Unlike existing benchmarks, PerCul is curated with input from native Persian annotators to ensure authenticity and to prevent the use of translation as a shortcut. We evaluate several state-of-the-art multilingual and Persian-specific LLMs, establishing a foundation for future research in cross-cultural NLP evaluation. Our experiments demonstrate a 11.3% gap between best closed source model and layperson baseline while the gap increases to 21.3% by using the best open-weight model. You can access the dataset from here: https://huggingface.co/datasets/teias-ai/percul

Figures

Figures reproduced from arXiv: 2502.07459 by the authors.

Figure 1
Figure 1. A translated example of PERCUL, implying a cultural concept in Visible Behavior category. the extent to which LLMs encapsulate knowledge about particular cultures. Despite the numerous benchmarks that evaluate various aspects of LLMs (Chang et al., 2024), a sig￾nificant gap remains in assessing their knowledge of culture across many non-English languages such as Persian. Although some efforts have been made to creat… view at source ↗
Figure 2
Figure 2. PERCUL was generated through a stepwise process: (1) identifying cultural categories using Hall’s Triad of Culture, (2) native annotators generating facets, topics, and metadata, (3) generating storylines with capable LLMs, (4) rigorous human correction and selection of stories, (5) creating comprehension options with heuristic rules, and (6) dataset compilation. descriptive facets, seed topics, and metadata for the… view at source ↗
Figure 3
Figure 3. Hall’s triad of cultural levels, (*) indicates our [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The accuracy on PERCUL for different fami￾lies of models against their number of parameters. According to our findings, the best-performing model on PERCUL is Anthropic Sonnet 3.5, with an accuracy of 81.7% (which is still 11.3% lower than the human baseline of 93%). N…
Figure 5
Figure 5. Figure 5: The performance of different models across the 11 cultural categories in P [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The impact of fine-tuning on Persian-specific [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: The degradation of best-performing models [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The impact and distribution of heuristic rules [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 10
Figure 10. Figure 10: Prompt used for Story Generation [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Prompt used for distractor generation. D User Interfaces We have developed various user interfaces for vari￾ous steps of our work. Some of them are shown in Figures 12, 13 and 14. E Full Benchmark Results This appendix includes important visual and tabu￾lar data that …
Figure 12
Figure 12. Figure 12: User interface for human baseline bench [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 14
Figure 14. Figure 14: User interface for story selection and refine [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 13
Figure 13. Figure 13: User interface for distractor selection. Par [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 15
Figure 15. Figure 15: Radar Chart for best performing models of [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: Full heatmap on the effectiveness of distractor options created by different heuristic rules in misleading [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 13 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Negin Abadani, Jamshid Mozafari, Afsaneh Fatemi, Mohammd Ali Nematbakhsh, and Arefeh Kazemi. 2021. https://doi.org/10.1109/ICWR51868.2021.9443126 Parsquad: Machine translated squad dataset for persian question answering . In 2021 7th International Conference on Web Research (ICWR), pages 163--168

    work page arXiv 2021

  4. [4]

    Amirhossein Abaskohi, Sara Baruni, Mostafa Masoudi, Nesa Abbasi, Mohammad Hadi Babalou, Ali Edalat, Sepehr Kamahi, Samin Mahdizadeh Sani, Nikoo Naghavian, Danial Namazifard, Pouya Sadeghi, and Yadollah Yaghoobzadeh. 2024. https://aclanthology.org/2024.lrec-main.197 Benchmarking large language models for P ersian: A preliminary study focusing on C hat GPT ...

    work page 2024

  5. [5]

    Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. https://doi.org/10.18653/v1/2024.acl-long.671 Investigating cultural alignment of large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404--12422, Bangkok, Thailand. Association for Compu...

    doi:10.18653/v1/2024.acl-long.671 2024

  6. [6]

    Hossein Amirkhani, Mohammad AzariJafari, Soroush Faridan-Jahromi, Zeinab Kouhkan, Zohreh Pourjafari, and Azadeh Amirak. 2023. https://doi.org/10.1007/s00500-023-08959-3 Farstail: a persian natural language inference dataset . Soft Computing

    doi:10.1007/s00500-023-08959-3 2023

  7. [7]

    Anthropic. 2024. https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf Claude 3.5 sonnet model card addendum

    2024

  8. [8]

    Shaily Bhatt and Fernando Diaz. 2024. https://arxiv.org/abs/2406.11565 Extrinsic evaluation of cultural competence in large language models . Preprint, arXiv:2406.11565

    work page arXiv 2024

Show all 43 references
  1. [9]

    Yu, Qiang Yang, and Xing Xie

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. https://doi.org/10.1145/3641289 A survey on evaluation of large language models ....

    2024 doi

  2. [10]

    Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2024. https://arxiv.org/abs/2410.02677 Culturalbench: a robust, diverse and challenging benchmark on measu...

    2024 arXiv

  3. [11]

    Kasra Darvishi, Newsha Shahbodaghkhan, Zahra Abbasiantaeb, and Saeedeh Momtazi. 2023. https://doi.org/10.1016/j.csl.2023.101486 Pquad: A persian question answering dataset . Computer Speech and Language, 80:101486

    2023

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. 2024. https://arxiv.org/abs/2402.09369 Massively multi-cultural knowledge acquisition and lm benchmarking . Preprint, arXiv:2402.09369

    2024 arXiv

  6. [14]

    Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, and Mohammad Hossein Rohban. 2024. https://arxiv.org/abs/2404.06644 Khayyam challenge (persianmmlu): Is your llm truly wise to the persian...

    2024 arXiv

  7. [15]

    Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. https://doi.org/10.18653/v1/2024.na...

    2024 doi

  8. [16]

    David Katan and Mustapha Taibi. 2021. https://www.amazon.com/Translating-Cultures-David-Katan/dp/113834446X Translating Cultures: An Introduction for Translators, Interpreters and Mediators, Third Edition

    2021

  9. [17]

    Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabagdi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shah...

    2021

  10. [18]

    Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. 2024. https://aclanthology.org/2024.lrec-main.296 CLI c K : A benchmark dataset of cultural and linguistic intelligence in K orean . In Proceedings of the 2024 Joint International Conference on Comput...

    2024

  11. [19]

    Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024. https://arxiv.org/abs/2402.10946 Culturellm: Incorporating cultural differences into large language models . Preprint, arXiv:2402.10946

    2024 arXiv

  12. [20]

    Manrai, Ajay K

    Lalita A. Manrai, Ajay K. Manrai, Dana-Nicoleta Lascu, and Stefanie Friedeborn. 2019. https://api.semanticscholar.org/CorpusID:149838700 Determinants and effects of cultural context: A review, conceptual model, and propositions . Journal of Global Marketing, 32:67 -- 82

    2019

  13. [21]

    Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues

    Reem I. Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. 2023. https://arxiv.org/abs/2309.12342 Cultural alignment in large language models: An explanatory analysis based on hofstede's cultural dimensions . Preprint, arXiv:2309.12342

    2023 arXiv

  14. [22]

    Jun-Hee Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla P \'e rez-Almendros, Abinew Ali Ayele, V'ictor Guti'errez-Basulto, Yazm'in Ib'anez-Garc'ia, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Rzay...

    2024 arXiv

  15. [23]

    Tarek Naous, Michael Ryan, Alan Ritter, and Wei Xu. 2024. https://doi.org/10.18653/v1/2024.acl-long.862 Having beer after prayer? measuring cultural bias in large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu...

    2024 doi

  16. [24]

    Kimia Noorbakhsh, Modar Sulaiman, Mahdi Sharifi, Kallol Roy, and Pooyan Jamshidi. 2021. https://api.semanticscholar.org/CorpusID:238419670 Pretrained language models are symbolic mathematics solvers too! ArXiv, abs/2110.03501

    2021 arXiv

  17. [25]

    PartAI. 2024. P art A I / D orna- L lama3-8 B - I nstruct. https://huggingface.co/PartAI/Dorna-Llama3-8B-Instruct. [Accessed 13-10-2024]

    2024

  18. [26]

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.772 I n F o B ench: Evaluating instruction following ability in large language models . In Finding...

    2024 doi

  19. [27]

    Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. https://arxiv.org/abs/2404.12464 Normad: A benchmark for measuring the cultural adaptability of large language models . Preprint, arXiv:2404.12464

    2024 arXiv

  20. [28]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. https://arxiv.org/abs/2311.12022 Gpqa: A graduate-level google-proof q&a benchmark . Preprint, arXiv:2311.12022

    2023 arXiv

  21. [29]

    Pedram Rostami, Ali Salemi, and Mohammad Javad Dousti. 2024. https://arxiv.org/abs/2401.06466 Persianmind: A cross-lingual persian-english large language model . Preprint, arXiv:2401.06466

    2024 arXiv

  22. [30]

    Hamidreza Saffari, Mohammadamin Shafiei, and Francesco Pierri. 2024. https://arxiv.org/abs/2406.09123 Psn: Persian social norms dataset for cross-cultural ai . Preprint, arXiv:2406.09123

    2024 arXiv

  23. [31]

    Pushpdeep Singh, Mayur Patidar, and Lovekesh Vig. 2024. https://arxiv.org/abs/2406.14504 Translating across cultures: Llms for intralingual cultural adaptation . Preprint, arXiv:2406.14504

    2024 arXiv

  24. [32]

    Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. 2024. https://arxiv.org/abs/2310.16049 Musr: Testing the limits of chain-of-thought with multistep soft reasoning . Preprint, arXiv:2310.16049

    2024 arXiv

  25. [33]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. https://doi.org/10.18653/v1/2023.findings-acl.824 Challenging BIG -bench tasks and whether chain-of-thought c...

    2023 doi

  26. [34]

    Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. https://doi.org/10.1093/pnasnexus/pgae346 Cultural bias and cultural alignment of large language models . PNAS Nexus, 3(9):pgae346

    2024 doi

  27. [35]

    OpenAI Team. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  28. [36]

    Dennis Tedlock and Bruce Mannheim. 1995. The dialogic emergence of culture. University of Illinois Press

    1995

  29. [37]

    Michael Thier. 2013. https://api.semanticscholar.org/CorpusID:268309397 Cultural awareness logs: A method for increasing international-mindedness among high school and middle school students . English Journal

    2013

  30. [38]

    Yamshchikov

    Alexey Tikhonov, Igor Samenko, and Ivan P. Yamshchikov. 2021. https://doi.org/10.18653/v1/2021.eval4nlp-1.4 S tory DB : Broad multi-language narrative dataset . In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 32--39, Punta Cana, Dominican ...

    2021 doi

  31. [39]

    Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael Lyu. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.345 Not all countries celebrate thanksgiving: On the cultural dominance in large language models . In Proceedings of the 62nd ...

    2024 doi

  32. [40]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024 b . https://arxiv.org/abs/2406.01574 Mmlu-pro: A more robust ...

    2024 arXiv

  33. [41]

    Yuhang Wang, Yanxu Zhu, Chao Kong, Shuyu Wei, Xiaoyuan Yi, Xing Xie, and Jitao Sang. 2024 c . https://doi.org/10.18653/v1/2024.c3nlp-1.1 CDE val: A benchmark for measuring the cultural dimensions of large language models . In Proceedings of the 2nd Workshop on Cross-Cultural C...

    2024 doi

  34. [42]

    Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. https://api.semanticscholar.org/CorpusID:268063798 How do large language models handle multilingualism? ArXiv, abs/2402.18815

    2024 arXiv

  35. [43]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. https://arxiv.org/abs/2311.07911 Instruction-following evaluation for large language models . Preprint, arXiv:2311.07911

    2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.