CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages

Bilal Elbouardi; Fajri Koto; Muhammad Dehan Al Kautsar; Salsabila Pranida

REVIEW 4 major objections 6 minor 12 references

CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages

T0 review · 4 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CultureTalk-ID presents the first human-curated dialogue benchmark for cultural commonsense across Indonesian and its local languages, and shows open-source models fall short, especially when generating local-language responses.

desk verdict A genuinely useful dialogue benchmark for Indonesian cultural commonsense that needs the data released and gold-label reliability quantified before the numbers can be trusted. read the letter →

arxiv 2607.21016 v1 pith:X3R6I2K6 submitted 2026-07-23 cs.CL

Muhammad Dehan Al Kautsar , Salsabila Pranida , Bilal Elbouardi , Fajri Koto This is my paper

classification cs.CL

keywords culturalcommonsensedialoguebenchmarkIndonesianlocallanguagesmultilingualevaluationmachinetranslationlanguagesteeringlow-resourceLLMreasoning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CultureTalk-ID claims that cultural commonsense is best measured inside conversation, not in isolated prompts. The paper builds the first dialogue-based benchmark for this in Indonesia, with 4,496 human-validated dialogues spanning Indonesian plus 10 local languages and 13 cultural topics. Each dialogue appears in both Indonesian and the local language, with the final utterance omitted and three candidate endings. Three tasks—multiple-choice cultural reasoning, bidirectional translation, and language steering—test whether models can understand, transfer, and generate culturally grounded language. The evaluation shows a clear gap: open-source models score far below proprietary ones, and the largest failures occur in local-language generation, where models often fall back to Indonesian.

What carries the argument

The load-bearing structure is the parallel dialogue pair. Every item exists in Indonesian and in one of ten local languages; the final utterance is omitted and three candidate continuations are offered, only one of which is culturally plausible. Because the same content is available in both languages, the benchmark supports three complementary tasks from one resource. The multi-stage human pipeline—two native-speaker annotators per region, partner cross-checking, and deliberately injected wrong entries—is what certifies the gold answers and translations.

What would settle it

Take a random sample of dialogues per region, remove the given answer labels, and ask a third independent native speaker to supply the most natural continuation; if the new speakers frequently disagree with the gold answer (or with each other), the benchmark's ground truth is ambiguous, which would falsify the assumption that the MCQ scores measure cultural commonsense.

Watch

Extended reading notes

Core claim

The paper's central claim is that a human-curated, dialogue-based benchmark can expose cultural-commonsense failures that single-turn benchmarks hide. CultureTalk-ID converts two existing Indonesian cultural MCQ datasets into multi-turn dialogues, translates every dialogue into the region's local language, and has native annotators validate content, correct answers, and translations through a staged pipeline with injected-error checks. Evaluating a range of proprietary, multilingual, and Southeast-Asia-focused models, the paper finds that proprietary models lead on all three tasks, that all models lose accuracy when the dialogue is in a local language rather than Indonesian, and that open-so

Load-bearing premise

The benchmark's validity rests on the human annotators' cultural judgments and translations being correct and consistent; if the gold answers are wrong or contested, every score on every task becomes uninterpretable.

Editorial extensions

If this is right

If the central claim holds, cultural-commonsense evaluation needs dialogue context rather than isolated prompts to reveal real model behavior.
Local-language proficiency is the binding constraint: every model group loses accuracy when the dialogue is in a local language instead of Indonesian.
Supervised fine-tuning on even a small training split improves translation fluency and register, suggesting targeted adaptation works for culturally grounded MT.
Language steering is a distinctive failure mode: many open-source models produce Indonesian even when a local variety is requested, revealing a generation gap that MCQ tests miss.
Providing province and language context helps open models most, indicating that explicit geographic grounding aids cultural reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same construction pipeline—seeding dialogues from existing cultural MCQ data and validating with native annotators—could be replicated for other multilingual countries, because the format, not the language, is what forces contextual reasoning.
The high MCQ scores of proprietary models could partly reflect residual answer-choice artefacts, even after the paper's style-consistency pass; an adversarial reordering of options would test this directly.
The benchmark's parallel structure could double as a training resource: a model fine-tuned on the local-language side may improve both translation and steering, since both tasks share the same surface forms.
Because local-language generation lags far more than local-language understanding, the next likely gain in Indonesia-focused NLP is expanding curated local-language pretraining data rather than simply increasing model size.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is one of the better Indonesian cultural NLP benchmarks I've seen recently, and the dialogue framing is genuinely new. The three-task design (MCQ, MT, language steering) is sensible, and the human pipeline—two fluent annotators per region, pilot tasks, injected errors, cross-checking—is real work. The finding that open models collapse on local-language generation is credible, and the qualitative Javanese example in Table 5 tells that story cleanly.

What would stop me from trusting the numbers as-is: first, the MCQ gold labels have no quantified reliability. No inter-annotator agreement, no independent audit, and the annotators who revised the options also supplied the answers. That's a self-confirming loop, and it's load-bearing: every MCQ accuracy number inherits the gold labels. This is fixable—report agreement on a sample, or get a third annotator to re-label a subset. Second, the dataset link is anonymized, so nobody can inspect the actual artifact. For a benchmark paper, that's a blocker in its current form. Third, the LLM-as-a-judge metric in language steering is demonstrably weak: Qwen3-8B gets a judge score of 3.11/5 while producing the target language only 1.5% of the time per Glot-LID. The paper itself says the judge gives high scores to Indonesian output in local-language tasks. So the judged 1-5 scores are inflated; the qualitative ordering still holds through Glot-LID, but the judge scores shouldn't be reported as a reliable measure.

The MT task is better: the paper runs a small human correlation for two languages and gets reasonably strong agreement, which is a mark of honesty. The pretraining comparison (Gemma vs Sahabat-AI) is a nice addition, and the finding that benefits transfer to provinces never seen in the training data is genuinely interesting.

Nothing here makes me doubt the central contribution. The benchmark fills a real gap and will be useful; the weaknesses are addressable, not conceptual. “Conditional” is the right verdict. If the data is released and the annotation reliability is quantified, I'd take it as a serious resource. It deserves a proper peer-review route, not a desk reject.

Referee Report

4 major / 6 minor

Summary. The paper introduces CultureTalk-ID, a dialogue-based benchmark for cultural commonsense in Indonesian and ten local Indonesian languages. Built by transforming IndoCulture and COPAL-ID through GPT-5 dialogue generation followed by a four-phase human pipeline, the benchmark contains 4,496 dialogues, each available in Indonesian and a local language, and supports three tasks: dialogue-based multiple-choice cultural commonsense reasoning, bidirectional machine translation, and language steering. The authors evaluate nine proprietary, multilingual, and SEA-centric models, reporting that open-source models underperform, particularly in generation tasks and local-language settings.

Significance. If the benchmark is valid, it fills a clear gap: existing Indonesian cultural commonsense benchmarks are single-turn and Indonesian-only, whereas CultureTalk-ID provides multi-turn, parallel Indonesian/local-language dialogues with human curation. The multi-stage pipeline includes useful quality controls—two annotators per region, injected attention checks, cross-checking, and annotator self-answering—and the MT evaluation includes a human-correlation analysis of the LLM-as-a-judge metric. The parallel structure enabling three complementary tasks is a genuine strength. However, the benchmark's validity rests on the reliability of the gold labels and on the fairness of the model comparisons; both need strengthening before the central claims are fully supported.

major comments (4)

[Sections 3.2–3.4] Gold-label reliability is not demonstrated. The pipeline uses two annotators per region with mutual cross-checking and injected errors, but no inter-annotator agreement (e.g., Cohen's kappa or percentage agreement) is reported, either for the MCQ answers or for translation quality. More importantly, Phase 4 (Section 3.4) has a single annotator per province revise all answer options to remove shortcut cues; there is no independent verification that these revisions preserve the uniqueness and cultural correctness of the gold answer. Since every MCQ score and model ranking depends on the gold labels being unambiguous and correct, this omission is load-bearing. I recommend reporting agreement statistics on a held-out double-annotated sample and adding an independent audit of the Phase-4 revisions (or having a second annotator review them).
[Section 4 and Appendix C] The MCQ evaluation protocol may be inconsistent across model categories. The text states that for open-source models a likelihood-based setup is used, but it does not specify how proprietary models are evaluated. Figure 5 shows a generation-style prompt ('Output only the OPTIONS [A/B/C]'), which suggests proprietary models may be evaluated via generation while open models are scored by likelihood. If so, the headline 'open-source models underperform' is confounded with evaluation method. Please clarify whether all models use the same decoding/scoring protocol; if different, run both likelihood-based and generation-based evaluation on all models and report both.
[Sections 2 and 3.1] The novelty claim needs to be stated more carefully. The paper calls CultureTalk-ID 'the first dialogue-based benchmark' for this setting, yet Section 2 describes SeaDialogues (Kautsar et al., 2025) as a culturally grounded multilingual dialogue benchmark for Southeast Asian languages. The differentiators—extensive human curation, Indonesian local languages, and three task formats—should be made explicit so the 'first' claim is precise. Additionally, because the dialogues are generated from IndoCulture/COPAL-ID instances, there is a risk that model performance reflects memorization of those source benchmarks. The paper should discuss this contamination risk and, ideally, test whether models trained on the source benchmarks obtain inflated scores.
[Section 5.2 and Table 3] The claim that supervised fine-tuning yields consistent improvements is central to the MT results, but the statistical significance statement is vague: 'We conduct statistical significance testing for all metrics with available standard deviations' is not backed by reported tests, confidence intervals, or p-values in the text. For the zero-shot vs. SFT comparisons (e.g., Gemma-2-9B: 20.54→22.60 BLEU; Sahabat-AI: 21.70→24.56), the differences are small relative to the reported standard deviations. Please report the test used and the resulting significance levels, or soften the claim.

minor comments (6)

[Abstract / Footnote 1] The phrase 'The dataset link is anonymized' should be replaced with a URL or a statement that the dataset will be released upon publication; this is normal for review, but the final version should not retain this sentence.
[Section 4 / Table 14] The model names are inconsistent: Section 4 and Table 2 list 'Gemini-2.5-flash,' while Table 14 and several appendix tables list 'Gemini-3-flash-preview.' Please reconcile the naming.
[Section 3.4] Phase 4 says 'one annotator from each province' revised the answer options, but Section 3.2 says two annotators per region. It would be helpful to state whether this was a random choice or a designated senior annotator, and whether the other annotator subsequently saw the revisions.
[Table 8] Wamesa is described as 'vulnerable' in the main text (citing Ritchie et al., 2024) but as 'Endangered' in Table 8 (citing Ethnologue). Please align these designations or specify the sources.
[Section 5.2 / Appendix G] The human evaluation correlation uses only 60 samples and two languages (Javanese, Minangkabau). The limitations section acknowledges this, but the abstract's general wording ('extensive human curation') may overstate the coverage of human judgment for the MT task; consider clarifying the scope of the human evaluation.
[Table 3 / Table 14] The footnote in Table 3 says standard deviations are shown for BLEU/BERTScore, but the table lists (±0.1) values only in the BLEU column and not BERTScore; please add the missing deviations or correct the footnote.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the benchmark is a transparent transformation of prior datasets and the evaluations are new measurements; self-citations are not load-bearing.

full rationale

CultureTalk-ID is a dataset-construction and evaluation paper, not a derivation with equations, so the main circularity patterns (self-definitional reductions, fitted predictions, imported uniqueness theorems) do not apply. The paper explicitly states it repurposes IndoCulture and COPAL-ID as pivot datasets (Section 3.1) and that the original answer options are adapted as candidate endings; this is transparent data reuse, not a claim that the new benchmark is independent of its source data. The gold labels are produced by two native annotators per region and then verified in a loop where the annotators also answer the MCQs and revise instances they miss (Section 3.3); this is a quality-control step, albeit one that could affect validity if annotators are unreliable. It is not a fitted parameter renamed as a prediction, because no model output is used to define the gold labels and no quantity is being predicted from a fit. The evaluation results on MCQ, MT, and language steering are measured on a held-out test split, with SFT performed only on the 448-dialogue training split, so the reported model rankings are not forced by construction. Self-citations (Kautsar et al. 2023, 2025; Koto et al. 2024) appear in related work or as dataset provenance; they are used for contrast and are not load-bearing evidence for the benchmark's validity or for any conclusion. The paper's own limitations about LLM-as-a-judge human correlation being restricted to Javanese and Minangkabau, and about incomplete province/topic coverage, are acknowledged gaps rather than circular steps. Overall, no prediction or first-principles result in the paper reduces to its inputs by construction; the strongest concerns are benchmark-validity issues, not circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted numerical parameters are introduced; the benchmark is a dataset, not a model. The central claim rests on domain assumptions: source benchmarks are correct, annotator judgments are reliable, dialogues have unique answers, and LLM-as-a-judge approximates humans. These are reasonable but mostly unverified beyond self-report.

assumptions (5)

domain assumption IndoCulture and COPAL-ID labels are correct cultural commonsense gold.
Section 3.1 repurposes their premises and answer options as dialogue endings; any errors in the source datasets propagate into CultureTalk-ID.
domain assumption GPT-5-generated dialogues, after human correction and translation, are culturally authentic and natural.
Section 3.2-3.4; naturalness is validated only by the two annotators per region, with no external audit.
domain assumption Each dialogue has exactly one culturally appropriate continuation among the three options.
Section 3.5 defines the MCQ task; ambiguity would make accuracy scores uninterpretable.
domain assumption Two annotators per region with >10 years residence provide reliable gold labels.
Section 3.2-3.3; no inter-annotator agreement statistics are reported, and 5% injected errors only test attentiveness, not judgment reliability.
domain assumption LLM-as-a-judge scores are a reliable proxy for human quality in MT and language steering.
Section 5.2/Appendix G validate on 60 samples in Javanese and Minangkabau only; Appendix H shows the judge gives high scores even when the model writes in the wrong language.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages." pith.science (2026). https://pith.science/paper/X3R6I2K6

@misc{pith2026260721016,
  author       = {Pith},
  title        = {Pith review of: CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X3R6I2K6}},
  note         = {Machine review of arXiv:2607.21016}
}

read the original abstract

Culture is lived through conversation, yet existing Indonesian cultural commonsense benchmarks evaluate LLMs on short and isolated prompts, stripping away the dialogic context in which cultural nuances actually surface. We introduce CultureTalk-ID, the first dialogue-based benchmark for cultural commonsense in Indonesian and its local languages, comprising 4,496 culturally grounded dialogues across 11 languages and 13 culturally salient topics, curated through a multi-stage human pipeline with native speakers to ensure authenticity. CultureTalk-ID introduces three complementary tasks, namely dialogue-based multiple-choice cultural commonsense reasoning, culturally faithful machine translation, and language steering, which jointly probe whether LLMs can understand, transfer, and generate culturally grounded language.

Figures

Figures reproduced from arXiv: 2607.21016 by the authors.

**Figure 1.** Overview of the CULTURETALK-ID benchmark. CULTURETALK-ID supports three tasks: (1) dialoguebased multiple-choice questions (MCQs) for cultural commonsense reasoning, (2) bidirectional machine translation between Indonesian and local languages, and (3) language steering. tive speakers to ensure cultural authenticity. Compared with recent benchmarks (Permadi et al., 2026; Aji and Cohn, 2025), CULTURETALK-ID incorpor… view at source ↗

**Figure 2.** The CULTURETALK-ID dataset creation pipeline. It consists of four phases: dialogue generation; correction of Indonesian dialogues, translation into local languages, and initial quality control (QC1); crosschecking and secondary quality control (QC2); and finally, ensuring consistency in dialogue style. 3.2 Phase 2: Correction of Indonesian Dialogues, Translation to Local, and QC1 The dialogues generated by GPT-5 we… view at source ↗

**Figure 3.** Prompt for the dialogue generation. B Detailed Dataset Statistics [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: An example of the dataset transformation process from an original IndoCulture instance into [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

**Figure 5.** Figure 5: Task 1 - Dialogue-based cultural common [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

**Figure 7.** Figure 7: Task 2 - Machine translation’s (Indonesian [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 6.** Figure 6: Task 2 - Machine translation’s unified system [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

**Figure 9.** Figure 9: Task 3 - Language steering’s user prompt. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 2 linked inside Pith

[1]

Output only the OPTIONS [A/B/C]
[2]

Fajri Koto, Timothy Baldwin, and Jey Han Lau

Seadialogues: A multilingual culturally grounded multi-turn dialogue dataset on southeast asian languages.Preprint, arXiv:2508.07069. Fajri Koto, Timothy Baldwin, and Jey Han Lau. 2022. Cloze evaluation for deeper understanding of com- monsense stories in Indonesian. InProceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 202...

arXiv 2022
[3]

Preserve meaning, tone, and level of formality
[4]

Do not change names, numbers, or entities
[5]

Qingyu Zhang, Xiaoyu Shen, Ernie Chang, Jidong Ge, and Pengke Chen

Qwen3 technical report.arXiv preprint arXiv:2505.09388. Qingyu Zhang, Xiaoyu Shen, Ernie Chang, Jidong Ge, and Pengke Chen. 2022. Mdia: A benchmark for multilingual dialogue generation in 46 languages. Preprint, arXiv:2208.13078. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: ...

arXiv 2022
[7]

Only the option label [A/B/C]

Do not add explanations, comments, or quotation marks. Only the option label [A/B/C]. You are tasked with selecting the most culturally appro- priate option based on the context provided below. Location: {province} Language: {language} Conversation: {dialogue} Give the option label only [A/B/C]. Options: {choices_str} Answer: Figure 5: Task 1 - Dialogue-b...
[8]

Output only the translated text in Latin (both In- donesian and other local languages)
[9]

Do not add explanations, comments, labels, or quotation marks

Show all 12 references

[12]

adequacy

If the input already matches the target variety, return it unchanged. Extra Rules: - Output ONLY the translation - No explanation - No reasoning (DO NOT EXPLAIN YOUR REASON- ING) - No extra text Figure 6: Task 2 - Machine translation’s unified system prompt. Minangkabau, and W...

2024
[2022]

InProceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates

Few-shot learning with multilingual generative language models. InProceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics. Chen Cecilia Liu, Iryna Gu...

2022
[2023]

InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 815–834, Dubrovnik, Croatia

NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages. InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 815–834, Dubrovnik, Croatia. Association for Com- putational Linguistics. Jinc...

2025
[2025]

InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32866–32895, Suzhou, China

CARE: Multilingual human preference learn- ing for cultural awareness. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32866–32895, Suzhou, China. Association for Computational Linguistics. Amir Hossein Kargaran, Ayyoob Imani, Fr...

2025

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Output only the OPTIONS [A/B/C]

[2] [2]

Fajri Koto, Timothy Baldwin, and Jey Han Lau

Seadialogues: A multilingual culturally grounded multi-turn dialogue dataset on southeast asian languages.Preprint, arXiv:2508.07069. Fajri Koto, Timothy Baldwin, and Jey Han Lau. 2022. Cloze evaluation for deeper understanding of com- monsense stories in Indonesian. InProceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 202...

arXiv 2022

[3] [3]

Preserve meaning, tone, and level of formality

[4] [4]

Do not change names, numbers, or entities

[5] [5]

Qingyu Zhang, Xiaoyu Shen, Ernie Chang, Jidong Ge, and Pengke Chen

Qwen3 technical report.arXiv preprint arXiv:2505.09388. Qingyu Zhang, Xiaoyu Shen, Ernie Chang, Jidong Ge, and Pengke Chen. 2022. Mdia: A benchmark for multilingual dialogue generation in 46 languages. Preprint, arXiv:2208.13078. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: ...

arXiv 2022

[6] [7]

Only the option label [A/B/C]

Do not add explanations, comments, or quotation marks. Only the option label [A/B/C]. You are tasked with selecting the most culturally appro- priate option based on the context provided below. Location: {province} Language: {language} Conversation: {dialogue} Give the option label only [A/B/C]. Options: {choices_str} Answer: Figure 5: Task 1 - Dialogue-b...

[7] [8]

Output only the translated text in Latin (both In- donesian and other local languages)

[8] [9]

Do not add explanations, comments, labels, or quotation marks