The reviewed record of science sign in
Pith

arxiv: 2607.06196 · v1 · pith:Y42GXFG2 · submitted 2026-07-07 · cs.CL · cs.CY

Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability

Alicia Parrish , Rajat Shinde , Sanket Badhe , Xinyi Bai , Sree Bhargavi Balija , Hua-Rong Chu , Emilio Ferrara , Armstrong Foundjem
show 52 more authors
Rajat Ghosh Aakash Gupta Xuanli He Ong Chen Hui Minji Jung Madhangi Karimanal Faiza Khan Khattak Boryoung Kim Eugenia Kim Liliya Lavitas Seok Min Lim Victor Lu Jim Moirangthem Dhivya Nagasubramanian Deepak Pandita Sita Rajagopal Geetha Raju Evgeniia Razumovskaia Aravind Reddy Federico Ricciuti Nobin Sarwar Sungpil Shin Sunayana Sitaram Snehal Thorat Tharindu Cyril Weerasooriya Jasmijn Bastings Joachim Baumann Kongtao Chen Murali Emani Mariya Hendriksen Jiho Jin Jun Seong Kim Younghoon Ko Alicja Kwasniewska Minjae Lee Tom Wei-cyuan Lin Kashyap Ramanandula Manjusha Junho Myung Junyeong Park Roma Patel Shyam Ratan Sudarsun Santhiappan Priyanka Suresh Tuesday Ksheeraj Sai Vepuri Laura Amortegui-Ordonez Claire Dennis Minsuk Kahng Chris Knotz Alice Oh Balaraman Ravindran Soojung Ryu William Bartholomew Hiwot Tesfaye Lora Aroyo
This is my paper

Reviewed by Pith2026-07-08 13:35 UTCglm-5.2pith:Y42GXFG2open to challenge →

classification cs.CL cs.CY
keywords multimodal safety benchmarkcultural alignmentvision-language modelsAsia-Pacificmultilingual evaluationLLM-as-a-Judgecultural appropriatenesssafety evaluation
0
0 comments X

The pith

Culture-first benchmark exposes VLM blind spots across Asia-Pacific

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Pluralis v0.1 introduces a multimodal safety benchmark built entirely from a culture-first perspective, spanning 6,448 prompts across six Asia-Pacific countries and eight languages. The core innovation is a multimodal evaluation paradigm where individually benign text and images—such as the question 'Should I gift this?' paired with a picture of a clock—synergistically trigger locale-specific legal or cultural violations when combined. The paper argues that existing safety benchmarks, which predominantly adapt Western-centric datasets and optimize for global consensus, structurally miss regional laws, cultural taboos, and socio-linguistic nuances. By disentangling universal safety violations from localized cultural appropriateness as a separate evaluation axis, Pluralis reveals that frontier vision-language models exhibit systematic, locale-specific failure modes—including image misidentification with downstream harm, missed item-context-locale interactions, and inadequate refusals—that globally averaged metrics conceal. The paper also introduces Judge-Pluralis, an agreement-gated LLM-as-a-Judge ensemble optimized via automatic prompt optimization on human-annotated ground truth, to operationalize evaluation at scale.

Core claim

The central discovery is that frontier vision-language models fail in culturally specific, systematic ways that are invisible to existing safety benchmarks. These failures are not random: they cluster by locale and language, and they arise from a mechanism the paper calls synergistic multimodal triggering—where neither the text nor the image is harmful alone, but their combination activates a region-specific legal violation (e.g., e-cigarette possession in Singapore) or cultural taboo (e.g., gifting a clock in Chinese contexts, where 'song zhong' is a near-homophone for attending a funeral). The paper demonstrates that separating safety from cultural appropriateness as distinct axes produces

What carries the argument

The paper's central mechanism is the culture-first multimodal prompt: a standardized text template (e.g., 'Should I...') paired with an open-domain image, both benign in isolation but jointly triggering a locale-specific hazard. Evaluation is operationalized through Judge-Pluralis, an agreement-gated ensemble of multiple LLMs that scores responses along two independent axes—safety (Safe/Unsafe/Unsure) and cultural appropriateness (Appropriate/Inappropriate/Unsure)—conditioned on the target locale. The ensemble uses automatic prompt optimization with curated few-shot examples to align with human consensus, and a tie-breaker model resolves disagreements.

If this is right

  • Frontier VLMs deployed in Asia-Pacific markets may systematically endorse locally illegal acts or violate cultural taboos in everyday, non-adversarial interactions—a risk invisible to current safety evaluations.
  • The multimodal synergistic-triggering paradigm could be extended beyond Asia-Pacific to any region where the combination of a benign image and benign text activates a local legal or cultural hazard, providing a scalable template for culture-first safety testing.
  • The bottom-up cultural taxonomy derived from annotator explanations reveals that root causes of harm differ substantially across locales—e.g., religious insensitivity dominates in India while linguistic and dialectal nuance drives harm in Bangladesh—suggesting that monolithic cultural alignment strategies will fail.
  • The finding that LLM-as-a-Judge ensembles miss roughly two-thirds of violations on low-base-rate systems implies that automated cultural safety evaluation remains an unsolved problem requiring fundamentally different approaches.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the synergistic-triggering mechanism is as general as the paper implies, then the same paradigm could expose analogous blind spots in audio-language or video-language models, where individually benign modalities might combine to trigger region-specific harms.
  • The observed pattern that all three frontier models fail identically on certain culturally specific prompts (e.g., uniformly misidentifying camphor as sugar) suggests shared training-data gaps rather than model-specific weaknesses—implying that the entire frontier model class may share a common cultural blind spot that competitive benchmarks cannot surface.
  • The low inter-annotator agreement for several locales (alpha as low as 0.35-0.5) may itself be a signal: cultural appropriateness may be inherently pluralistic within a locale, meaning no single ground-truth label can capture the full distribution of acceptable responses, and evaluation frameworks may need to model appropriateness as a distribution rather than a binary.

Load-bearing premise

The paper's comparative conclusions about model safety and cultural appropriateness rest on Judge-Pluralis, an automated evaluator whose own reliability the paper acknowledges is limited: it misses 61-68% of violations for low-base-rate systems, grade bands span two to three tiers under evaluator variance, and inter-annotator agreement for several locales is moderate at best (alpha 0.35-0.5). The entire model comparison framework depends on this evaluator being reliableenough

What would settle it

If frontier VLMs were tested with prompts that pair the same benign images and text but in locales where the combined content carries no cultural or legal violation, and the models performed identically, then the failure modes would be attributable to general multimodal reasoning deficits rather than culture-specific blind spots.

Figures

Figures reproduced from arXiv: 2607.06196 by Aakash Gupta, Alice Oh, Alicia Parrish, Alicja Kwasniewska, Aravind Reddy, Armstrong Foundjem, Balaraman Ravindran, Boryoung Kim, Chris Knotz, Claire Dennis, Deepak Pandita, Dhivya Nagasubramanian, Emilio Ferrara, Eugenia Kim, Evgeniia Razumovskaia, Faiza Khan Khattak, Federico Ricciuti, Geetha Raju, Hiwot Tesfaye, Hua-Rong Chu, Jasmijn Bastings, Jiho Jin, Jim Moirangthem, Joachim Baumann, Junho Myung, Jun Seong Kim, Junyeong Park, Kongtao Chen, Ksheeraj Sai Vepuri Laura Amortegui-Ordonez, Liliya Lavitas, Lora Aroyo, Madhangi Karimanal, Mariya Hendriksen, Minjae Lee, Minji Jung, Minsuk Kahng, Murali Emani, Nobin Sarwar, Ong Chen Hui, Priyanka Suresh, Rajat Ghosh, Rajat Shinde, Roma Patel, Sanket Badhe, Seok Min Lim, Shyam Ratan, Sita Rajagopal, Snehal Thorat, Soojung Ryu William Bartholomew, Sree Bhargavi Balija, Sudarsun Santhiappan, Sunayana Sitaram, Sungpil Shin, Tharindu Cyril Weerasooriya, Tom Wei-cyuan Lin Kashyap Ramanandula Manjusha, Tuesday, Victor Lu, Xinyi Bai, Xuanli He, Younghoon Ko.

Figure 1
Figure 1. Figure 1: In Pluralis, each prompt pairs a formulaic text (e.g., “should I . . . ”) with an image, conforming to three key dimensions. (1) Location-conditioned safety: packing an e-cigarette is unremarkable in most jurisdictions but is an illegal import/use in Singapore, Taiwan, India.(2) Safety ⊥ cultural appro￾priateness: gifting a clock is physically harmless, yet in Chinese-cultural contexts it is taboo (giving … view at source ↗
Figure 2
Figure 2. Figure 2: Pluralis data pipeline. Six regional partner teams author independent, culture-specific English seed sets of multimodal prompts (text + image) grounded in local legal, religious, and social norms. The prompts are then translated into 1–2 primary languages for the locale and all prompts and translations are expert-validated.This prompt set is used to generate responses from VLM SUTs and evaluated for safety… view at source ↗
Figure 3
Figure 3. Figure 3: Judge-Pluralis development and evaluation framework. Human annotations establish curated ground truth for evaluator development. Stratified few-shot examples selected across locales and languages are used by Automatic Prompt Optimization (APO) to iteratively refine the evaluator prompt. The optimized prompt is evaluated using the agreement-gated Judge-Pluralis ensemble against the reference annotations unt… view at source ↗
Figure 4
Figure 4. Figure 4: Per-axis SUT grading using Judge-Pluralis. Diamond markers indicate the [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Safety Violation and Cultural Inappropriateness rates (%) for each SUT (A, B, C) within each [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Hazard category distribution per locale in [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: t-SNE visualisation of per-locale English prompts. More detailed description of the Figure can be [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cross-cultural distribution of hazard categories and extracted root causes of cultural appropriate [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: 5-point ordinal severity scales applied by the evaluator. (a) Safety scale calibrates the harmfulness [PITH_FULL_IMAGE:figures/full_fig_p025_9.png] view at source ↗
read the original abstract

Current AI safety evaluation and benchmarking frameworks predominantly rely on Western-centric culture-agnostic defaults that mask critical regional laws, socio-linguistic nuances, and cultural taboos, leaving Vision-Language Models (VLMs) vulnerable in global deployments. We introduce Pluralis v0.1: a novel multimodal, multi-regional, and multilingual dataset built from a culture-first perspective. Spanning 6,448 prompts across six Asia-Pacific countries (Bangladesh, India, Korea, Pakistan, Singapore, Taiwan) and eight languages, Pluralis diverges from prior work by natively sourcing localized safety hazards rather than adapting Western datasets. Crucially, it introduces a multimodal evaluation paradigm: user text (e.g., "Should I gift this?") and an image referring to "this" (e.g., a clock) - both innocuous in isolation, but synergistically triggering specific legal or cultural violations. Pluralis disentangles universal safety violations from localized cultural appropriateness, establishing the latter as a first-class evaluation axis. To operationalize this, we present Judge-Pluralis, an agreement-gated LLM-as-a-Judge ensemble trained on examples classified in an empirically derived cultural taxonomy. Observing VLM behavior on a subset of the Pluralis surfaces recurring, locale-specific failure modes such as image misidentifications with downstream harm, missed item-context-locale interactions, and inadequate refusals. These failure modes vary systematically across locales and languages, exposing blind spots that globally averaged metrics conceal. Ultimately, Pluralis is not presented as a solved evaluation framework for cultural alignment, but rather as a first step and catalyst for future innovation. We call upon the research community to utilize this foundation to advance the science of multilingual, multicultural evaluation to better support AI cultural alignment globally.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 8 minor

Summary. This paper introduces Pluralis v0.1, a multicultural, multimodal, multilingual safety benchmark comprising 6,448 prompts across six Asia-Pacific locales and eight languages. The dataset is constructed from a culture-first perspective: regional experts natively author safety hazards and cultural appropriateness scenarios, rather than adapting Western datasets. A key innovation is the multimodal paradigm, where benign text and benign images synergistically trigger locale-specific violations. The paper also introduces Judge-Pluralis, an agreement-gated LLM-as-a-Judge ensemble optimized via Automatic Prompt Optimization (APO) against human annotations, and presents preliminary SUT evaluation results for three frontier VLMs. A bottom-up cultural appropriateness taxonomy is derived from annotator explanations. The paper is transparent about limitations, labeling all SUT comparison results as preliminary.

Significance. The dataset construction methodology is the paper's strongest contribution. The culture-first approach—where hazards are conceptualized natively by regional experts rather than translated from English-centric sources—is a genuine methodological advance for safety benchmarking. The multimodal paradigm (benign text + benign image → synergistic violation) is well-motivated and illustrated with compelling examples (Figure 1, Table 7). The disentanglement of safety from cultural appropriateness as separate evaluation axes is a useful conceptual contribution. The bottom-up taxonomy (§4, §B) is empirically grounded and the annotator demographic reporting (§3.5) is commendably detailed. The linguistic diversity analysis (Table 1) demonstrates that even with eight APAC languages, the dataset achieves typological diversity comparable to benchmarks with larger language samples. The qualitative failure catalogue (Table 7) provides falsifiable, concrete examples of model failure modes.

major comments (3)
  1. §6, Table 5, Figure 4: The abstract claims that frontier VLMs 'exhibit systematic, locale-specific failure modes that globally averaged metrics conceal.' However, the paper's own evaluator reliability analysis undermines the strength of this claim. §7 reports a 61–68% false negative violation rate (FNVR) for low-base-rate systems and notes that 'this error reverses with the system's base rate.' This means Judge-Pluralis is systematically better at detecting violations for high-violation SUTs (e.g., SUT C) than for low-violation SUTs (e.g., SUT A/B), creating directional bias in the comparative scores in Table 5. Additionally, Figure 4 shows that grade-band uncertainty spans two to three tiers for all SUTs. The paper should either (a) scale back the 'systematic failure modes' claim in the abstract and §6 to match what the evaluator can reliably support, or (b) provide per-locale, per-SUT-
  2. §3.5, §5: Inter-annotator agreement is as low as α≈0.35–0.5 for India, Pakistan, Taiwan, and Korean-language prompts (§3.5). This human ground truth is used to train Judge-Pluralis via APO on a 70/30 split (§5). The paper does not analyze how low-IRR ground truth propagates into evaluator reliability. Since the APO loop optimizes the judge to match human consensus, and that consensus is noisy for several locales, the evaluator's accuracy in those locales may be overstated. The reported aggregate accuracy improvements (4.7% safety, 18.9% cultural, §5) should be disaggregated by locale and reported alongside the corresponding IRR, so readers can assess whether evaluator performance is concentrated in high-IRR locales (Bangladesh, Singapore) while remaining unreliable for the moderate-IRR locales.
  3. §6, Table 5: The SUT performance scores are computed using locale-stratified weighting (Eq. 1) and mapped to grade bands via S* = S_SUT/S_Ref against an anonymized reference model (SUT B). However, the choice of SUT B as reference is not justified, and since SUT B's own scores vary substantially across locales and languages (e.g., safety scores ranging from 0.0000 to 0.2400), using it as a denominator makes S* unstable. The paper should either justify this choice or present absolute scores alongside the relative ratios to allow readers to assess robustness independently of the reference choice.
minor comments (8)
  1. Table 3 is labeled 'Intra-modal agreement' but appears to report intra-model response variance (same model, multiple samples). The term 'intra-modal' is confusing in a multimodal context; consider 'intra-model' or 'response stability.'
  2. §3.4: The variance pilot study (Table 3) covers only India, Korea, and Singapore, but the dataset includes six locales. The paper should note this coverage gap explicitly when generalizing the single-sample decision to all locales.
  3. Table 4: The Korean subset has three variants (honorific, casual, contextualized) with different train/dev/test splits, which complicates cross-locale comparability. A footnote or note in the table would help readers understand this structural difference.
  4. §B: The taxonomy development process uses an 'LLM-assisted feedback loop' but does not specify which LLM was used, how many iterations were run, or how convergence was validated beyond the '<50 remaining in Other' threshold. Adding these details would improve reproducibility.
  5. Figure 4: The x-axis label 'Relative Ratio Score (S*)' could be clearer about directionality. While 'Lower is better' is noted, the axis range (0–5) is not explained—readers may not know that S*≥3.0 maps to 'Poor.'
  6. §E.4: The few-shot example for Taiwan (pit bull) labels the response as 'Critically Violating' (Severity 5) for safety, but the explanation for why pit bull ownership is critically unsafe in Taiwan is not provided in the main text. A brief note would help readers understand the locale-specific legal context.
  7. The paper uses 'APAC' and 'Asia-Pacific' interchangeably; standardizing on one term would improve consistency.
  8. Several references appear to be from 2026 (e.g., Choi et al., 2026; Ploeger et al., 2026); if these are forthcoming/accepted, the bibliographic status should be clarified.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee identifies three major concerns: (1) the abstract's 'systematic failure modes' claim is not fully supported given evaluator reliability limitations; (2) low inter-annotator agreement in several locales may propagate into Judge-Pluralis accuracy, and aggregate APO improvements should be disaggregated by locale; (3) the choice of SUT B as the reference model for relative scoring is unjustified and potentially unstable. We address each point below and commit to revisions for all three.

read point-by-point responses

  1. Referee: The abstract claims frontier VLMs 'exhibit systematic, locale-specific failure modes that globally averaged metrics conceal,' but the evaluator reliability analysis (§7, 61–68% FNVR for low-base-rate systems, directional bias favoring high-violation SUTs) and grade-band uncertainty spanning 2–3 tiers (Figure 4) undermine this claim. The paper should either scale back the claim or provide per-locale, per-SUT confidence intervals.

    Authors: The referee is correct that the FNVR analysis in §7 creates a directional bias that qualifies the strength of the 'systematic failure modes' claim as stated in the abstract. We acknowledge this gap honestly. In the revised manuscript, we will take approach (a): we will scale back the abstract and §6 language to match what the evaluator can reliably support. Specifically, we will replace 'systematic, locale-specific failure modes' with language that frames the findings as preliminary observations of locale-specific patterns, explicitly conditioned on evaluator reliability limitations. We will also add a sentence in §6 cross-referencing the FNVR bias and its implications for comparing low-violation SUTs (A, B) versus high-violation SUTs (C). We note that the qualitative failure catalogue (Table 7) provides concrete, human-verified examples that are independent of Judge-Pluralis's reliability, so a weaker but still meaningful version of the claim can be honestly maintained. However, we agree that the quantitative 'systematic' framing in the abstract overstates what the current evaluator supports. revision: yes

  2. Referee: Inter-annotator agreement is as low as α≈0.35–0.5 for India, Pakistan, Taiwan, and Korean-language prompts (§3.5). This ground truth trains Judge-Pluralis via APO (§5), but the paper does not analyze how low-IRR propagates into evaluator reliability. Aggregate APO improvements (4.7% safety, 18.9% cultural) should be disaggregated by locale alongside corresponding IRR.

    Authors: This is a fair and important point. The manuscript currently reports only aggregate APO improvements, which could indeed mask concentration of gains in high-IRR locales (Bangladesh, Singapore). We will add a per-locale disaggregation of Judge-Pluralis accuracy improvements alongside the corresponding IRR values in a new table in §5. This will allow readers to directly assess whether evaluator performance tracks IRR across locales. We acknowledge that we cannot fully answer the deeper question of how low-IRR ground truth propagates into evaluator reliability without additional analysis that we have not yet conducted; we will state this as an explicit limitation in §7 rather than claiming we have resolved it. We will also add a sentence in §5 noting that APO optimization against noisy consensus may partially overfit to annotator disagreement patterns in moderate-IRR locales, and that the per-locale disaggregation is provided to allow readers to judge this for themselves. revision: yes

  3. Referee: The choice of SUT B as the reference model for relative scoring (S* = S_SUT/S_Ref) is not justified, and SUT B's own scores vary substantially across locales (e.g., 0.0000 to 0.2400 for safety), making S* unstable. The paper should justify this choice or present absolute scores alongside relative ratios.

    Authors: The referee is correct that the choice of SUT B as reference is not justified in the current manuscript and that its locale-level score variance introduces instability in S*. We will address this in two ways. First, we will add a brief justification: SUT B was selected as the reference model following MLCommons AILuminate conventions, where a mid-tier open-weights model serves as the baseline for relative grading. However, we agree this justification is insufficient given the score instability the referee identifies. Second, and more importantly, we will add absolute scores (S_SUT) alongside the relative ratios (S*) in Table 5, so readers can assess robustness independently of the reference choice. We note that Table 5 already reports the absolute S_SUT values per locale-language-axis cell; what is missing is an aggregate absolute score per SUT per axis. We will add these aggregate absolute scores in a summary row or supplementary table. We also agree that the instability of S* when the denominator approaches zero (e.g., SUT B safety = 0.0000 for Hindi and Urdu) is a real limitation that should be explicitly acknowledged. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; the benchmark and evaluator are constructed from independent human cultural knowledge and validated on held-out data.

full rationale

The paper's central claims rest on three derivation chains, none of which reduce to their inputs by construction. (1) The Pluralis dataset is built bottom-up from locale-specific cultural knowledge sourced by regional experts and annotators — it is not derived from or normalized against the SUTs being tested. (2) Judge-Pluralis is trained via APO on 70% of the human-annotated ground truth and validated on a held-out 30% split; the reported 4.7% safety and 18.9% cultural accuracy improvements are measured on this held-out data, not on the training data. The judge is then applied to the full dataset (including the test split, which was not used for APO tuning), so SUT scores in Table 5 are not reproductions of training labels. (3) The cultural taxonomy (Appendix B) is developed iteratively from annotator explanations and used for descriptive analysis (Figures 5, 8), not as a prediction that is forced by construction. The paper cites MSTS (Röttger et al., 2025) for prompt templates; Alicia Parrish appears on both author lists, but this is a methodological borrowing of templates, not a load-bearing self-citation that forces the paper's conclusions. The paper is transparent about evaluator limitations (61–68% FNVR, low IAA for some locales, grade bands spanning 2–3 tiers), but these are correctness/reliability concerns, not circularity. No step was found where a claimed prediction or first-principles result is equivalent to its inputs by definition or by fit.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The benchmark introduces a cultural appropriateness taxonomy and an evaluator ensemble as new entities, both grounded in human annotation data. The free parameters are mostly standard choices (temperature, split ratio) or adopted from MLCommons. The key axioms are domain assumptions about the separability of safety from appropriateness and the representativeness of locale labels, both of which the paper itself partially challenges in its limitations section.

free parameters (5)
  • Temperature τ=0.7 for SUT response generation = 0.7
    Chosen for response generation; the variance pilot (Table 3) justifies single sampling but the choice of 0.7 itself is not optimized.
  • AILuminate grade band thresholds (S* < 1.5, 1.5 ≤ S* < 3.0, S* ≥ 3.0) = 1.5, 3.0
    Adopted from MLCommons AILuminate rubric (§6); not derived from Pluralis data.
  • APO target disagreement threshold = Not specified
    The APO loop terminates when disagreement on the 30% validation set 'falls below a target threshold' (§5), but the threshold value is not stated.
  • 70/30 train/validation split for APO = 0.7
    Standard split ratio chosen for evaluator optimization; no sensitivity analysis provided.
  • Number of few-shot examples in evaluator prompt = 70
    Curated few-shot examples spanning three regions (§E.4); the count and selection are manually determined.
axioms (4)
  • domain assumption Safety and cultural appropriateness are orthogonal axes that can be independently assessed.
    The entire benchmark framework rests on this separation (§1, §3.5). The paper provides qualitative justification (Figure 1) but does not empirically validate orthogonality.
  • domain assumption A single locale label can serve as a meaningful unit of analysis for cultural appropriateness.
    The benchmark assigns each prompt to one locale and evaluates responses against that locale's norms. The paper itself acknowledges this is a limitation (§7: 'Locales are not Internally Homogeneous'), with IRR as low as α≈0.35.
  • ad hoc to paper LLM-as-a-Judge ensembles can reliably evaluate cultural appropriateness when optimized against human annotations.
    The Judge-Pluralis design (§5) assumes that APO-tuned LLM judges can approximate human cultural judgments. The 61-68% false negative rate (§7) challenges this assumption.
  • domain assumption Single-sample estimation at τ=0.7 is sufficient for aggregate-level SUT comparison.
    The variance pilot (Table 3) shows 80-97% safety agreement and 72-83% appropriateness agreement across three samples, which the authors use to justify single sampling (§3.4). The paper acknowledges this is a limitation for per-prompt rates.
invented entities (2)
  • Bottom-up cultural appropriateness taxonomy (9 categories + Other) independent evidence
    purpose: Classifies root causes of cultural inappropriateness in VLM responses
    The taxonomy is derived from annotator explanations via an iterative LLM-assisted process (§B) and is applied to classify prompts (Figure 8). It makes falsifiable predictions about which categories trigger failures in different locales.
  • Judge-Pluralis evaluator ensemble independent evidence
    purpose: Automated multi-axis evaluation of VLM safety and cultural appropriateness
    The evaluator is validated against held-out human annotations (30% split) and its performance is reported. However, the high false-negative rate means it is a triage tool, not a replacement for human judgment.

pith-pipeline@v1.1.0-glm · 31046 in / 3546 out tokens · 554997 ms · 2026-07-08T13:35:35.300609+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

128 extracted references · 128 canonical work pages · 16 internal anchors

  1. [1]

    Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Models , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , year=. doi:10.18653/v1/2025.naacl-long.490 , url=

    work page doi:10.18653/v1/2025.naacl-long.490 2025

  2. [2]

    Whose View of Safety? A Deep

    Charvi Rastogi and Tian Huey Teh and Pushkar Mishra and Roma Patel and Ding Wang and Mark Diaz and Alicia Parrish and Aida Mostafazadeh Davani and Zoe Ashwood and Michela Paganini and Vinodkumar Prabhakaran and Verena Rieser and Lora Aroyo , booktitle=. Whose View of Safety? A Deep. 2025 , url=

    work page 2025

  3. [3]

    2023 , eprint=

    Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction , author=. 2023 , eprint=

    work page 2023

  4. [4]

    Transactions on Machine Learning Research , year =

    Haoyi Qiu and Kung-Hsiang Huang and Ruichen Zheng and Jiao Sun and Nanyun Peng , title =. Transactions on Machine Learning Research , year =

    work page

  5. [5]

    Pfohl and Alicia Parrish and Tian Huey Teh and Roma Patel and Mark Diaz and Ding Wang and Michela Paganini and Vinodkumar Prabhakaran and Lora Aroyo and Verena Rieser , title =

    Pushkar Mishra and Charvi Rastogi and Stephen R. Pfohl and Alicia Parrish and Tian Huey Teh and Roma Patel and Mark Diaz and Ding Wang and Michela Paganini and Vinodkumar Prabhakaran and Lora Aroyo and Verena Rieser , title =. Transactions on Machine Learning Research , year =

    work page

  6. [6]

    Transactions on Machine Learning Research , year =

    Haoqin Tu and Bingchen Zhao and Chen Wei and Cihang Xie , title =. Transactions on Machine Learning Research , year =

    work page

  7. [7]

    Transactions on Machine Learning Research , year =

    Vaidehi Patil and Yi-Lin Sung and Peter Hase and Jie Peng and Tianlong Chen and Mohit Bansal , title =. Transactions on Machine Learning Research , year =

    work page

  8. [8]

    Edelman and Zhaowei Zhang and Mario G

    Usman Anwar and Abulhair Saparov and Javier Rando and Daniel Paleka and Miles Turpin and Peter Hase and Ekdeep Singh Lubana and Erik Jenner and Stephen Casper and Oliver Sourbut and Benjamin L. Edelman and Zhaowei Zhang and Mario G. Foundational Challenges in Assuring Alignment and Safety of Large Language Models , journal =. 2024 , month =

    work page 2024

  9. [9]

    CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks

    Peiqin Lin and Chenyang Lyu and Wenjiang Luo and Haotian Ye and Md Mehrab Hossain and Chunlan Ma and Shaoxiong Ji and Younes Samih and Bo Zeng and Fan Jiang and others , year=. 2604.19262 , archivePrefix=

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity

    Dasol Choi and Eugenia Kim and Jaewon Noh and Sang Seo and Eunmi Kim and Myunggyo Oh and Yunjin Park and Brigitta Jesica Kartono and Josef Pichlmeier and Helena Berndt and Sai Krishna Mendu and Glenn Johannes Tungka and Özlem Gökçe and Suresh Gehlot and Katherine Pratt and Amanda Minnich and Haon Park , year=. 2605.05662 , archivePrefix=

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    International Conference on Learning Representations (ICLR) , year=

    Multimodal Situational Safety , author=. International Conference on Learning Representations (ICLR) , year=

    work page

  12. [12]

    2026 , eprint=

    Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models , author=. 2026 , eprint=

    work page 2026

  13. [13]

    Romero, David and Lyu, Chenyang and Wibowo, Haryo Akbarianto and others , booktitle=

    work page

  14. [14]

    arXiv preprint arXiv:2505.14972 , year=

    Multimodal Cultural Safety: Evaluation Framework and Alignment Strategies , author=. arXiv preprint arXiv:2505.14972 , year=

    work page arXiv

  15. [15]

    2025 , eprint=

    SEA-SafeguardBench: Evaluating AI Safety in SEA Languages and Cultures , author=. 2025 , eprint=

    work page 2025

  16. [16]

    2026 , month =

    A Robust, Defensible, and Reproducible Methodology for Benchmarking Single-Turn Jailbreak Attacks on Large Language Models , author =. 2026 , month =

    work page 2026

  17. [17]

    and Khiu, Eric and Do

    Khan, Aditya and Shipton, Mason and Anugraha, David and Duan, Kaiyao and Hoang, Phuong H. and Khiu, Eric and Do. Proceedings of the 31st International Conference on Computational Linguistics (COLING) , pages=. 2025 , url=

    work page 2025

  18. [18]

    and Lin, Ke and Kairis, Katherine and Turner, Carlisle and Levin, Lori , booktitle=

    Littell, Patrick and Mortensen, David R. and Lin, Ke and Kairis, Katherine and Turner, Carlisle and Levin, Lori , booktitle=. 2017 , url=

    work page 2017

  19. [19]

    Computational Linguistics , volume=

    A principled framework for evaluating on typologically diverse languages , author=. Computational Linguistics , volume=. 2026 , publisher=

    work page 2026

  20. [20]

    Science Advances , volume=

    Grambank reveals the importance of genealogical constraints on linguistic diversity and highlights the impact of language loss , author=. Science Advances , volume=. 2023 , doi=

    work page 2023

  21. [21]

    Universal Sentence Encoder

    Universal Sentence Encoder , author=. arXiv preprint arXiv:1803.11175 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    Journal of Machine Learning Research , volume=

    Visualizing Data using t-SNE , author=. Journal of Machine Learning Research , volume=

    work page

  23. [23]

    Grootendorst, Maarten , journal=

    work page

  24. [24]

    Sentence-

    Reimers, Nils and Gurevych, Iryna , booktitle=. Sentence-. 2019 , url=

    work page 2019

  25. [25]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=. 2020 , url=

    work page 2020

  26. [26]

    Constructing Grounded Theory , author=

    work page

  27. [27]

    Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory , author=

    work page

  28. [28]

    The Interpretation of Cultures: Selected Essays , author=

    work page

  29. [29]

    Outline of a Theory of Practice , author=

    work page

  30. [30]

    Purity and Danger: An Analysis of Concepts of Pollution and Taboo , author=

    work page

  31. [31]

    The Ritual Process: Structure and Anti-Structure , author=

    work page

  32. [32]

    Journal of Personality and Social Psychology , volume=

    Liberals and Conservatives Rely on Different Sets of Moral Foundations , author=. Journal of Personality and Social Psychology , volume=. 2009 , doi=

    work page 2009

  33. [33]

    The Righteous Mind: Why Good People Are Divided by Politics and Religion , author=

    work page

  34. [34]

    Online Readings in Psychology and Culture , volume=

    An Overview of the Schwartz Theory of Basic Values , author=. Online Readings in Psychology and Culture , volume=. 2012 , doi=

    work page 2012

  35. [35]

    Cultures and Organizations: Software of the Mind , author=

    work page

  36. [36]

    Culture, Leadership, and Organizations: The GLOBE Study of 62 Societies , author=

    work page

  37. [37]

    First Conference on Language Modeling (COLM) , year=

    Towards Measuring the Representation of Subjective Global Opinions in Language Models , author=. First Conference on Language Modeling (COLM) , year=

    work page

  38. [38]

    Myung, Junho and Lee, Nayeon and Zhou, Yi and Jin, Jiho and Putri, Rifki Afina and Antypas, Dimosthenis and Borkakoty, Hsuvas and Kim, Eunsu and Perez-Almendros, Carla and Ayele, Abinew Ali and others , booktitle=

    work page

  39. [39]

    Chiu, Yu Ying and Jiang, Liwei and Lin, Bill Yuchen and Park, Chan Young and Li, Shuyue Stella and Ravi, Sahithya and Bhatia, Mehar and Antoniak, Maria and Tsvetkov, Yulia and Shwartz, Vered and Choi, Yejin , journal=

    work page

  40. [40]

    Rao, Abhinav and Yerukola, Akhila and Shah, Vishwa and Reinecke, Katharina and Sap, Maarten , journal=

    work page

  41. [41]

    Towards Measuring and Modeling ``Culture'' in LLM s: A Survey

    Adilazuarda, Muhammad Farid and Mukherjee, Sagnik and Lavania, Pradhyumna and Singh, Siddhant Shivdutt and Aji, Alham Fikri and O ' Neill, Jacki and Modi, Ashutosh and Choudhury, Monojit. Towards Measuring and Modeling ``Culture'' in LLM s: A Survey. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653...

    work page doi:10.18653/v1/2024.emnlp-main.882 2024

  42. [42]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , pages=

    Having Beer after Prayer? Measuring Cultural Bias in Large Language Models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , pages=

    work page

  43. [43]

    Challenges and Strategies in Cross-Cultural

    Hershcovich, Daniel and Frank, Stella and Lent, Heather and others , booktitle=. Challenges and Strategies in Cross-Cultural. 2022 , doi=

    work page 2022

  44. [44]

    , journal=

    Yong, Zheng-Xin and Menghini, Cristina and Bach, Stephen H. , journal=. Low-Resource Languages Jailbreak

    work page

  45. [45]

    The Twelfth International Conference on Learning Representations (ICLR) , year=

    Multilingual Jailbreak Challenges in Large Language Models , author=. The Twelfth International Conference on Learning Representations (ICLR) , year=

    work page

  46. [46]

    Findings of the Association for Computational Linguistics: ACL 2024 , year=

    All Languages Matter: On the Multilingual Safety of Large Language Models , author=. Findings of the Association for Computational Linguistics: ACL 2024 , year=

    work page 2024

  47. [47]

    2026 , eprint=

    Korean Culture into LLM Alignment: Toward Cultural Coherence , author=. 2026 , eprint=

    work page 2026

  48. [48]

    2026 , eprint=

    TWGuard: A Case Study of LLM Safety Guardrails for Localized Linguistic Contexts , author=. 2026 , eprint=

    work page 2026

  49. [49]

    2025 , eprint=

    JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation , author=. 2025 , eprint=

    work page 2025

  50. [50]

    2024 , doi=

    Liu, Xin and Zhu, Yichen and Gu, Jindong and Lan, Yunshi and Yang, Chao and Qiao, Yu , booktitle=. 2024 , doi=

    work page 2024

  51. [51]

    Gong, Yichen and Ran, Delong and Liu, Jinyuan and Wang, Conglei and Cong, Tianshuo and Wang, Anyu and Duan, Sisi and Wang, Xiaoyun , booktitle=

    work page

  52. [52]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    Safe inputs but unsafe output: Benchmarking cross-modality safety alignment of large vision-language models , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    work page 2025

  53. [53]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Visually Grounded Reasoning across Languages and Cultures , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    work page 2021

  54. [54]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    work page 2021

  55. [55]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Benchmarking Vision Language Models for Cultural Understanding , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    work page 2024

  56. [56]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , booktitle=. Judging. 2023 , eprint=

    work page 2023

  57. [57]

    2023 , doi=

    Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang , booktitle=. 2023 , doi=

    work page 2023

  58. [58]

    Prometheus: Inducing Fine-grained Evaluation Capability in Language Models

    Prometheus: Inducing Fine-grained Evaluation Capability in Language Models , author=. International Conference on Learning Representations (ICLR) , year=. 2310.08491 , archivePrefix=

    work page internal anchor Pith review Pith/arXiv arXiv

  59. [59]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=. 2024 , doi=

    work page 2024

  60. [60]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , pages=

    Large Language Models are not Fair Evaluators , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , pages=. 2024 , doi=

    work page 2024

  61. [61]

    and Feng, Shi , booktitle=

    Panickssery, Arjun and Bowman, Samuel R. and Feng, Shi , booktitle=. 2024 , eprint=

    work page 2024

  62. [62]

    2024 , publisher=

    Watts, Ishaan and Gumma, Varun and Yadavalli, Aditya and Seshadri, Vivek and Swaminathan, Manohar and Sitaram, Sunayana , booktitle=. 2024 , publisher=. doi:10.18653/v1/2024.emnlp-main.451 , url=

    work page doi:10.18653/v1/2024.emnlp-main.451 2024

  63. [63]

    and Hadfield, Evan and Siddarth, Divya and Bali, Kalika and Sitaram, Sunayana , booktitle=

    Hamna, Hamna and Bhat, Gayatri and Mukherjee, Sourabrata and Lalani, Faisal M. and Hadfield, Evan and Siddarth, Divya and Bali, Kalika and Sitaram, Sunayana , booktitle=. Building Benchmarks from the Ground Up: Community-Centered Evaluation of. 2026 , publisher=

    work page 2026

  64. [64]

    Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions

    Bianchi, Federico and Suzgun, Mirac and Attanasio, Giuseppe and R. Safety-Tuned. International Conference on Learning Representations (ICLR) , year=. 2309.07875 , archivePrefix=

    work page internal anchor Pith review Pith/arXiv arXiv

  65. [65]

    Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models

    Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models , author=. arXiv preprint arXiv:2409.00598 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  66. [66]

    2025 , eprint=

    Xie, Tinghao and Qi, Xiangyu and Zeng, Yi and Huang, Yangsibo and Sehwag, Udari Madhushani and Huang, Kaixuan and He, Luxi and Wei, Boyi and Li, Dacheng and Sheng, Ying and Jia, Ruoxi and Li, Bo and Li, Kai and Chen, Danqi and Henderson, Peter and Mittal, Prateek , booktitle=. 2025 , eprint=

    work page 2025

  67. [67]

    Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT) , pages=

    Taxonomy of Risks posed by Language Models , author=. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT) , pages=. 2022 , doi=

    work page 2022

  68. [68]

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback , author=. arXiv preprint arXiv:2204.05862 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  69. [69]

    Constitutional

    Bai, Yuntao and Kadavath, Saurav and Kundu, Sandipan and Askell, Amanda and Kernion, Jackson and Jones, Andy and Chen, Anna and Goldie, Anna and Mirhoseini, Azalia and McKinnon, Cameron and others , journal=. Constitutional

    work page

  70. [70]

    Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

    Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned , author=. arXiv preprint arXiv:2209.07858 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  71. [72]

    AI Magazine , volume=

    Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation , author=. AI Magazine , volume=. 2015 , doi=

    work page 2015

  72. [73]

    A Roadmap to Pluralistic Alignment

    A Roadmap to Pluralistic Alignment , author=. Proceedings of the 41st International Conference on Machine Learning (ICML) , year=. 2402.05070 , archivePrefix=

    work page internal anchor Pith review Pith/arXiv arXiv

  73. [74]

    DICES Dataset: Diversity in Conversational AI Evaluation for Safety

    Aroyo, Lora and Taylor, Alex S. and Diaz, Mark and Homan, Christopher M. and Parrish, Alicia and Serapio-Garc. Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , year=. 2306.11247 , archivePrefix=

    work page internal anchor Pith review Pith/arXiv arXiv

  74. [75]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    The `Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=. 2022 , eprint=

    work page 2022

  75. [76]

    Transactions of the Association for Computational Linguistics , volume=

    Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , doi=

    work page 2022

  76. [77]

    Dey, Priyanka and Bothra, Aayush and Khanter, Yugal and Zhao, Jieyu and Ferrara, Emilio , booktitle=. Can. 2025 , url=

    work page 2025

  77. [78]

    Coppolillo, Erica and Ferrara, Emilio , journal=

    work page

  78. [79]

    CoRR , volume =

    A Bit of Progress in Language Modeling , author =. CoRR , volume =

    work page

  79. [80]

    Computer Speech & Language , volume = 15, number = 4, pages =

    A bit of progress in language modeling , author =. Computer Speech & Language , volume = 15, number = 4, pages =. doi:10.1006/csla.2001.0174 , issn =

    work page doi:10.1006/csla.2001.0174 2001

  80. [81]

    CoRR , volume =

    Supervised Grammar Induction Using Training Data with Limited Constituent Information , author =. CoRR , volume =

    work page

Showing first 80 references.