Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM safety research is overwhelmingly English-centric, and the imbalance has grown wider every year from 2020 to 2024, according to a hand-annotated survey of nearly 300 papers.

desk verdict Useful survey and dataset, but the headline 'growing language gap' is contradicted by the paper's own Figure 1. read the letter →

arxiv 2505.24119 v1 pith:V3PYSLA3 submitted 2025-05-30 cs.CL

Zheng-Xin Yong , Beyza Ermis , Marzieh Fadaee , Stephen H. Bach , Julia Kreutzer This is my paper
classification cs.CL
keywords multilingualsafetyLLMEnglish-centricbiaslanguagegapevaluationdocumentationcrosslingualgeneralizationsurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a systematic survey of nearly 300 LLM safety publications from 2020 to 2024, drawn from the ACL family of NLP venues and annotated by hand for which languages each paper actually studies. Its central claim is that LLM safety research is overwhelmingly English-centric, that even the second-most-studied language (Mandarin Chinese) receives about ten times less attention than English, and that the absolute gap is widening over time, from 5 English-only papers more than multilingual ones in 2020 to 83 more in 2024. It also finds that non-English languages are usually folded into broad multilingual evaluations rather than studied in depth, and that only half of English-only safety papers even name the language they study. If the survey is right, the field's safety assurance is systematically under-validated for non-English speakers, exactly the populations models are increasingly deployed to serve.

What carries the argument

The carrying object is the survey corpus itself: nearly 300 publications from 2020 to 2024 at *ACL conferences and workshops, selected by keyword matching of 'safe' and 'safety' in abstracts, manually categorized into a seven-way safety taxonomy (jailbreaking attacks, toxicity and bias, factuality and hallucination, AI privacy, policy, LLM alignment, and 'not related to safety'), and annotated for the languages each work actually studies. The annotation scheme (English-only, monolingual non-English, or multilingual) shows inter-annotator agreement between 0.80 and 0.96, which is what lets the paper report gap statistics with confidence. Two analytical devices carry the argument: a frequency-versus-multilinguality plot showing that high-resource non-English languages are studied mostly in bulk multilingual papers, and a language-documentation measure, following Bender's rule that papers should name the languages they study, which reveals that 50.6% of English-only papers never mention English. The paper also re-scores an existing ten-language harmlessness table with a 'worst-case' column, showing that average scores hide catastrophic per-language failures — Vicuna's Bengali score of 18.4 versus a high average — as a concrete illustration of what English-centric reporting obscures.

What would settle it

Run the same keyword-and-annotation procedure on the machine-learning conferences the paper excludes (for example, ICLR, NeurIPS, and ICML) or on a year of fresh arXiv submissions, and compare the share of multilingual and non-English safety papers. If that share is close to the English-only share, or if the absolute count of non-English safety papers in those venues is large, the claim that the gap is widening over time would be weakened. A second check: re-annotate the papers the survey counts as 'multilingual' for whether they report per-language results; if most report only aggregate multilingual scores, the claim that non-English coverage is shallow is confirmed, and if many report per-language breakdowns, it is not.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in its own words, is that 'the vast majority of safety research is centered on English-language models, while comparatively little work addresses safety in non-English or multilingual contexts,' and that this imbalance has become more pronounced over time. The evidence is a hand-annotated corpus of nearly 300 papers from 2020 to 2024 at *ACL venues, filtered by the keywords 'safe' and 'safety' in abstracts and grouped into six safety subtopics. English-only work outnumbers multilingual and non-English work combined in every year, every subtopic, and both conference and workshop settings, and the proportional imbalance has persisted even as overall publication counts rose. Non-English languages appear mostly 'in herds,' as items inside large multilingual test suites — Swahili, Telugu, and Afrikaans, for example, appear almost exclusively that way — and 50.6% of English-only papers never state explicitly that they studied English. The paper reads these findings as a safety failure, not merely a diversity gap, because refusal training and other alignments have repeatedly been shown not to transfer across languages, leaving language-specific harms undetected as models deploy globally.

Load-bearing premise

The survey's quantitative conclusions rest on the premise that LLM safety papers published at *ACL venues whose abstracts contain 'safe' or 'safety' fairly represent the whole field; if a substantial body of multilingual safety work lives in machine-learning venues or under different framing, the measured gap would be overstated, and the Limitations note the venue exclusion without quantifying it.

Editorial extensions

If this is right

  • Reporting only average safety scores can certify a model as safe while one language remains badly unprotected; adding worst-case per-language scores to evaluations and leaderboards is a concrete minimal fix.
  • Current safety benchmarks are built almost entirely from English and Chinese content, so alignment validated on those tests cannot be assumed to hold for other languages; models evaluated on the full language set can fail badly in languages that were exempted from red-teaming.
  • Making the language-coverage metadata field in the submission system public would let the community track linguistic representation in safety research with essentially no extra effort.
  • Dedicated conference tracks and shared workshop tasks on multilingual safety would give non-English safety work a more accessible outlet, since the survey finds such work already appears disproportionately in workshops.
  • Future research should prioritize culturally grounded evaluation benchmarks, diverse multilingual safety training data (including constitutional-AI pipelines and machine translation with cultural checks), and mechanistic or influence-based study of how safety alignment transfers across languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extending the same keyword-and-annotation count to the machine-learning venues the paper excludes (for example, ICLR, NeurIPS, and ICML) could plausibly show an even wider gap, since those venues have weaker language-diversity norms than the ACL family; the Limitations acknowledge the exclusion but do not quantify its effect.
  • The 'studied in herds' pattern generates a testable prediction: re-annotating the multilingual papers for whether they report per-language failure rates rather than aggregate scores would show that much non-English coverage is inclusion-by-checklist rather than depth.
  • The worst-case score device could be generalized into a reporting standard for model cards and leaderboards, since a per-language minimum would have flagged Vicuna as unsafe for Bengali deployment despite its acceptable average.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents a systematic survey of LLM safety research published at *ACL venues from 2020 to 2024, based on abstracts containing 'safe' or 'safety'. The authors manually annotate the languages studied and safety subtopics for nearly 300 publications, report high inter-annotator agreement, and use the resulting data to argue that safety research is overwhelmingly English-centric, that non-English languages are usually studied only in broad multilingual evaluations, and that English-only papers often fail to document their language coverage. They then propose recommendations for ACL organizers and outline three future research directions: multilingual safety evaluation, culturally contextualized synthetic training data, and crosslingual safety generalization. The paper also includes a short limitations section acknowledging the venue restriction and annotation imprecision.

Significance. If its central claims hold, the paper is a useful community resource: it provides a manually curated, publicly released annotation of language coverage across a substantial slice of LLM safety research, with transparent methodology and high pairwise inter-annotator agreement (0.80-0.96 in Table 2). The survey connects a well-documented multilingual safety problem to a concrete measurement of publication practice, and the proposed future directions, especially the worst-case evaluation metric illustrated in Table 4, are actionable. The paper is not circular: the main measurements are external annotations of the literature rather than derivations from fitted parameters. The principal weakness is that the paper's headline 'growing gap' claim is not supported by its own Figure 1 counts when imbalance is measured relatively, which affects the abstract, the introduction, and Section 2.2.

major comments (4)
  1. [Section 2.2 and Figure 1] The paper's central claim that the language gap is 'growing', 'widening', and 'more pronounced over time' (Abstract, Section 1, Figure 1 caption, Section 2.2) is not supported by the counts reported in Figure 1. The English-only counts are 6, 8, 11, 26, 118 for 2020-2024, while the monolingual non-English plus multilingual counts are 1, 3, 2, 8, 35, giving English-only shares of 85.7%, 72.7%, 84.6%, 76.5%, and 77.1%, and English-to-other ratios of 6.0, 2.7, 5.5, 3.3, and 3.4. Neither series shows a widening trend, and the endpoint comparison indicates a narrowing relative gap. The only metric that widens is the absolute difference (from 5 to 83), which is expected under roughly proportional growth and is not a direct measure of imbalance. The statement in Section 2.2 that 'the increase is disproportionately concentrated in English-only research' is also contradicted by the growth rates: the non-English/multilingual category grew about 35x (from 1 to 35) versus about 19.7x for English-only (from 6 to 118). The sentence later in the same section that 'the proportional imbalance remains' is consistent with the data, but the abstract, introduction, and Figure caption assert a growing gap. The authors should revise the trend claims to describe a persistent, not growing, relative imbalance, or explicitly and consistently frame the result as a growing absolute gap with the relative trend stated as a caveat.
  2. [Section 2.1] The quantitative conclusions (gap size, growth rate, documentation percentages) depend on the corpus defined by two restrictions: venue selection limited to *ACL conferences and workshops, and keyword filtering restricted to abstracts containing 'safe' or 'safety'. Section 2.1 justifies these choices but does not quantify their effect, and the Limitations section acknowledges the venue exclusion without discussing its likely direction or magnitude. Safety work published at ICLR, NeurIPS, ICML, or in journals, and safety work framed as 'jailbreak', 'harm', 'refusal', or 'offensive language' without the selected keywords, is excluded. Because the paper's headline numbers are presented as a measurement of the field's state ('a significant and growing language gap'), the authors should either add a sensitivity analysis over alternative keyword sets and venue sets or explicitly state that the quantitative claims are scoped to *ACL venues with safe/safety abstracts, with an assessment of how the exclusion could bias the reported gap and its trend.
  3. [Table 3 and Section 2.2] The documentation comparison in Table 3 is partly tautological given the annotation protocol described in Section 2.1. A paper is classified as monolingual non-English or multilingual only if the annotation identifies non-English languages, but the documentation metric asks whether the paper explicitly mentions the languages studied; if language identification often depends on explicit mention, then the 100% documentation rates for non-English and multilingual papers may be an artifact of how the categories were constructed. The paper reports that annotators followed up on datasets when languages were not explicitly mentioned (footnote 1), but the magnitude of such inference is not reported. The authors should clarify how many papers in each category were classified without explicit language mention, and should re-analyze the documentation comparison on the subset where the language was identified independently of explicit mention, so that the documentation claim is not circular.
  4. [Abstract and Section 2.1] The abstract says the paper reviews 'nearly 300 publications', but Section 2.1 reports that 28% of the keyword-matched papers were false positives filtered out before analysis, which would leave roughly 216 analyzed papers. Reporting only the raw count overstates the analyzed corpus and could mislead readers about the strength of the evidence. The authors should state both the initial keyword-matched count and the final analyzed count in the abstract or at the start of Section 2.2.
minor comments (6)
  1. [Figure 3(a)] The axis label 'JailbreakingattacksT oxicity' in Figure 3(a) appears to have rendering or spacing errors; it should read 'Jailbreaking attacks' and 'Toxicity and bias'.
  2. [Table 4] The header of Table 4 contains spacing artifacts such as 'A verage↑' and 'W orst Case ∗ ↑'; these should be cleaned to 'Average' and 'Worst Case'.
  3. [References] The reference 'OpenAI. Openai gpt-4.5 system card' should be capitalized as 'OpenAI GPT-4.5 System Card'.
  4. [Section 2.2] The sentence about 'the increase is disproportionately concentrated in English-only research' is not only unsupported but also inconsistent with the growth-rate comparison; this should be corrected even if the Authors prefer a non-relative framing.
  5. [Section 3.1] The notation in Table 4 uses a red exclamation mark and bold text, but the caption does not define what the red text indicates for color-blind readers; consider replacing color cues with explicit labels.
  6. [Section 2.1] The inter-annotator agreement table reports means and standard deviations, but not confidence intervals or the number of repeated annotations per category beyond the 4x20 design; adding this detail would strengthen the reliability claim.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the paper's central measurements are external annotations, self-citations are contextual, and the 'growing gap' claim is a contestable metric choice rather than a derivation from the paper's own inputs.

full rationale

The paper's quantitative core is an original annotation of nearly 300 *ACL publications, released with inter-annotator agreement scores (Table 2), so the English-centricity and gap statistics do not reduce to any fitted constant or to the authors' earlier results. Self-citations such as Yong et al. (2023a), Aakanksha et al. (2024), and Peppin et al. (2025) are used to support the motivating claim that safety behavior does not transfer across languages, but that claim is independently corroborated by many non-author citations (Deng et al., 2024; Shen et al., 2024; Yoo et al., 2024) and is not the load-bearing inference of the survey; the survey's own counts stand on their own annotation procedure. The 'widening gap' conclusion is based on the absolute difference (5 to 83) and is statistically contestable because the relative English share actually fell, but choosing absolute difference as the gap metric is a methodological choice, not a circular definition. The Table 3 claim that non-English and multilingual papers have 100% documentation may be an artifact of the annotation rule in footnote 1 (languages identified from datasets when not mentioned), yet this is an internal-validity or coding concern, not a reduction of the conclusion to its definition. No equation or parameter fit is used to manufacture a prediction, and no uniqueness theorem or prior author result is invoked to forbid alternatives. Accordingly, no circular step meets the bar of the seven enumerated patterns.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This survey introduces no fitted parameters or invented entities. Its central claim depends on the corpus-defining assumptions listed above; the authors disclose the venue and annotation limitations in their Limitations section.

assumptions (3)
  • domain assumption *ACL venues are representative of LLM safety research.
    Section 2.1 'Venue selection' restricts the corpus to ACL/EMNLP/NAACL-family conferences and workshops, explicitly excluding ICLR, NeurIPS, and ICML where much LLM safety work is published. If non-English safety work disproportionately appears at those ML venues, the measured gap is overstated.
  • domain assumption Abstracts containing 'safe' or 'safety' are a good proxy for the LLM safety literature.
    Section 2.1 'Keyword filter'. Papers framed around jailbreaks, harm, refusal, or toxicity without these exact words are missed, and no recall check against a hand-curated gold set is reported.
  • domain assumption Manual annotation correctly recovers the languages studied in each paper.
    Section 2.1 'Manual categorization' and 'Language Documentation'. Annotators infer language coverage from datasets when not explicit; inter-annotator agreement is high (0.80 to 0.96) but false negatives in recall are acknowledged in the Limitations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It." pith.science (2026). https://pith.science/paper/V3PYSLA3

@misc{pith2026250524119,
  author       = {Pith},
  title        = {Pith review of: The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V3PYSLA3}},
  note         = {Machine review of arXiv:2505.24119}
}
read the original abstract

This paper presents a comprehensive analysis of the linguistic diversity of LLM safety research, highlighting the English-centric nature of the field. Through a systematic review of nearly 300 publications from 2020--2024 across major NLP conferences and workshops at *ACL, we identify a significant and growing language gap in LLM safety research, with even high-resource non-English languages receiving minimal attention. We further observe that non-English languages are rarely studied as a standalone language and that English safety research exhibits poor language documentation practice. To motivate future research into multilingual safety, we make several recommendations based on our survey, and we then pose three concrete future directions on safety evaluation, training data generation, and crosslingual safety generalization. Based on our survey and proposed directions, the field can develop more robust, inclusive AI safety practices for diverse global populations.

Figures

Figures reproduced from arXiv: 2505.24119 by the authors.

Figure 1
Figure 1. Trends of English-only and multilingual LLM safety publications in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Measure of how often a language is studied (“Frequency”) and the average number of [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Distribution of LLM safety publications by (a) safety subtopics and (b) publication venues. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    ROK-FORTRESS shows Korean-language prompts increase LLM safety suppression compared with English, while Korean geopolitical grounding often reduces that suppression, indicating translation-only evaluations miss langua...

  2. The Problem with Safety Classification is not just the Models

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Multilingual safety classifiers perform unevenly across languages, and the evaluation datasets used to test them contain many harmless prompts mislabeled as harmful.

Reference graph

Works this paper leans on

148 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  4. [4]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

  5. [5]

    The multilingual alignment prism: Aligning global and local preferences to reduce harm

    Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. The multilingual alignment prism: Aligning global and local preferences to reduce harm. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp....

    doi:10.18653/v1/2024.emnlp-main.671 2024

  6. [6]

    Why knowledge distillation amplifies gender bias and how to mitigate from the perspective of D istil BERT

    Jaimeen Ahn, Hwaran Lee, Jinhwa Kim, and Alice Oh. Why knowledge distillation amplifies gender bias and how to mitigate from the perspective of D istil BERT . In Christian Hardmeier, Christine Basta, Marta R. Costa-juss \`a , Gabriel Stanovsky, and Hila Gonen (eds.), Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), p...

    doi:10.18653/v1/2022.gebnlp-1.27 2022

  7. [7]

    Jailbreaking LLM s with A rabic transliteration and A rabizi

    Mansour Al Ghanim, Saleh Almohaimeed, Mengxin Zheng, Yan Solihin, and Qian Lou. Jailbreaking LLM s with A rabic transliteration and A rabizi. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 18584--18600, Miami, Florida, USA, November 2024. Association...

    doi:10.18653/v1/2024.emnlp-main.1034 2024

  8. [8]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

Show all 148 references
  1. [9]

    Foundational challenges in assuring alignment and safety of large language models

    Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932, 2024

    2024 arXiv

  2. [10]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  3. [11]

    Growing up in a multilingual community: Insights from language socialization

    Patricia Baquedano-L \'o pez and Shlomy Kattan. Growing up in a multilingual community: Insights from language socialization. Handbook of multilingualism and multilingual communication, 5: 0 69--99, 2007

    2007

  4. [12]

    Open problems in machine unlearning for ai safety

    Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O'Gara, Robert Kirk, Ben Bucknall, Tim Fist, et al. Open problems in machine unlearning for ai safety. arXiv preprint arXiv:2501.04952, 2025

    2025 arXiv

  5. [13]

    The dark side of language models: Exploring the potential of llms in multimedia disinformation generation and dissemination

    Dipto Barman, Ziyi Guo, and Owen Conlan. The dark side of language models: Exploring the potential of llms in multimedia disinformation generation and dissemination. Machine Learning with Applications, pp.\ 100545, 2024

    2024

  6. [14]

    The \#benderrule: On naming the languages we study and why it matters

    Emily Bender. The \#benderrule: On naming the languages we study and why it matters. The Gradient, 2019

    2019

  7. [15]

    On achieving and evaluating language-independence in nlp

    Emily M Bender. On achieving and evaluating language-independence in nlp. Linguistic Issues in Language Technology, 6, 2011

    2011

  8. [16]

    International ai safety report 2025

    Yoshua Bengio, S \"o ren Mindermann, and Daniel Privitera. International ai safety report 2025. 2025

    2025

  9. [17]

    S ee GULL multilingual: a dataset of geo-culturally situated stereotypes

    Mukul Bhutani, Kevin Robinson, Vinodkumar Prabhakaran, Shachi Dave, and Sunipa Dev. S ee GULL multilingual: a dataset of geo-culturally situated stereotypes. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for ...

    2024 doi

  10. [18]

    Ai auditing: The broken bus on the road to ai accountability

    Abeba Birhane, Ryan Steed, Victor Ojewale, Briana Vecchione, and Inioluwa Deborah Raji. Ai auditing: The broken bus on the road to ai accountability. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp.\ 612--643. IEEE, 2024

    2024

  11. [19]

    Systematic inequalities in language technology performance across the world`s languages

    Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. Systematic inequalities in language technology performance across the world`s languages. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for ...

    2022 doi

  12. [20]

    Language contamination helps explains the cross-lingual capabilities of E nglish pretrained models

    Terra Blevins and Luke Zettlemoyer. Language contamination helps explains the cross-lingual capabilities of E nglish pretrained models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Proces...

    2022 doi

  13. [21]

    Offensive words in chinese dialects

    Lorna Carson and Ning Jiang. Offensive words in chinese dialects. In An Anatomy of Chinese Offensive Words: A Lexical and Semantic Analysis, pp.\ 99--143. Springer, 2021

    2021

  14. [22]

    Speak easy: Eliciting harmful jailbreaks from llms with simple interactions

    Yik Siu Chan, Narutatsu Ri, Yuxin Xiao, and Marzyeh Ghassemi. Speak easy: Eliciting harmful jailbreaks from llms with simple interactions. arXiv preprint arXiv:2502.04322, 2025

    2025 arXiv

  15. [23]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90\ URL https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  16. [24]

    Command a: An enterprise-ready large language model

    Team Cohere, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Rapha \"e l Avalos, et al. Command a: An enterprise-ready large language model. arXiv preprint arXiv:2504.00698, 2025

    2025 arXiv

  17. [25]

    Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale

    Marta Costa-juss \`a , Pierre Andrews, Eric Smith, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Daniel Licht, and Carleigh Wood. Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale. In Houda B...

    2023

  18. [26]

    Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale

    Marta Costa-juss \`a , Pierre Andrews, Eric Smith, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Daniel Licht, and Carleigh Wood. Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale. In Houda B...

    2023

  19. [27]

    Toxicity in multilingual machine translation at scale

    Marta Costa-juss \`a , Eric Smith, Christophe Ropers, Daniel Licht, Jean Maillard, Javier Ferrando, and Carlos Escolano. Toxicity in multilingual machine translation at scale. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Li...

    2023 doi

  20. [28]

    Overview of the shared task on machine translation gender bias evaluation with multilingual holistic bias

    Marta Costa-juss \`a , Pierre Andrews, Christine Basta, Juan Ciro, Agnieszka Falenska, Seraphina Goldfarb-Tarrant, Rafael Mosquera, Debora Nozza, and Eduardo S \'a nchez. Overview of the shared task on machine translation gender bias evaluation with multilingual holistic bias....

    2024

  21. [29]

    Risk taxonomy, mitigation, and assessment benchmarks of large language model systems

    Tianyu Cui, Yanling Wang, Chuanpu Fu, Yong Xiao, Sijia Li, Xinhao Deng, Yunpeng Liu, Qinglin Zhang, Ziyi Qiu, Peiyang Li, et al. Risk taxonomy, mitigation, and assessment benchmarks of large language model systems. arXiv preprint arXiv:2401.05778, 2024

    2024 arXiv

  22. [30]

    Aya vision: Advancing the frontier of multilingual multimodality

    Saurabh Dash, Yiyang Nan, John Dang, Arash Ahmadian, Shivalika Singh, Madeline Smith, Bharat Venkitesh, Vlad Shmyhlo, Viraat Aryabumi, Walter Beller-Morales, et al. Aya vision: Advancing the frontier of multilingual multimodality. arXiv preprint arXiv:2505.08751, 2025

    2025 arXiv

  23. [31]

    A systematic review of translation and cross-cultural adaptation of instruments for the selection of assistive technologies

    B \'a rbara Ians \ a de Lima Barroso, Cl \'a udia Regina Cabral Galv \ a o, Luiz Bueno da Silva, and Selma Lancman. A systematic review of translation and cross-cultural adaptation of instruments for the selection of assistive technologies. Occupational Therapy International, ...

    2018

  24. [32]

    Emerging security challenges of large language models

    Herve Debar, Sven Dietrich, Pavel Laskov, Emil C Lupu, and Eirini Ntoutsi. Emerging security challenges of large language models. arXiv preprint arXiv:2412.17614, 2024

    2024 arXiv

  25. [33]

    Multilingual jailbreak challenges in large language models

    Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak challenges in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=vESNKdEMGp

    2024

  26. [34]

    o rklund, and Henrik Bj \

    Hannah Devinney, Jenny Bj \"o rklund, and Henrik Bj \"o rklund. We don`t talk about that: Case studies on intersectional analysis of social bias in large language models. In Agnieszka Fale \'n ska, Christine Basta, Marta Costa-juss \`a , Seraphina Goldfarb-Tarrant, and Debora ...

    2024 doi

  27. [35]

    The emotional force of swearwords and taboo words in the speech of multilinguals

    Jean-Marc Dewaele. The emotional force of swearwords and taboo words in the speech of multilinguals. Journal of multilingual and multicultural development, 25 0 (2-3): 0 204--222, 2004

    2004

  28. [36]

    Attacks, defenses and evaluations for LLM conversation safety: A survey

    Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, and Yu Qiao. Attacks, defenses and evaluations for LLM conversation safety: A survey. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association fo...

    2024 doi

  29. [37]

    Reducing privacy risks in online self-disclosures with language models

    Yao Dou, Isadora Krsek, Tarek Naous, Anubha Kabra, Sauvik Das, Alan Ritter, and Wei Xu. Reducing privacy risks in online self-disclosures with language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association fo...

    2024 doi

  30. [38]

    From one to many: Expanding the scope of toxicity mitigation in language models

    Beyza Ermis, Luiza Pozzobon, Sara Hooker, and Patrick Lewis. From one to many: Expanding the scope of toxicity mitigation in language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 1504...

    2024 doi

  31. [39]

    Language: The cultural tool

    Daniel L Everett. Language: The cultural tool. Vintage, 2012

    2012

  32. [40]

    Sociocultural factors that determine language choice in a multilingual society

    Mahmud Fasya and Dini Gilang Sari. Sociocultural factors that determine language choice in a multilingual society. In Fifth International Conference on Language, Literature, Culture, and Education (ICOLLITE 2021), pp.\ 412--418. Atlantis Press, 2021

    2021

  33. [41]

    Can AI relate: Testing large language model response for mental health support

    Saadia Gabriel, Isha Puri, Xuhai Xu, Matteo Malgaroli, and Marzyeh Ghassemi. Can AI relate: Testing large language model response for mental health support. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: ...

    2024 doi

  34. [42]

    Code-switching

    Penelope Gardner-Chloros. Code-switching. Cambridge university press, 2009

    2009

  35. [43]

    Regulation and NLP ( R eg NLP ): Taming large language models

    Catalina Goanta, Nikolaos Aletras, Ilias Chalkidis, Sofia Ranchord \'a s, and Gerasimos Spanakis. Regulation and NLP ( R eg NLP ): Taming large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natu...

    2023 doi

  36. [44]

    Language translation & cross-cultural adaptation guideline

    Claudia Gorecki, Julia M Brown, Michelle Briggs, Suzanne Coleman, Carol Dealey, Elizabeth McGinnis, E Andrea Nelson, Nikki Stubbs, Lyn Wilson, and Jane Nixon. Language translation & cross-cultural adaptation guideline. Recommendations for language translation and cross-cultura...

    2014

  37. [45]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  38. [46]

    Studying large language model generalization with influence functions

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023

    2023 arXiv

  39. [47]

    Deliberative alignment: Reasoning enables safer language models

    Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339, 2024

    2024 arXiv

  40. [48]

    Guerreiro, Duarte M

    Nuno M. Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and Andr \'e F. T. Martins. Hallucinations in large multilingual translation models. Transactions of the Association for Computational Linguistics, 11: 0 1500--1517, 2023. doi:...

    2023 doi

  41. [49]

    Care: Aligning language models for regional cultural awareness

    Geyang Guo, Tarek Naous, Hiromi Wakaki, Yukiko Nishimura, Yuki Mitsufuji, Alan Ritter, and Wei Xu. Care: Aligning language models for regional cultural awareness. arXiv preprint arXiv:2504.05154, 2025

    2025

  42. [50]

    Model merging and safety alignment: One bad model spoils the bunch

    Hasan Abed Al Kader Hammoud, Umberto Michieli, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem, and Mete Ozay. Model merging and safety alignment: One bad model spoils the bunch. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for...

    2024 doi

  43. [51]

    Active learning for robust and representative LLM generation in safety-critical scenarios

    Sabit Hassan, Anthony Sicilia, and Malihe Alikhani. Active learning for robust and representative LLM generation in safety-critical scenarios. In Sachin Kumar, Vidhisha Balachandran, Chan Young Park, Weijia Shi, Shirley Anugrah Hayati, Yulia Tsvetkov, Noah Smith, Hannaneh Haji...

    2024

  44. [52]

    Tuba: Cross-lingual transferability of backdoor attacks in llms with instruction tuning

    Xuanli He, Jun Wang, Qiongkai Xu, Pasquale Minervini, Pontus Stenetorp, Benjamin IP Rubinstein, and Trevor Cohn. Tuba: Cross-lingual transferability of backdoor attacks in llms with instruction tuning. arXiv preprint arXiv:2404.19597, 2024

    2024 arXiv

  45. [53]

    Language in culture; conference on the interrelations of language and other aspects of culture

    Harry Ed Hoijer. Language in culture; conference on the interrelations of language and other aspects of culture. 1954

    1954

  46. [54]

    T rust A gent: Towards safe and trustworthy LLM -based agents

    Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. T rust A gent: Towards safe and trustworthy LLM -based agents. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics:...

    2024 doi

  47. [55]

    A survey on large language models with multilingualism: Recent advances and new frontiers

    Kaiyu Huang, Fengran Mo, Xinyu Zhang, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, et al. A survey on large language models with multilingualism: Recent advances and new frontiers. arXiv preprint arXiv:2405.10936, 2024 a

    2024 arXiv

  48. [56]

    Meralion-textllm: Cross-lingual understanding of large language models in chinese, indonesian, malay, and singlish

    Xin Huang, Tarun Kumar Vangani, Minh Duc Pham, Xunlong Zou, Bin Wang, Zhengyuan Liu, and Ai Ti Aw. Meralion-textllm: Cross-lingual understanding of large language models in chinese, indonesian, malay, and singlish. arXiv preprint arXiv:2501.08335, 2024 b

    2024 arXiv

  49. [57]

    Polyglotoxicityprompts: Multilingual evaluation of neural toxic degeneration in large language models

    Devansh Jain, Priyanshu Kumar, Samuel Gehman, Xuhui Zhou, Thomas Hartvigsen, and Maarten Sap. Polyglotoxicityprompts: Multilingual evaluation of neural toxic degeneration in large language models. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum...

    2024

  50. [58]

    Safe rlhf-v: Safe reinforcement learning from human feedback in multimodal large language models

    Jiaming Ji, Xinyu Chen, Rui Pan, Han Zhu, Conghui Zhang, Jiahao Li, Donghai Hong, Boyuan Chen, Jiayi Zhou, Kaile Wang, et al. Safe rlhf-v: Safe reinforcement learning from human feedback in multimodal large language models. arXiv preprint arXiv:2503.17682, 2025

    2025 arXiv

  51. [59]

    u tze, J \

    Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayy \'a n O'Brien, Hengyu Luo, Hinrich Sch \"u tze, J \"o rg Tiedemann, et al. Emma-500: Enhancing massively multilingual adaptation of large language models. arXiv preprint arXiv:2409.17892, 2024

    2024

  52. [60]

    The relationship between culture and language

    Wenying Jiang. The relationship between culture and language. ELT journal, 54 0 (4): 0 328--334, 2000

    2000

  53. [61]

    The state and fate of linguistic diversity and inclusion in the NLP world

    Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the ...

    2020 doi

  54. [62]

    Towards safe multilingual frontier AI

    Arturs Kanepajs, Vladimir Ivanov, and Richard Moulange. Towards safe multilingual frontier AI . In Workshop on Socially Responsible Language Modelling Research, 2024. URL https://openreview.net/forum?id=iFHsnIkj4q

    2024

  55. [63]

    a si, Atte Oksanen, and Pekka R \

    Teo Keipi, Matti N \"a si, Atte Oksanen, and Pekka R \"a s \"a nen. Online hate and harmful content: Cross-national perspectives. Taylor & Francis, 2016

    2016

  56. [64]

    Banana Bending: Asian-Australian and Asian-Canadian Literatures

    Tseen-Ling Khoo. Banana Bending: Asian-Australian and Asian-Canadian Literatures. Hong Kong University Press, 2003

    2003

  57. [65]

    Comparative study of multilingual idioms and similes in large language models

    Paria Khoshtab, Danial Namazifard, Mostafa Masoudi, Ali Akhgary, Samin Mahdizadeh Sani, and Yadollah Yaghoobzadeh. Comparative study of multilingual idioms and similes in large language models. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugeni...

    2025

  58. [66]

    L ife T ox: Unveiling implicit toxicity in life advice

    Minbeom Kim, Jahyun Koo, Hwanhee Lee, Joonsuk Park, Hwaran Lee, and Kyomin Jung. L ife T ox: Unveiling implicit toxicity in life advice. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association f...

    2024 doi

  59. [67]

    The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models

    Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew M Bean, Katerina Margatina, Rafael Mosquera-Gomez, Juan Ciro, Max Bartolo, Adina Williams, He He, et al. The prism alignment dataset: What participatory, representative and individualised human feedback reveals about...

    2024

  60. [68]

    Untangling hate speech definitions: A semantic componential analysis across cultures and domains

    Katerina Korre, Arianna Muti, Federico Ruggeri, and Alberto Barr \'o n-Cede \ n o. Untangling hate speech definitions: A semantic componential analysis across cultures and domains. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational...

    2025

  61. [69]

    Language and culture

    Claire Kramsch. Language and culture. AILA review, 27 0 (1): 0 30--55, 2014

    2014

  62. [70]

    D 'ej a vu: Multilingual llm evaluation through the lens of machine translation evaluation

    Julia Kreutzer, Eleftheria Briakou, Sweta Agrawal, Marzieh Fadaee, and Kocmi Tom. D 'ej a vu: Multilingual llm evaluation through the lens of machine translation evaluation. arXiv preprint arXiv:2504.11829, 2025

    2025 arXiv

  63. [71]

    LLM -based synthetic datasets: Applications and limitations in toxicity detection

    Udo Kruschwitz and Maximilian Schmidhuber. LLM -based synthetic datasets: Applications and limitations in toxicity detection. In Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, Bharathi Raja Chakravarthi, Bornini Lahiri, Siddharth Singh, and Shyam Ratan (eds.), Proceedings of th...

    2024

  64. [72]

    Specific versus general principles for constitutional ai

    Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, et al. Specific versus general principles for constitutional ai. arXiv preprint arXiv:2310.13798, 2023

    arXiv

  65. [73]

    C3ai: Crafting and evaluating constitutions for constitutional ai

    Yara Kyrychenko, Ke Zhou, Edyta Bogucka, and Daniele Quercia. C3ai: Crafting and evaluating constitutions for constitutional ai. In Proceedings of the ACM on Web Conference 2025, WWW '25, pp.\ 3204–3218, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 979840...

    2025

  66. [74]

    Ai value alignment: How we can align artificial intelligence with human values

    Benjamin Larsen and Virginia Dignum. Ai value alignment: How we can align artificial intelligence with human values. World Economic Forum, 10 2024. URL https://weforum.org/stories/2024/10/ai-value-alignment-how-we-can-align-artificial-intelligence-with-human-values/

    2024

  67. [75]

    Preference tuning for toxicity mitigation generalizes across languages

    Xiaochen Li, Zheng Xin Yong, and Stephen Bach. Preference tuning for toxicity mitigation generalizes across languages. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 13422--13440, Miami, ...

    2024 doi

  68. [76]

    Beyond single-turn: A survey on multi-turn interactions with large language models

    Yubo Li, Xiaobin Shen, Xinyu Yao, Xueying Ding, Yidi Miao, Ramayya Krishnan, and Rema Padman. Beyond single-turn: A survey on multi-turn interactions with large language models. arXiv preprint arXiv:2504.04717, 2025

    2025 arXiv

  69. [77]

    o rg Tiedemann, Andr \'e FT Martins, and Hinrich Sch \

    Peiqin Lin, Shaoxiong Ji, J \"o rg Tiedemann, Andr \'e FT Martins, and Hinrich Sch \"u tze. Mala-500: Massive language adaptation of large language models. arXiv preprint arXiv:2401.13303, 2024

    2024 arXiv

  70. [78]

    Systematicity, compositionality and transitivity of deep NLP models: a metamorphic testing perspective

    Edoardo Manino, Julia Rozanova, Danilo Carvalho, Andre Freitas, and Lucas Cordeiro. Systematicity, compositionality and transitivity of deep NLP models: a metamorphic testing perspective. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Assoc...

    2022 doi

  71. [79]

    Translation and cultural adaptation of the survey instruments

    Beverley Maxwell, MO Martin, and DL Kelly. Translation and cultural adaptation of the survey instruments. Third international mathematics and science study (TIMSS) technical report, 1: 0 159--169, 1996

    1996

  72. [80]

    Language and culture

    Abdelfattah Mazari and Naoual Derraz. Language and culture. International Journal of Humanities and Cultural Studies, 2 0 (2): 0 350--359, 2015

    2015

  73. [81]

    Translation, cross-cultural adaptation and psychometric evaluation of yoruba version of the short-form 36 health survey

    Chidozie Emmanuel Mbada, Gafar Atanda Adeogun, Michael Opeoluwa Ogunlana, Rufus Adesoji Adedoyin, Adesanmi Akinsulore, Taofeek Oluwole Awotidebe, Opeyemi Ayodiipo Idowu, and Olumide Ayoola Olaoye. Translation, cross-cultural adaptation and psychometric evaluation of yoruba ver...

    2015

  74. [82]

    COVID -19 in B ulgarian social media: Factuality, harmfulness, propaganda, and framing

    Preslav Nakov, Firoj Alam, Shaden Shaar, Giovanni Da San Martino, and Yifan Zhang. COVID -19 in B ulgarian social media: Factuality, harmfulness, propaganda, and framing. In Ruslan Mitkov and Galia Angelova (eds.), Proceedings of the International Conference on Recent Advances...

    2021

  75. [83]

    Large language models in healthcare and medical domain: A review

    Zabir Al Nazi and Wei Peng. Large language models in healthcare and medical domain: A review. In Informatics, volume 11, pp.\ 57. MDPI, 2024

    2024

  76. [84]

    An llm-driven chatbot in higher education for databases and information systems

    Alexander Tobias Neumann, Yue Yin, Sulayman Sowe, Stefan Decker, and Matthias Jarke. An llm-driven chatbot in higher education for databases and information systems. IEEE Transactions on Education, 2024

    2024

  77. [85]

    “i searched for a religious song in amharic and got sexual content instead’’: Investigating online harm in low-resourced languages on youtube

    Hellina Hailu Nigatu and Inioluwa Deborah Raji. “i searched for a religious song in amharic and got sexual content instead’’: Investigating online harm in low-resourced languages on youtube. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparenc...

    2024

  78. [86]

    The zeno`s paradox of low-resource' languages

    Hellina Hailu Nigatu, Atnafu Lambebo Tonja, Benjamin Rosman, Thamar Solorio, and Monojit Choudhury. The zeno`s paradox of low-resource' languages. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural La...

    2024 doi

  79. [87]

    code switching

    Chad Nilep. “code switching” in sociocultural linguistics. Colorado research in linguistics, 2006

    2006

  80. [88]

    Niraula, Saurab Dulal, and Diwa Koirala

    Nobal B. Niraula, Saurab Dulal, and Diwa Koirala. Offensive language detection in N epali social media. In Aida Mostafazadeh Davani, Douwe Kiela, Mathias Lambert, Bertie Vidgen, Vinodkumar Prabhakaran, and Zeerak Waseem (eds.), Proceedings of the 5th Workshop on Online Abuse a...

    2021 doi

  81. [89]

    Exposing the limits of zero-shot cross-lingual hate speech detection

    Debora Nozza. Exposing the limits of zero-shot cross-lingual hate speech detection. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confe...

    2021 doi

  82. [90]

    Towards ai accountability infrastructure: Gaps and opportunities in ai audit tooling

    Victor Ojewale, Ryan Steed, Briana Vecchione, Abeba Birhane, and Inioluwa Deborah Raji. Towards ai accountability infrastructure: Gaps and opportunities in ai audit tooling. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp.\ 1--29, 2025

    2025

  83. [91]

    Introducing ChatGPT

    OpenAI . Introducing ChatGPT . https://openai.com/blog/chatgpt/, November 2022. Accessed [Insert Date You Accessed This Page]

    2022

  84. [92]

    Openai gpt-4.5 system card

    OpenAI. Openai gpt-4.5 system card. Technical report, OpenAI, 2 2025

    2025

  85. [93]

    Gemini goes to M ed school: Exploring the capabilities of multimodal large language models on medical challenge problems & hallucinations

    Ankit Pal and Malaikannan Sankarasubbu. Gemini goes to M ed school: Exploring the capabilities of multimodal large language models on medical challenge problems & hallucinations. In Tristan Naumann, Asma Ben Abacha, Steven Bethard, Kirk Roberts, and Danielle Bitterman (eds.), ...

    2024 doi

  86. [94]

    Towards multidomain and multilingual abusive language detection: a survey

    Endang Wahyu Pamungkas, Valerio Basile, and Viviana Patti. Towards multidomain and multilingual abusive language detection: a survey. Personal and Ubiquitous Computing, 27 0 (1): 0 17--43, 2023

    2023

  87. [95]

    The multilingual divide and its impact on global ai safety

    Aidan Peppin, Julia Kreutzer, Alice Schoenauer Sebag, Kelly Marchisio, Beyza Ermis, John Dang, Samuel Cahyawijaya, Shivalika Singh, Seraphina Goldfarb-Tarrant, Viraat Aryabumi, Aakanksha, Wei-Yin Ko, Ahmet Üstün, Matthias Gallé, Marzieh Fadaee, and Sara Hooker. The multilingua...

    2025 arXiv

  88. [96]

    The brazilian version of start back screening tool-translation, cross-cultural adaptation and reliability

    Bruna Pilz, Rodrigo A Vasconcelos, Freddy B Marcondes, Samuel S Lodovichi, Wilson Mello, and D \'e bora B Grossi. The brazilian version of start back screening tool-translation, cross-cultural adaptation and reliability. Brazilian journal of physical therapy, 18: 0 453--461, 2014

    2014

  89. [97]

    Civics: Building a dataset for examining culturally-informed values in large language models

    Giada Pistilli, Alina Leidinger, Yacine Jernite, Atoosa Kasirzadeh, Alexandra Sasha Luccioni, and Margaret Mitchell. Civics: Building a dataset for examining culturally-informed values in large language models. In Proceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and ...

    2024

  90. [98]

    Countering hateful and offensive speech online - open challenges

    Flor Miriam Plaza-del Arco, Debora Nozza, Marco Guerini, Jeffrey Sorensen, and Marcos Zampieri. Countering hateful and offensive speech online - open challenges. In Jessy Li and Fei Liu (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process...

    2024 doi

  91. [99]

    Towards understanding the fragility of multilingual llms against fine-tuning attacks

    Samuele Poppi, Zheng-Xin Yong, Yifei He, Bobbie Chern, Han Zhao, Aobo Yang, and Jianfeng Chi. Towards understanding the fragility of multilingual llms against fine-tuning attacks. Proceedings of the 2025 Conference of the North American Chapter of the Association for Computati...

    2025

  92. [100]

    Multilingual large language model: A survey of resources, taxonomy and frontiers

    Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. Multilingual large language model: A survey of resources, taxonomy and frontiers. arXiv preprint arXiv:2404.04925, 2024

    2024 arXiv

  93. [101]

    Evaluating cultural and social awareness of llm web agents

    Haoyi Qiu, Alexander R Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, and Chien-Sheng Wu. Evaluating cultural and social awareness of llm web agents. arXiv preprint arXiv:2410.23252, 2024

    2024 arXiv

  94. [102]

    Multimodal cultural safety: Evaluation frameworks and alignment strategies

    Haoyi Qiu, Kung-Hsiang Huang, Ruichen Zheng, Jiao Sun, and Nanyun Peng. Multimodal cultural safety: Evaluation frameworks and alignment strategies. arXiv preprint arXiv:2505.14972, 2025

    2025

  95. [103]

    Open problems in technical ai governance

    Anka Reuel, Ben Bucknall, Stephen Casper, Tim Fist, Lisa Soder, Onni Aarne, Lewis Hammond, Lujain Ibrahim, Alan Chan, Peter Wills, et al. Open problems in technical ai governance. arXiv preprint arXiv:2407.14981, 2024

    2024 arXiv

  96. [104]

    Investigating bias in multilingual language models: Cross-lingual transfer of debiasing techniques

    Manon Reusens, Philipp Borchert, Margot Mieskes, Jochen De Weerdt, and Bart Baesens. Investigating bias in multilingual language models: Cross-lingual transfer of debiasing techniques. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on E...

    2023 doi

  97. [105]

    Safetyprompts: a systematic review of open datasets for evaluating and improving large language model safety

    Paul R \"o ttger, Fabio Pernisi, Bertie Vidgen, and Dirk Hovy. Safetyprompts: a systematic review of open datasets for evaluating and improving large language model safety. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp.\ 27617--27627, 2025

    2025

  98. [106]

    Square one bias in NLP : Towards a multi-dimensional exploration of the research manifold

    Sebastian Ruder, Ivan Vuli \'c , and Anders S gaard. Square one bias in NLP : Towards a multi-dimensional exploration of the research manifold. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 20...

    2022 doi

  99. [107]

    Procedural knowledge in pretraining drives reasoning in large language models

    Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwaraknath Gnaneshwar, Acyr Locatelli, Robert Kirk, Tim Rockt \"a schel, Edward Grefenstette, and Max Bartolo. Procedural knowledge in pretraining drives reasoning in large language models. In The Thirteenth I...

    2025

  100. [108]

    Rainbow teaming: Open-ended generation of diverse adversarial prompts

    Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, et al. Rainbow teaming: Open-ended generation of diverse adversarial prompts. Advances in Neural Information Proces...

    2024

  101. [109]

    Gender bias in machine translation

    Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9: 0 845--874, 2021

    2021

  102. [110]

    Processing figurative language in a multi-lingual task: Translation, transfer and metaphor

    Ayse Pinar Saygin. Processing figurative language in a multi-lingual task: Translation, transfer and metaphor. In Proceedings of the Workshop on Corpus-based and Processing Approaches to Figurative Language. Citeseer, 2001

    2001

  103. [111]

    The language barrier: Dissecting safety challenges of LLM s in multilingual contexts

    Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. The language barrier: Dissecting safety challenges of LLM s in multilingual contexts. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Finding...

    2024 doi

  104. [112]

    Large language model safety: A holistic survey, 2024 a

    Dan Shi, Tianhao Shen, Yufei Huang, Zhigen Li, Yongqi Leng, Renren Jin, Chuang Liu, Xinwei Wu, Zishan Guo, Linhao Yu, Ling Shi, Bojian Jiang, and Deyi Xiong. Large language model safety: A holistic survey, 2024 a . URL https://arxiv.org/abs/2412.17686

    2024 arXiv

  105. [113]

    ULMR : Unlearning large language models via negative response and model parameter average

    Shaojie Shi, Xiaoyu Tan, Xihe Qiu, Chao Qu, Kexin Nie, Yuan Cheng, Wei Chu, Xu Yinghui, and Yuan Qi. ULMR : Unlearning large language models via negative response and model parameter average. In Franck Dernoncourt, Daniel Preo t iuc-Pietro, and Anastasia Shimorina (eds.), Proc...

    2024 doi

  106. [114]

    Large language models encode clinical knowledge

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620 0 (7972): 0 172--180, 2023

    2023

  107. [115]

    Toward expert-level medical question answering with large language models

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, pp.\ 1--8, 2025

    2025

  108. [116]

    Multilingual blending: Large language model safety alignment evaluation with language mixture

    Jiayang Song, Yuheng Huang, Zhehua Zhou, and Lei Ma. Multilingual blending: Large language model safety alignment evaluation with language mixture. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp.\ 34...

    2025

  109. [117]

    Societal multilingualism

    Kamal K Sridhar. Societal multilingualism. Sociolinguistics and language teaching, 47: 0 70, 1996

    1996

  110. [118]

    Clio: Privacy-preserving insights into real-world ai use

    Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, Ankur Rathi, Saffron Huang, Alfred Mountfield, Jerry Hong, Stuart Ritchie, et al. Clio: Privacy-preserving insights into real-world ai use. arXiv preprint arXiv:2412.13678, 2024

    2024 arXiv

  111. [119]

    Cultural bias and cultural alignment of large language models

    Yan Tao, Olga Viberg, Ryan S Baker, and Ren \'e F Kizilcec. Cultural bias and cultural alignment of large language models. PNAS nexus, 3 0 (9): 0 pgae346, 2024

    2024

  112. [120]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  113. [121]

    twinkie,

    Monica M Trieu. Understanding the use of “twinkie,”“banana,” and “fob”: Identifying the origin, role, and consequences of internalized racism within asian america. Sociology Compass, 13 0 (5): 0 e12679, 2019

    2019

  114. [122]

    Aya model: An instruction finetuned open-access multilingual language model

    Ahmet \"U st \"u n, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D ' souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. Aya mode...

    2024

  115. [123]

    Measuring and mitigating name biases in neural machine translation

    Jun Wang, Benjamin Rubinstein, and Trevor Cohn. Measuring and mitigating name biases in neural machine translation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volu...

    2022 doi

  116. [125]

    All languages matter: On the multilingual safety of LLM s

    Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael Lyu. All languages matter: On the multilingual safety of LLM s. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: A...

    2024 doi

  117. [126]

    Reinforcement learning-driven LLM agent for automated attacks on LLM s

    Xiangwen Wang, Jie Peng, Kaidi Xu, Huaxiu Yao, and Tianlong Chen. Reinforcement learning-driven LLM agent for automated attacks on LLM s. In Ivan Habernal, Sepideh Ghanavati, Abhilasha Ravichander, Vijayanta Jain, Patricia Thaine, Timour Igamberdiev, Niloofar Mireshghallah, an...

    2024

  118. [127]

    Refusal direction is universal across safety-aligned languages

    Xinpeng Wang, Mingyang Wang, Yihong Liu, Hinrich Schütze, and Barbara Plank. Refusal direction is universal across safety-aligned languages. arXiv preprint arXiv:2505.17306, 2025

    2025

  119. [128]

    Yixu Wang, Yan Teng, Kexin Huang, Chengqi Lyu, Songyang Zhang, Wenwei Zhang, Xingjun Ma, Yu-Gang Jiang, Yu Qiao, and Yingchun Wang. Fake alignment: Are LLM s really aligned well? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the N...

    2024

  120. [129]

    Mediagpt: A large language model for chinese media

    Zhonghao Wang, Zijia Lu, Bo Jin, and Haiying Deng. Mediagpt: A large language model for chinese media. arXiv preprint arXiv:2307.10930, 2023

    2023 arXiv

  121. [130]

    Ai for education (ai4edu): Advancing personalized education with llm and adaptive learning

    Qingsong Wen, Jing Liang, Carles Sierra, Rose Luckin, Richard Tong, Zitao Liu, Peng Cui, and Jiliang Tang. Ai for education (ai4edu): Advancing personalized education with llm and adaptive learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Da...

    2024

  122. [131]

    The decades progress on code-switching research in NLP : A systematic survey on trends and challenges

    Genta Winata, Alham Fikri Aji, Zheng Xin Yong, and Thamar Solorio. The decades progress on code-switching research in NLP : A systematic survey on trends and challenges. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computationa...

    2023 doi

  123. [132]

    The bitter lesson learned from 2,000+ multilingual benchmarks

    Minghao Wu, Weixuan Wang, Sinuo Liu, Huifeng Yin, Xintong Wang, Yu Zhao, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. The bitter lesson learned from 2,000+ multilingual benchmarks. arXiv preprint arXiv:2504.15521, 2025

    2025 arXiv

  124. [133]

    A survey of multilingual models for automatic speech recognition

    Hemant Yadav and Sunayana Sitaram. A survey of multilingual models for automatic speech recognition. arXiv preprint arXiv:2202.12576, 2022

    2022 arXiv

  125. [134]

    arabizi

    Mohammad Ali Yaghan. " arabizi": A contemporary style of arabic slang. Design issues, 24 0 (2): 0 39--52, 2008

    2008

  126. [135]

    Benchmarking llm guardrails in handling multilingual toxicity

    Yahan Yang, Soham Dan, Dan Roth, and Insup Lee. Benchmarking llm guardrails in handling multilingual toxicity. arXiv preprint arXiv:2410.22153, 2024 a

    2024 arXiv

  127. [136]

    Self-distillation bridges distribution gap in language model fine-tuning

    Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu. Self-distillation bridges distribution gap in language model fine-tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association f...

    2024 doi

  128. [137]

    Safeworld: Geo-diverse safety alignment

    Da Yin, Haoyi Qiu, Kung-Hsiang Huang, Kai-Wei Chang, and Nanyun Peng. Safeworld: Geo-diverse safety alignment. Advances in Neural Information Processing Systems, 37: 0 128734--128768, 2024

    2024

  129. [138]

    Low-resource languages jailbreak GPT -4

    Zheng Xin Yong, Cristina Menghini, and Stephen Bach. Low-resource languages jailbreak GPT -4. In Socially Responsible Language Modelling Research, 2023 a . URL https://openreview.net/forum?id=pn83r8V2sv

    2023

  130. [139]

    BLOOM +1: Adding language support to BLOOM for zero-shot prompting

    Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, and Vassilina Nikoulina. BLOOM +1: Adding l...

    2023

  131. [140]

    Crosslingual reasoning through test-time scaling

    Zheng-Xin Yong, M Farid Adilazuarda, Jonibek Mansurov, Ruochen Zhang, Niklas Muennighoff, Carsten Eickhoff, Genta Indra Winata, Julia Kreutzer, Stephen H Bach, and Alham Fikri Aji. Crosslingual reasoning through test-time scaling. arXiv preprint arXiv:2505.05408, 2025

    2025 arXiv

  132. [141]

    Code-switching red-teaming: Llm evaluation for safety and multilingual understanding

    Haneul Yoo, Yongjin Yang, and Hwaran Lee. Code-switching red-teaming: Llm evaluation for safety and multilingual understanding. arXiv preprint arXiv:2406.15481, 2024

    2024 arXiv

  133. [142]

    R -judge: Benchmarking safety risk awareness for LLM agents

    Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, and Gongshen Liu. R -judge: Benchmarking safety risk awareness for LLM agents. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds....

    2024 doi

  134. [143]

    How johnny can persuade LLM s to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing LLM s

    Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade LLM s to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing LLM s. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annua...

    2024 doi

  135. [144]

    Toward mitigating misinformation and social media manipulation in llm era

    Yizhou Zhang, Karishma Sharma, Lun Du, and Yan Liu. Toward mitigating misinformation and social media manipulation in llm era. In Companion Proceedings of the ACM Web Conference 2024, WWW '24, pp.\ 1302–1305, New York, NY, USA, 2024 a . Association for Computing Machinery. ISB...

    2024

  136. [145]

    Simulating classroom education with llm-empowered agents

    Zheyuan Zhang, Daniel Zhang-Li, Jifan Yu, Linlu Gong, Jinchang Zhou, Zhanxin Hao, Jianxiao Jiang, Jie Cao, Huiqin Liu, Zhiyuan Liu, et al. Simulating classroom education with llm-empowered agents. arXiv preprint arXiv:2406.19226, 2024 b

    2024 arXiv

  137. [146]

    Mpo: Multilingual safety alignment via reward gap optimization

    Weixiang Zhao, Yulin Hu, Yang Deng, Tongtong Wu, Wenxuan Zhang, Jiahe Guo, An Zhang, Yanyan Zhao, Bing Qin, Tat-Seng Chua, et al. Mpo: Multilingual safety alignment via reward gap optimization. arXiv preprint arXiv:2505.16869, 2025

    2025 arXiv

  138. [147]

    Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization

    Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational...

    2024 doi

  139. [148]

    Do PLM s and annotators share the same gender bias? definition, dataset, and framework of contextualized gender bias

    Shucheng Zhu, Bingjie Du, Jishun Zhao, Ying Liu, and Pengyuan Liu. Do PLM s and annotators share the same gender bias? definition, dataset, and framework of contextualized gender bias. In Agnieszka Fale \'n ska, Christine Basta, Marta Costa-juss \`a , Seraphina Goldfarb-Tarran...

    2024 doi

  140. [149]

    Crosslingual Generalization through Multitask Finetuning

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.