Pith. sign in

REVIEW 2 major objections 5 minor 47 references

IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languages

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning on local safety data cuts unsafe AI outputs in Indonesian

desk verdict IndoSafety fills a real gap in Indonesian safety evaluation, and the dataset/taxonomy are solid contributions, but the GPT-4o-only judge and fine-tuning circularity mean the headline numbers should be treated as provisional. read the letter →

arxiv 2506.02573 v1 pith:JLQK3JGS submitted 2025-06-03 cs.CL

Muhammad Falensi Azmi , Muhammad Dehan Al Kautsar , Alfan Farizki Wicaksono , Fajri Koto This is my paper
classification cs.CL
keywords LLMsafetyculturallygroundedIndonesianlanguagesevaluationdatasetJavaneseSundaneseMinangkabaufine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IndoSafety is a human-verified safety evaluation dataset built for Indonesia's linguistic landscape: formal and colloquial Indonesian plus Javanese, Sundanese, and Minangkabau. The paper argues that translated English safety benchmarks miss what is actually dangerous in this context — ethnic stereotypes, misrepresentation of traditional practices, misinterpretation of Pancasila, regional separatism advocacy, and treating the supernatural as fact — and that this blind spot matters because Indonesia is populous, culturally diverse, and rapidly adopting AI. Evaluating ten LLMs, it finds that Indonesian-centric and regional models produce unsafe responses at substantial rates, worst in colloquial and local-language settings. It then shows that LoRA fine-tuning an open-weight model on the dataset's training split cuts unsafe responses across all risk areas, including Javanese and Sundanese that were not used in tuning, while leaving downstream task performance essentially unchanged. If right, the work makes a concrete case that LLM safety must be evaluated and trained in local cultural terms, not merely translated.

What carries the argument

The carrying mechanism is a fine-grained safety taxonomy: risk areas I–V (discrimination and toxicity, human–chatbot interaction harms, information hazards, malicious uses, misinformation harms) are adopted from the Do-Not-Answer framework, and a new risk area VI, 'region-specific sensitivities', adds eight culturally grounded harm types — ethnicities and cultural practices, historical controversies, Indonesian entities, Pancasila misinterpretation, regional separatism advocacy, religions and beliefs, and supernatural claims. Around this taxonomy the paper builds a three-part dataset: IndoSafety-Eval-1 (2,514 prompts), a parallel test set IndoSafety-Eval-2 (500 prompts in each of five variants), and IndoSafety-Train (2,014 prompt–response pairs with GPT-4o-generated safe answers). Harmfulness is scored by a GPT-4o judge answering per-risk-area binary question sets in Indonesian, adapted from a prior Chinese safeguard evaluation for areas I–V and newly written for area VI. The fine-tuning demonstration uses LoRA on the 8B Sailor2 model for one epoch.

What would settle it

Take a random sample of the Javanese, Sundanese, and Minangkabau response sets, have native-speaker annotators apply the paper's own per-harm question sets, and compare their labels with GPT-4o's; if agreement on these local languages falls to or below the 0.71–0.72 Pearson correlation seen for Indonesian, the reported unsafe rates and the measured fine-tuning improvement would need recomputation.

Watch

Extended reading notes

Core claim

The paper's central claim is that culturally grounded safety for LLMs cannot be supplied by translated English datasets, and that a purpose-built, human-verified resource changes both measurement and behavior. With IndoSafety, the authors introduce the first safety evaluation dataset tailored to the Indonesian context, spanning five language varieties and a 19-category taxonomy. Their measurements show that existing Indonesian-centric LLMs often generate unsafe outputs — Sailor2 at 36.7% unsafe on the first evaluation set and 32–40% across colloquial, Javanese, and Sundanese variants, with region-specific sensitivities among the most frequent failure areas. The intervention result is the load-bearing outcome: fine-tuning Sailor2 with 2,014 safety prompt–response pairs in formal Indonesian reduced unsafe responses in formal and colloquial Indonesian, Javanese, and Sundanese, with all differences significant at $\alpha = 0.05$ by McNemar's test, while 3-shot accuracy on six Indonesian benchmarks dropped by at most a fraction of a point. The authors read this as evidence that safety alignment in a high-resource standardized variant can generalize to related low-resource languages.

Load-bearing premise

Every safety number in the paper — the model rankings and the fine-tuning improvement — comes from a single automatic judge, GPT-4o, whose agreement with humans was checked on only 200 formal and colloquial Indonesian prompts and never on Javanese, Sundanese, or Minangkabau, so its judgment of culturally specific content is assumed rather than shown.

Editorial extensions

If this is right

  • Safety evaluations that rely on translated English prompts will understate risk in Indonesian colloquial and local-language use; a culturally grounded taxonomy is needed to see those failures.
  • The dataset works as both a benchmark (the two evaluation sets) and a training resource (IndoSafety-Train), so it can be reused for measurement and alignment in one package.
  • Safety fine-tuning on formal Indonesian data transfers to related low-resource varieties such as Javanese and Sundanese, suggesting a cheaper alignment path than collecting safety data in every language.
  • Open-weight regional models below 10B parameters can be made substantially safer with a one-epoch LoRA run without meaningful loss on downstream Indonesian benchmarks.
  • Models that look safe in formal Indonesian can still fail on interrogative prompts and on region-specific sensitivity categories, so deployment screening should include those conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because human agreement with the GPT-4o judge was measured only for formal and colloquial Indonesian and fell at a Pearson correlation near 0.71, the reported unsafe rates for Javanese, Sundanese, and Minangkabau rest on an unvalidated judge; a native-speaker recoding of those 1,500 responses would tell whether the ranking of models survives.
  • The fine-tuning gains may partly reflect the evaluator's own preferences rather than an absolute safety improvement; testing the same before/after models with a second judge or with human annotators would separate the two.
  • If the transfer result generalizes, it offers a recipe for other low-resource language clusters: align once in the standardized variety, then evaluate in the dialects — but only after verifying that the evaluator, not just the model, is fluent in those dialects.
  • The taxonomy's normative choices (e.g., supernatural claims must be labeled unproven, separatist advocacy must be refused) invite an annotator-disagreement study, since reasonable native speakers may differ on some category boundaries.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces IndoSafety, a human-verified safety evaluation dataset for Indonesian and three local languages (Javanese, Sundanese, Minangkabau) as well as formal and colloquial Indonesian. It extends the taxonomy of Wang et al. (2024c) with a region-specific risk area (VI) covering 19 harm types, builds a 2,514-prompt evaluation set and a 2,500-prompt parallel test set, and uses the remaining 2,014 prompts as a safety-alignment training set. The authors report unsafe-response rates for ten LLMs across variants (Tables 2 and 10), analyze behavior by risk area and prompt type, and fine-tune Sailor2 with LoRA on IndoSafety-Train, reporting large judged safety improvements (Table 3) with little loss on Indonesian benchmarks (Table 4).

Significance. If the evaluator reliability concerns are resolved, this is a genuinely useful resource: it is the first Indonesian safety evaluation dataset with human-verified prompts across five language varieties, it proposes a culturally grounded taxonomy, and it ships both evaluation and training splits together with a multi-model comparison. The paper is honest in its limitations and follows established evaluation practice, but the central quantitative claims currently rest on a single LLM judge whose agreement with humans is checked only on 200 formal/colloquial prompts. Because the dataset itself is independently human-verified and the taxonomy is well motivated, the contribution does not collapse; it needs stronger validation of the automatic judge and of the fine-tuning effect before the headline numbers can be taken at face value.

major comments (2)
  1. [Section 5.2 and Appendix G] All unsafe-response rates in Tables 2 and 3 and Figure 10 are produced by GPT-4o answering binary evaluation questions (Section 5.2, Figure 8). The only human comparison (Appendix G) covers 100 formal and 100 colloquial prompts and shows that GPT-4o rated as safe 9 of the 30 responses human annotators flagged unsafe (4/17 formal, 5/13 colloquial), a 30% false-negative rate on the unsafe class, and no human check is reported for Javanese, Sundanese, or Minangkabau. These are exactly the variants where the new region-specific category VI is most dependent on cultural nuance. The reported effect sizes, including the regional generalization claim, are therefore not yet established. I recommend a stratified human validation sample for the local-language variants, with particular attention to risk area VI, and agreement metrics appropriate for binary decisions (e.g., Cohen's kappa and class-wise recall), not only Pearson correlation.
  2. [Section 4.2, Section 5.2, Table 3] The fine-tuning demonstration is exposed to a circularity concern: the safe responses in IndoSafety-Train were generated by GPT-4o (Section 4.2, Figure 6), and the harmfulness of the fine-tuned model's outputs is judged by GPT-4o (Section 5.2, Figure 8). The large reductions in Table 3 (e.g., risk area VI from 53/57/60/54 to 5/11/15/15 in colloquial/formal/Javanese/Sundanese) may therefore reflect the judge recognizing GPT-4o-style refusal phrasing rather than a verified reduction in culturally unsafe content. The Limitations section acknowledges the single-judge risk, but an acknowledgment does not validate the effect sizes. I ask for an independent evaluation of a sample of pre- and post-tuning outputs, preferably human-annotated or judged by a model not involved in generating the training targets, with the judge blind to whether each response is from the base or fine-tuned model.
minor comments (5)
  1. [Appendix G] The confusion matrices in Figures 11 and 12 are difficult to read because the row and column labels are ambiguous; please draw them as standard confusion matrices with human annotation on one axis and GPT-4o prediction on the other, and report the cell counts in the caption.
  2. [Table 3] The statement that all differences are significant at alpha = 0.05 under McNemar's test is not accompanied by p-values; given the number of comparisons across variants and risk areas, please report exact p-values or corrected q-values to make the significance claim auditable.
  3. [Section 6.2] The language-identification accuracy of 97% and prompt-type classification accuracy of 96% are reported only as aggregate numbers based on 100 manual samples; a per-language breakdown would be more informative, especially for Minangkabau, where Figure 4 already shows mixed responses.
  4. [Section 4.1.1 and Figure 3] The translation quality, fluency, and relevance scores in Figure 3 are reported without confidence intervals or the number of annotators; please state whether the values are percentages, how ties were resolved, and whether the 100 samples were scored by a single annotator.
  5. [Section 5.1] The exclusion of Cendol and Komodo because they 'performed poorly' is reported only in a footnote; specifying the failure mode (e.g., empty responses or language mismatch) would help readers interpret model coverage.

Circularity Check

1 steps flagged · score 6.0 of 10

Fine-tuning improvement is measured by the same GPT-4o model that generated the training targets, making the reported safety gain partially circular; the dataset itself is human-verified and independent.

  1. other [Section 4.2 (IndoSafety-Train generation), Section 5.2 (automatic evaluation), Limitations]
    "Using GPT-4o, we generated safe responses for each harmful prompt, following a structured prompt template shown in Figure 6. This resulted in 2,014 prompt–response pairs, forming the IndoSafety-Train dataset. ... We used GPT-4o to answer this question using the prompt in Figure 8, and additionally compare GPT-4o predictions with human judgment to ensure validity. ... relying on a single LLM to assess the safety of responses introduces potential biases, particularly in favor of the model being used as the evaluator."

    The fine-tuning result in Section 6.1 relies on a closed loop: GPT-4o generates the gold safe responses in IndoSafety-Train (Section 4.2), and GPT-4o is the judge for whether post-tuning responses are harmful (Section 5.2). Because the model is trained to imitate GPT-4o's refusals, and the evaluator is GPT-4o, the reductions in Table 3 partially measure GPT-4o's self-consistency, not an independent safety property. The human agreement check (Appendix G) covers only 200 formal/colloquial prompts and misses 9 of 30 human-flagged unsafe responses, with no check for local languages; the Limitations acknowledges single-judge bias, but the reported effect sizes are not corrected for it.

full rationale

The core resource, IndoSafety, is independently grounded: prompts are human-verified by native speakers, the taxonomy extends external frameworks (Wang et al., 2024c; Ashraf et al., 2025), and the translation and augmentation steps include manual review. No self-citation chain is load-bearing for the dataset construction. The main circularity is confined to the safety-tuning demonstration: the training responses for IndoSafety-Train are generated by GPT-4o, and the same model is used to evaluate harmfulness before and after fine-tuning. Consequently, the large improvements in Table 3 are partly forced by the model learning to imitate the evaluator's preferred safe style, rather than by an independently validated reduction in culturally unsafe content. The human validation (Appendix G) is too narrow to break this loop for the local languages where cultural nuance matters most, and the paper explicitly acknowledges the single-judge bias in the Limitations. The external benchmark results in Table 4 are independent and not circular, and the dataset publication itself is a legitimate contribution. Overall, the central claim of a high-quality human-verified dataset stands, but the fine-tuning safety claim is partially circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numerical free parameters are fitted in the paper. The claims rest on the cultural taxonomy, the use of GPT-4o as evaluator and training-data generator, and the transfer assumption for safety tuning.

assumptions (4)
  • domain assumption The safety taxonomy from Wang et al. (2024c) is a valid basis for general safety categories I-V.
    The paper adopts these categories without re-validation in Indonesian context; any mismatch could bias the evaluation.
  • domain assumption GPT-4o provides reliable binary safety judgments for Indonesian and local language responses.
    Used as the sole automatic evaluator; human agreement is only measured on 200 formal/colloquial prompts (Pearson r~0.71), not on Javanese/Sundanese/Minangkabau.
  • ad hoc to paper The newly proposed region-specific taxonomy (category VI) captures the important cultural sensitivities of Indonesia.
    Created by the authors via expert discussion and literature review; no independent validation or external benchmark confirms it.
  • domain assumption Safety fine-tuning on formal Indonesian transfers to colloquial and related local languages.
    Inferred from one model (Sailor2) after one LoRA epoch; no cross-model evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languages." pith.science (2026). https://pith.science/paper/JLQK3JGS

@misc{pith2026250602573,
  author       = {Pith},
  title        = {Pith review of: IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JLQK3JGS}},
  note         = {Machine review of arXiv:2506.02573}
}
read the original abstract

Although region-specific large language models (LLMs) are increasingly developed, their safety remains underexplored, particularly in culturally diverse settings like Indonesia, where sensitivity to local norms is essential and highly valued by the community. In this work, we present IndoSafety, the first high-quality, human-verified safety evaluation dataset tailored for the Indonesian context, covering five language varieties: formal and colloquial Indonesian, along with three major local languages: Javanese, Sundanese, and Minangkabau. IndoSafety is constructed by extending prior safety frameworks to develop a taxonomy that captures Indonesia's sociocultural context. We find that existing Indonesian-centric LLMs often generate unsafe outputs, particularly in colloquial and local language settings, while fine-tuning on IndoSafety significantly improves safety while preserving task performance. Our work highlights the critical need for culturally grounded safety evaluation and provides a concrete step toward responsible LLM deployment in multilingual settings. Warning: This paper contains example data that may be offensive, harmful, or biased.

Figures

Figures reproduced from arXiv: 2506.02573 by the authors.

Figure 1
Figure 1. Examples of unsafe responses from SahabatAI, an Indonesian-centric language model. based on direct translations from English (Wang et al., 2023; Song et al., 2025), which overlook key aspects of local communication, such as infor￾mal registers, regional languages, and culturally specific norms. As the fourth most populous country1 and a rapidly expanding hub for AI adoption, Indonesia faces serious risks from deploy… view at source ↗
Figure 2
Figure 2. Overview of the IndoSafety creation pipeline. quality fluency relevance 0 20 40 60 80 100 95 90 72 92 75 90 Do-Not-Answer Dataset from Bianchi, et al [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overall translation quality, fluency, and rele [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Language distribution in model responses across variants. The y-axis indicates the model and the language [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Prompt template for Indonesian-specific dataset augmentation. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Prompt template to generate safe responses for [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Prompt template for prompt classification. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Prompt template for automatic evaluation using GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Prompt template for language identification [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Heatmap of unsafe cases percentage over six risk areas. I = discrimination, exclusion, toxicity, hateful, offensive, II = human-chatbot interaction harms, III = information hazards, IV = malicious uses, V = misinformation harms, VI = region-specific sensitivity. For t…
Figure 11
Figure 11. Figure 11: Confusion matrix for human and automatic evaluation for formal Indonesian variant. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Confusion matrix for human and automatic evaluation for colloquial Indonesian variant. [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Percentage of imperative, interrogative, and declarative prompts across variants. [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Pairwise Venn diagrams of language variants in [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 8 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Yasser Ashraf, Yuxia Wang, Bin Gu, Preslav Nakov, and Timothy Baldwin. 2025. https://aclanthology.org/2025.naacl-long.285/ A rabic dataset for LLM safeguard evaluation . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5...

    work page 2025

  4. [4]

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2024. http://arxiv.org/abs/2309.07875 Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions

    arXiv 2024

  5. [5]

    Samuel Cahyawijaya, Holy Lovenia, Fajri Koto, Rifki Putri, Wawan Cenggoro, Jhonson Lee, Salsabil Akbar, Emmanuel Dave, Nuurshadieq Nuurshadieq, Muhammad Mahendra, Rr Putri, Bryan Wilie, Genta Winata, Alham Aji, Ayu Purwarianti, and Pascale Fung. 2024. https://doi.org/10.18653/v1/2024.acl-long.796 Cendol: Open instruction-tuned generative large language mo...

    doi:10.18653/v1/2024.acl-long.796 2024

  6. [6]

    Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. 2024. https://doi.org/10.18653/v1/2024.acl-long.568 Defending against alignment-breaking attacks via robustly aligned LLM . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10542--10560, Bangkok, Thailand. Association for Computation...

    doi:10.18653/v1/2024.acl-long.568 2024

  7. [7]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  8. [8]

    Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.88 Toxicity in chatgpt: Analyzing persona-assigned language models . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1236--1270, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.88 2023

Show all 47 references
  1. [9]

    Tran, Mike Zhang, Shiqi Chen, Tianyu Pang, Chao Du, Xinyi Wan, Wei Lu, and Min Lin

    Longxu Dou, Qian Liu, Fan Zhou, Changyu Chen, Zili Wang, Ziqi Jin, Zichen Liu, Tongyao Zhu, Cunxiao Du, Penghui Yang, Haonan Wang, Jiaheng Liu, Yongchi Zhao, Xiachong Feng, Xin Mao, Man Tsung Yeung, Kunat Pipatanakul, Fajri Koto, Min Si Thu, Hynek Kydl \' c ek, Zeyi Liu, Qunsh...

    2025 arXiv

  2. [10]

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfi...

    2022 arXiv

  3. [11]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  4. [12]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.301 R eal T oxicity P rompts: Evaluating neural toxic degeneration in language models . In Findings of the Association for Computational Linguist...

    2020 doi

  5. [13]

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. https://doi.org/10.18653/v1/2022.acl-long.234 T oxi G en: A large-scale machine-generated dataset for adversarial and implicit hate speech detection . In Proceedings of the 60th A...

    2022 doi

  6. [14]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. http://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models

    2021 arXiv

  7. [15]

    Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.148 Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2038--2047, Ab...

    2022 doi

  8. [16]

    Devansh Jain, Priyanshu Kumar, Samuel Gehman, Xuhui Zhou, Thomas Hartvigsen, and Maarten Sap. 2024. Polyglotoxicityprompts: Multilingual evaluation of neural toxic degeneration in large language models. In Proceedings of Conference on Language Modeling (COLM 2024), Philadelphia, USA

    2024

  9. [17]

    Fajri Koto. 2025. https://aclanthology.org/2025.naacl-industry.69/ Cracking the code: Multi-domain LLM evaluation on real-world professional exams in I ndonesia . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational ...

    2025

  10. [18]

    Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.760 Large language models only pass primary school exams in I ndonesia: A comprehensive test on I ndo MMLU . In Proceedings of the 2023 Conference on Empirical Methods i...

    2023 doi

  11. [19]

    Fajri Koto, Timothy Baldwin, and Jey Han Lau. 2022. https://doi.org/10.18653/v1/2022.csrr-1.2 Cloze evaluation for deeper understanding of commonsense stories in I ndonesian . In Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022), pages 8...

    2022 doi

  12. [20]

    Fajri Koto, Rahmad Mahendra, Nurul Aisyah, and Timothy Baldwin. 2024. https://doi.org/10.1162/tacl_a_00726 I ndo C ulture: Exploring geographically influenced cultural commonsense reasoning across eleven I ndonesian provinces . Transactions of the Association for Computational...

    2024 doi

  13. [21]

    Sharon Levy, Emily Allaway, Melanie Subbiah, Lydia Chilton, Desmond Patton, Kathleen McKeown, and William Yang Wang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.154 S afe T ext: A benchmark for exploring physical safety in language models . In Proceedings of the 2022 Con...

    2022 doi

  14. [22]

    Tianlong Li, Zhenghua Wang, Wenhao Liu, Muling Wu, Shihan Dou, Changze Lv, Xiaohua Wang, Xiaoqing Zheng, and Xuanjing Huang. 2025. https://aclanthology.org/2025.coling-main.212/ Revisiting jailbreaking for large language models: A representation engineering perspective . In Pr...

    2025

  15. [23]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 T ruthful QA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2022 doi

  16. [24]

    Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. https://doi.org/10.18653/v1/2024.naacl-long.112 Are multilingual LLM s culturally-diverse reasoners? an investigation into multicultural proverbs and sayings . In Proceedings of the 2024 Conference of the North A...

    2024 doi

  17. [25]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Pai...

    2024 arXiv

  18. [26]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  19. [27]

    Louis Owen, Vishesh Tripathi, Abhay Kumar, and Biddwan Ahmed. 2024. http://arxiv.org/abs/2403.09362 Komodo: A linguistic expedition into indonesia's regional languages

    2024 arXiv

  20. [28]

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. https://doi.org/10.18653/v1/2022.findings-acl.165 BBQ : A hand-built bias benchmark for question answering . In Findings of the Association for ...

    2022 doi

  21. [29]

    Jiayang Song, Yuheng Huang, Zhehua Zhou, and Lei Ma. 2025. https://aclanthology.org/2025.findings-naacl.191/ Multilingual blending: Large language model safety alignment evaluation with language mixture . In Findings of the Association for Computational Linguistics: NAACL 2025...

    2025

  22. [30]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  23. [31]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  24. [32]

    Qwen Team. 2025. https://qwenlm.github.io/blog/qwq-32b/ Qwq-32b: Embracing the power of reinforcement learning

    2025

  25. [33]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. http://arxiv.org/abs/2302.13971 Llama: ...

    2023 arXiv

  26. [34]

    Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael Lyu. 2024 a . https://doi.org/10.18653/v1/2024.findings-acl.349 All languages matter: On the multilingual safety of LLM s . In Findings of the Association for Computational Linguist...

    2024 doi

  27. [35]

    Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R Lyu. 2023. All languages matter: On the multilingual safety of large language models. arXiv preprint arXiv:2310.00905

    2023 arXiv

  28. [36]

    Yihan Wang, Zhouxing Shi, Andrew Bai, and Cho-Jui Hsieh. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.948 Defending LLM s against jailbreaking attacks via backtranslation . In Findings of the Association for Computational Linguistics: ACL 2024, pages 16031--16046, Ba...

    2024 doi

  29. [37]

    Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2024 c . https://aclanthology.org/2024.findings-eacl.61/ Do-not-answer: Evaluating safeguards in LLM s . In Findings of the Association for Computational Linguistics: EACL 2024, pages 896--911, St. Julian '...

    2024

  30. [38]

    Yuxia Wang, Zenan Zhai, Haonan Li, Xudong Han, Shom Lin, Zhenxuan Zhang, Angela Zhao, Preslav Nakov, and Timothy Baldwin. 2024 d . https://doi.org/10.18653/v1/2024.findings-acl.184 A C hinese dataset for evaluating the safeguards in large language models . In Findings of the A...

    2024 doi

  31. [39]

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. http://arxiv.org/abs/2307.02483 Jailbroken: How does llm safety training fail?

    2023 arXiv

  32. [40]

    Haryo Wibowo, Erland Fuadi, Made Nityasya, Radityo Eko Prasojo, and Alham Aji. 2024. https://doi.org/10.18653/v1/2024.naacl-long.77 COPAL - ID : I ndonesian language reasoning with local culture and nuances . In Proceedings of the 2024 Conference of the North American Chapter ...

    2024 doi

  33. [41]

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. 2024. https://doi.org/10.18653/v1/2024.acl-long.303 S afe D ecoding: Defending against jailbreak attacks via safety-aware decoding . In Proceedings of the 62nd Annual Meeting of the As...

    2024 doi

  34. [42]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conferenc...

    2021 doi

  35. [43]

    Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. http://arxiv.org/abs/2304.05302 Rrhf: Rank responses to align language models with human feedback without tears

    2023 arXiv

  36. [44]

    Wenxuan Zhang, Hou Pong Chan, Yiran Zhao, Mahani Aljunied, Jianyu Wang, Chaoqun Liu, Yue Deng, Zhiqiang Hu, Weiwen Xu, Yew Ken Chia, Xin Li, and Lidong Bing. 2024 a . http://arxiv.org/abs/2407.19672 Seallms 3: Open foundation and chat multilingual large language models for sou...

    2024 arXiv

  37. [45]

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.830 S afety B ench: Evaluating the safety of large language models . In Proceedings of the 62nd Annual ...

    2024 doi

  38. [46]

    Zhexin Zhang, Junxiao Yang, Pei Ke, Fei Mi, Hongning Wang, and Minlie Huang. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.481 Defending large language models against jailbreaking attacks through goal prioritization . In Proceedings of the 62nd Annual Meeting of the Assoc...

    2024 doi

  39. [47]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. http://arxiv.org/abs/2307.15043 Universal and transferable adversarial attacks on aligned language models

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.