Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

KoBALT: Korean Benchmark For Advanced Linguistic Tasks

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read KoBALT, a 700-question linguist-curated benchmark, tops out at 61% accuracy for the best model and shows a strong correlation with native-speaker preferences.

desk verdict A genuinely useful Korean linguistic benchmark with a careful model evaluation, but the headline human-preference validation is confounded because annotators were shown the correct answer before judging responses. read the letter →

arxiv 2505.16125 v1 pith:FXPOCMDO submitted 2025-05-22 cs.CL

Hyopil Shin , Sangah Lee , Dongjun Jang , Wooseok Song , Jaeyoon Kim , Chaeyoung Oh , Hyemi Jo , Youngchae Ahn
show 4 more authors
Sihyun Oh Hyohyeong Chang Sunkyoung Kim Jinsik Lee
This is my paper
classification cs.CL
keywords KoreanbenchmarklinguisticcompetenceLLMevaluationdatacontaminationhumanpreferencephonologymorphologyten-choicequestions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

KoBALT is a 700-question multiple-choice benchmark built by Korean linguists to test whether large language models actually understand Korean grammar, meaning, and usage rather than simply handling translated tasks. The paper's central claim is that the benchmark is a valid, discriminative measure of Korean linguistic competence: a model's KoBALT accuracy predicts how much native Korean speakers prefer its answers, with a correlation of $r=0.638$ for the top model. The paper also claims the questions have minimal n-gram overlap with standard Korean corpora, reducing the chance that high scores come from memorized benchmark text. On 20 current models the best accuracy is 61%, with a consistent pattern of strength in semantics and weakness in phonology and morphology.

What carries the argument

The load-bearing object is the KoBALT dataset itself, built by seven linguists with a ten-choice format, diverse question types (cloze, grammaticality judgment, minimal pairs, context interpretation), and difficulty levels assigned from four strong models' performance. The validation machinery is a pairwise preference collection from 95 annotators, aggregated by a Bradley-Terry model, whose scores are then correlated with phenomenon-level accuracy.

What would settle it

Give any model the full KoBALT item set in its pretraining corpus (or check overlap against each model's true training data) and show that KoBALT accuracy jumps while performance on a held-out paraphrase of the same phenomena does not; that would show the score reflects memorization rather than linguistic competence.

Watch

Extended reading notes

Core claim

The paper claims that KoBALT separates models by genuine Korean linguistic competence. Using 700 expert-crafted ten-choice questions spanning syntax, semantics, pragmatics, phonetics/phonology, and morphology, it finds that no tested model clears 61% overall accuracy, that performance drops steadily from semantics (66%) to phonology (31%) and morphology (36%), and that these accuracy scores track human preference judgments from 95 native-speaker annotators, with Pearson correlations of $r=0.638$, $0.632$, $0.570$, and $0.570$ across the four models used in the human evaluation.

Load-bearing premise

The claim that KoBALT resists data contamination rests on low n-gram overlap with standard Korean corpora, but the paper never checks overlap with the actual training data of the models it evaluates, so a model could still have memorized the questions through other routes.

Editorial extensions

If this is right

  • KoBALT accuracy can stand in for native-speaker preference when ranking Korean language capability of LLMs.
  • Models' biggest deficits are concentrated in phonology and morphology, suggesting these domains deserve more training data and evaluation attention.
  • The difficulty-stratified items give future models a stable yardstick: a model that passes 61% can be said to outperform the current frontier on Korean linguistics.
  • Low n-gram overlap with standard corpora makes KoBALT scores harder to dismiss as memorization of the benchmark.
  • The construction methodology provides a template for linguistically rigorous benchmarks in other morphologically rich languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The contamination argument is only as strong as the corpus comparison: checking KoBALT items against each model's actual pretraining data would make the memorization defense conclusive rather than presumptive.
  • A correlation near 0.6, while non-trivial, leaves much preference variance unexplained; the paper's own qualitative analysis suggests style and tone dominate when both answers are correct, which implies the benchmark may under-represent pragmatic quality.
  • The domain ordering (semantics strongest, phonology weakest) could be tested in other languages to see whether it reflects a general property of text-pretrained models rather than a quirk of Korean.
  • Because only four models were used in the human evaluation, extending the preference study to the full 20-model set would clarify whether the accuracy-preference correlation holds for weaker models as well.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces KoBALT, a set of 700 original, linguist-crafted Korean multiple-choice questions spanning 24 phenomena in syntax, semantics, pragmatics, phonetics/phonology, and morphology. The authors evaluate 20 open and closed LLMs, report per-domain and per-phenomenon accuracies, assign post hoc difficulty levels, and report low n-gram overlap with standard Korean corpora as evidence against contamination. To validate the benchmark, they conduct a Chatbot-Arena-style human preference study with 95 annotators on a 99-question subset, aggregate 51,528 pairwise votes with a Bradley-Terry model, and report a phenomenon-level correlation of r=0.638 between KoBALT accuracy and human preference. The central claim is that KoBALT is a valid discriminative measure of Korean linguistic competence in LLMs.

Significance. If the validity claim held, KoBALT would be a useful resource: it addresses a genuine gap in linguistically grounded Korean evaluation, uses expert curation rather than translated benchmarks, and its 10-choice format reduces chance-level scoring. The accuracy results across 20 models, especially the consistent weakness in phonology and morphology, are concrete and potentially informative. The paper also deserves credit for reporting domain-level breakdowns and for being candid in its Limitations section about the model-based difficulty levels and the small number of models in the human study. However, the main validation evidence is undermined by a confound in the human preference protocol, and the contamination-mitigation claim is supported only by a weak proxy. The resource may still be valuable, but the paper's strongest claim is not empirically established in its current form.

major comments (3)
  1. [Section 6, Figure 6; Tables 10-12] The human-preference validation is confounded because the instruction shown to annotators explicitly provides the 'Correct answer choice' before they are asked which response they prefer. Table 12 shows that when exactly one model is correct, the correct response is preferred 82.0-92.1% of the time, which is exactly the pattern expected if raters were verifying the gold answer rather than expressing holistic preference. The Abstract and Section 7 claim that the r=0.638 correlation validates KoBALT as a discriminative measure of Korean language understanding, but this correlation is at least partly mechanical. The authors should either rerun the preference study without revealing the correct answer, or demonstrate that the correlation persists after excluding or statistically controlling for comparison pairs in which one response is correct and the other is not.
  2. [Section 1 and Section 4] The claim that KoBALT 'substantially mitigates the risk of data contamination' is based only on n-gram overlap with 'standard Korean corpora' (bigrams <8.6%, trigrams <0.7%). This does not test overlap with the actual pretraining data of the 20 evaluated models, so it cannot support the abstract's assertion. The authors should either check overlap against the relevant public pretraining corpora or model-specific data sources, or soften the claim to state that only overlap with the named public corpora was measured. As written, a model could still have memorized these questions through other routes.
  3. [Section 6.1, Tables 9-10] The correlation analysis is incompletely specified. Table 10 reports phenomenon-level Pearson correlations with n=24, but only domain-level Bradley-Terry scores are shown in Table 9; the per-phenomenon BT scores and the exact pairing procedure are not described. No confidence intervals are reported, and the p-values treat 24 phenomena as independent observations even though votes are clustered by annotator and by question. The authors should report the underlying per-phenomenon scores, confidence intervals, and either a bootstrap or multilevel analysis to support the claimed correlation.
minor comments (6)
  1. [Section 3.1 and Limitations] The difficulty levels are defined post hoc from four high-performing models, as the authors acknowledge in Limitations. Since these levels are used to stratify the human-evaluation sample, the paper should state how sensitive the sampling and the resulting preference scores are to the choice of difficulty model.
  2. [Section 6] No inter-annotator agreement statistic is reported for the 95 annotators. For a benchmark whose validity rests partly on human judgments, at least a summary of rater agreement (e.g., pairwise agreement or Krippendorff's alpha) should be provided.
  3. [Section 6, Figure 6 caption] The caption of Figure 6 says 'Example prompt used for model response generation,' but the figure actually shows the instruction given to human participants in the preference collection; the caption should be corrected.
  4. [Section 6.1] The handling of ties is described only as 'a no difference vote awarded 0.5 point to each side.' The paper should specify how ties were entered into the Bradley-Terry likelihood and whether the reported rankings are robust to alternative tie treatments.
  5. [Section 8 and general] The Ethics Statement says the dataset is openly available, but no URL or repository link is provided, and no code for reproducing the evaluation is included. The paper should point to the actual release location.
  6. [General] There are several typographical and formatting issues: 'ModelFor open-source models' in Section 4.1, 'dealth' in Section 6.1, an empty Section 5.1 heading followed by Section 5.2, and inconsistent capitalization of phenomenon names in tables. These should be cleaned up in revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: KoBALT's accuracy scores, difficulty stratification, and human-preference correlation do not reduce by construction to the benchmark's own labels.

full rationale

KoBALT is a new benchmark, and the paper's central quantities are model accuracies computed against expert gold labels. The difficulty levels in Section 3.1 are explicitly post hoc ('we classified our benchmark items into three difficulty levels in a post hoc manner based on the performance patterns of leading models'), and they are used only for descriptive stratification and sampling, not for computing accuracy or preference scores. The human-preference study is an empirical correlation, not a derivation: BT scores are estimated from pairwise votes, and the reported r=0.638 is a measured statistic. The strongest concern is a validity confound, not circularity: Section 6's instruction provides annotators with the 'Correct answer choice' before they state a preference, so the human-preference criterion is not fully independent of KoBALT's gold labels; Table 12 confirms that correct responses win 82-92% of one-correct pairs. This weakens the ecological-validity claim, but both variables are still separately measured, so the result is not equivalent to its input by construction. The self-citation to KLUE (whose author list overlaps via S. Kim) is only used as a related-work critique and is not load-bearing. Overall, the benchmark construction, model evaluation, and accuracy reporting are self-contained and do not exhibit circular derivation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper does not rest on mathematical derivations; its central evidence is a newly created dataset and an empirical evaluation. The main 'unpaid' inputs are the taxonomic choices, the contamination proxy, and the validity assumption behind the human-preference test.

assumptions (4)
  • domain assumption Korean linguistic competence is adequately represented by 24 phenomena across five domains (syntax, semantics, pragmatics, phonetics/phonology, morphology)
    The paper defines the benchmark structure from educational curricula and theoretical considerations (Section 3), but does not provide external evidence that this taxonomy is complete or balanced for measuring 'true language understanding'.
  • domain assumption Low n-gram overlap with 'standard Korean corpora' implies low data contamination
    Used in the Abstract and Section 1 to argue the benchmark mitigates contamination; the authors do not compare against actual training data of the 20 evaluated models.
  • domain assumption Human preference judgments, given the correct answer choice, are a valid proxy for linguistic competence
    The human-preference study (Section 6) provides raters with the correct answer and asks which model response they prefer; the paper assumes this measures quality relevant to 'true language understanding'.
  • standard math Bradley-Terry model assumptions
    Used to aggregate pairwise votes in Section 6.1; assumes no order effects and stable preferences, which are plausible but not tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KoBALT: Korean Benchmark For Advanced Linguistic Tasks." pith.science (2026). https://pith.science/paper/FXPOCMDO

@misc{pith2026250516125,
  author       = {Pith},
  title        = {Pith review of: KoBALT: Korean Benchmark For Advanced Linguistic Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FXPOCMDO}},
  note         = {Machine review of arXiv:2505.16125}
}
read the original abstract

We introduce KoBALT (Korean Benchmark for Advanced Linguistic Tasks), a comprehensive linguistically-motivated benchmark comprising 700 multiple-choice questions spanning 24 phenomena across five linguistic domains: syntax, semantics, pragmatics, phonetics/phonology, and morphology. KoBALT is designed to advance the evaluation of large language models (LLMs) in Korean, a morphologically rich language, by addressing the limitations of conventional benchmarks that often lack linguistic depth and typological grounding. It introduces a suite of expert-curated, linguistically motivated questions with minimal n-gram overlap with standard Korean corpora, substantially mitigating the risk of data contamination and allowing a more robust assessment of true language understanding. Our evaluation of 20 contemporary LLMs reveals significant performance disparities, with the highest-performing model achieving 61\% general accuracy but showing substantial variation across linguistic domains - from stronger performance in semantics (66\%) to considerable weaknesses in phonology (31\%) and morphology (36\%). Through human preference evaluation with 95 annotators, we demonstrate a strong correlation between KoBALT scores and human judgments, validating our benchmark's effectiveness as a discriminative measure of Korean language understanding. KoBALT addresses critical gaps in linguistic evaluation for typologically diverse languages and provides a robust framework for assessing genuine linguistic competence in Korean language models.

Figures

Figures reproduced from arXiv: 2505.16125 by the authors.

Figure 1
Figure 1. Distribution of samples across linguistic domains in KoBALT [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Example prompt used for model response generation. Models were instructed to select from ten possible [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Accuracy per each linguistic phenomena [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Human preference evaluation process utilizing the Online Live Chatbot Area concept [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Screenshot of the full instruction given to participants in human preference collection stage. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Example prompt used for model response generation. Models were instructed to select from ten possible [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HanjaBridge: Resolving Semantic Ambiguity in Korean LLMs via Hanja-Augmented Pre-Training

    cs.CL 2025-07 conditional novelty 5.0 of 10

    HanjaBridge, a continual pre-training method that appends all candidate Hanja forms for Korean homophones, improves KoBALT scores by 21 percent relative while keeping English performance mostly intact.

Reference graph

Works this paper leans on

37 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. URL https://arxiv.org/abs/1909.08053

    arXiv 2020

  2. [2]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner...

    arXiv 2024

  3. [3]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  5. [5]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8: 0 377--392, 2019. URL https://api.semanticscholar.org/CorpusID:208527435

    work page 2019

  6. [6]

    Holmes Recorder a benchmark to assess the linguistic competence of language models

    Andreas Waldis, Yotam Perlitz, Leshem Choshen, Yufang Hou, and Iryna Gurevych. Holmes Recorder a benchmark to assess the linguistic competence of language models. Transactions of the Association for Computational Linguistics, 12: 0 1616--1647, 2024. doi:10.1162/tacl_a_00718. URL https://aclanthology.org/2024.tacl-1.88/

    doi:10.1162/tacl_a_00718 2024

  7. [7]

    Syntaxgym: An online platform for targeted evaluation of language models

    Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. Syntaxgym: An online platform for targeted evaluation of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70--76, 2020

    work page 2020

  8. [8]

    Costa-jussà

    Eduardo Sánchez, Belen Alastruey, Christophe Ropers, Pontus Stenetorp, Mikel Artetxe, and Marta R. Costa-jussà. Linguini: A benchmark for language-agnostic linguistic reasoning, 2024. URL https://arxiv.org/abs/2409.12126

    arXiv 2024

Show all 37 references
  1. [9]

    Iolbench: Benchmarking llms on linguistic reasoning, 2025

    Satyam Goyal and Soham Dan. Iolbench: Benchmarking llms on linguistic reasoning, 2025. URL https://arxiv.org/abs/2501.04249

    2025

  2. [10]

    Bean, Simi Hellsten, Harry Mayne, Jabez Magomere, Ethan A

    Andrew M. Bean, Simi Hellsten, Harry Mayne, Jabez Magomere, Ethan A. Chi, Ryan Chi, Scott A. Hale, and Hannah Rose Kirk. Lingoly: A benchmark of olympiad-level linguistic reasoning puzzles in low-resource and extinct languages. ArXiv, abs/2406.06196, 2024. URL https://api.sema...

    2024 arXiv

  3. [11]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP@EMNLP, 2018. URL https://api.semanticscholar.org/CorpusID:5034059

    2018

  4. [12]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. ArXiv, abs/1905.00537, 2019. URL https://api.semanticscholar.org/Corp...

    1905 arXiv

  5. [13]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/2009.03300, 2020. URL https://api.semanticscholar.org/CorpusID:221516475

    2009 arXiv

  6. [14]

    Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, ...

    2022 arXiv

  7. [15]

    Pub: A pragmatics understanding benchmark for assessing llms’ pragmatics capabilities

    Settaluri Lakshmi Sravanthi, Meet Doshi, Tankala Pavan Kalyan, Rudra Murthy, Pushpak Bhattacharyya, and Raj Dabre. Pub: A pragmatics understanding benchmark for assessing llms’ pragmatics capabilities. Findings of the Association for Computational Linguistics ACL 2024, 2024. U...

    2024

  8. [16]

    Multiprageval: Multilingual pragmatic evaluation of large language models

    Dojun Park, Jiwoo Lee, Seohyun Park, Hyeyun Jeong, Youngeun Koo, Soonha Hwang, Seonwoo Park, and Sungeun Lee. Multiprageval: Multilingual pragmatic evaluation of large language models. ArXiv, abs/2406.07736, 2024 a . URL https://api.semanticscholar.org/CorpusID:270392017

    2024 arXiv

  9. [17]

    Phonologybench: Evaluating phonological skills of large language models

    Ashima Suvarna, Harshita Khandelwal, and Nanyun Peng. Phonologybench: Evaluating phonological skills of large language models. ArXiv, abs/2404.02456, 2024. URL https://api.semanticscholar.org/CorpusID:268876168

    2024 arXiv

  10. [18]

    Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Tae Hwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Young kuk Jeong, I. V. Lee, Sang gyu Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungw...

    2021 arXiv

  11. [19]

    Open ko-llm leaderboard: Evaluating large language models in korean with ko-h5 benchmark

    Chanjun Park, Hyeonwoo Kim, Dahyun Kim, Seonghwan Cho, Sanghoon Kim, Sukyung Lee, Yungi Kim, and Hwalsuk Lee. Open ko-llm leaderboard: Evaluating large language models in korean with ko-h5 benchmark. ArXiv, abs/2405.20574, 2024 b . URL https://api.semanticscholar.org/CorpusID:...

    2024 arXiv

  12. [20]

    K o BEST : K orean balanced evaluation of significant tasks

    Myeongjun Jang, Dohyung Kim, Deuk Sin Kwon, and Eric Davis. K o BEST : K orean balanced evaluation of significant tasks. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao ...

    2022

  13. [21]

    HAE - RAE bench: Evaluation of K orean knowledge in language models

    Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jae cheol Lee, Je Won Yeom, Jihyu Jung, Jung woo Kim, and Songseong Kim. HAE - RAE bench: Evaluation of K orean knowledge in language models. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti,...

    2024

  14. [22]

    CLI c K : A benchmark dataset of cultural and linguistic intelligence in K orean

    Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. CLI c K : A benchmark dataset of cultural and linguistic intelligence in K orean. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proc...

    2024

  15. [23]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  16. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  17. [25]

    Mistral small 3.1, 2025

    Mistral AI. Mistral small 3.1, 2025. URL https://mistral.ai/news/mistral-small-3-1. Accessed: 2025-05-01

    2025

  18. [26]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  19. [27]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, ...

    2025 arXiv

  20. [28]

    Qwen2.5 technical report

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, ...

    2024 arXiv

  21. [29]

    Aya expanse: Combining research breakthroughs for a new multilingual frontier, 2024

    John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztaj...

    2024 arXiv

  22. [30]

    Claude 3.5 sonnet, 2024

    Anthropic. Claude 3.5 sonnet, 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet. Accessed: 2025-04-01

    2024

  23. [31]

    Claude 3.7 sonnet, 2025

    Anthropic. Claude 3.7 sonnet, 2025. URL https://www.anthropic.com/news/claude-3-7-sonnet. Accessed: 2025-05-01

    2025

  24. [32]

    Team Cohere, Aakanksha, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Raphaël Avalos, Zahara Aviv, Sammie Bae, Saurabh Baji, Alexandre Barbet, Max Bartolo, Björn Bebensee, Neeral Beladia, W...

    2025 arXiv

  25. [33]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2025 arXiv

  26. [34]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024. URL https://arxiv.org/a...

    2024 arXiv

  27. [35]

    Dissecting human and LLM preferences

    Junlong Li, Fan Zhou, Shichao Sun, Yikai Zhang, Hai Zhao, and Pengfei Liu. Dissecting human and LLM preferences. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

    2024 doi

  28. [36]

    Uncovering factor level preferences to improve human-model alignment, 2024

    Juhyun Oh, Eunsu Kim, Jiseon Kim, Wenda Xu, Inha Cha, William Yang Wang, and Alice Oh. Uncovering factor level preferences to improve human-model alignment, 2024. URL https://arxiv.org/abs/2410.06965

    2024

  29. [37]

    Human feedback is not gold standard, 2024

    Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback is not gold standard, 2024. URL https://arxiv.org/abs/2309.16349

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.