Pith. sign in

REVIEW 3 major objections 5 minor 46 references

Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Commercial LLMs outscore open-weight rivals in all 24 EU languages

desk verdict First systematic multi-turn EU-24 LLM benchmark; the coverage and cost analysis are genuinely useful, but the universal 'every language' claim rests on single-run measurements and should be revised or re-run. read the letter →

arxiv 2608.01395 v1 pith:QGUIOKH3 submitted 2026-08-02 cs.CL cs.AI

Sherzod Hakimov , Karl Osswald , Jelle Psurek , Eszter Bukovszky , A. Altar Lüser , David Schlangen This is my paper · ORCID
classification cs.CLcs.AI
keywords multilingualevaluationdialoguegamesEUlanguagesopen-weightvscommercialLLMstokenizercostlanguageequalityclemscorelow-resource
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that, under an interactive multi-turn test, no open-weight large language model currently serves all 24 official EU languages adequately. It reports that two commercial systems outscore every one of the seven open-weight models in all 24 official languages, by margins from 5.6 points in Greek to 35.8 in Irish. It also claims that equal access is not equal service: pooled over models and languages, the median non-English language costs 31% more to run than English and scores 10% lower. The evaluation method makes full institutional coverage cheap, because adding a language means localising a fixed set of game files rather than writing reference answers. The reason to care is that EU law promises citizens service in their own language, and the paper's numbers test whether current AI systems can deliver that.

What carries the argument

The load-bearing object is a dialogue-game evaluation harness built on clembench: 14 goal-directed games (Taboo, Codenames, Wordle, Reference Game, Image Game, and others) played in self-play, with each episode scored by $\%\mathrm{Played}$ (share of episodes completed without aborting) and $\mathrm{Quality}$ (task score over completed episodes), combined into $\mathrm{clemscore} = \%\mathrm{Played} \times \mathrm{Quality}$ scaled to $[0,100]$. Because the game mechanics are language-agnostic, a new language is added by localising prompt text, response-parsing rules, feedback messages and word lists; the paper does this with a two-stage machine-localisation pipeline in which one non-evaluate

What would settle it

Rerun the nine models across the EU-24 with fixed seeds and multiple repetitions; if any commercial model ever falls below an open-weight model in any official language, or if the Greek margin (5.6 points) vanishes, the universal ordering claim is falsified. Separately, have native speakers audit all 30 localised game packs; if the near-zero-completion cells turn out to be parsing failures, the benchmark overstates some language gaps.

Watch

Extended reading notes

Core claim

Across 30 languages and 14 dialogue games, both commercial systems—GPT-5.4 and Claude Opus 4.8—score above every open-weight model in all 24 official EU languages; the margins run from 5.6 points in Greek to 35.8 in Irish. The open-weight deficit is concentrated: for all seven open models, the weakest EU language is Irish or Maltese, and the commercial systems keep 80.7% and 83.0% of their English score in their weakest EU language. Open-weight scores track public web-text volume ($\rho=0.72$, $p<0.001$) and a language's economic footprint (up to $\rho=0.78$); commercial models show no such correlation. Pooled, the median non-English language costs 31% more than English and scores 10% lower,

Load-bearing premise

The claim rests on treating each language's single-run clemscore as a stable measurement: the paper runs every language once, at temperature 1 with no fixed seed, and itself warns that small differences between adjacent cells should not be read as meaningful, so a 5.6-point margin may sit within run-to-run noise.

Editorial extensions

If this is right

  • Any EU-wide deployment today cannot get full EU-24 coverage from an evaluated open-weight model; the realistic options are commercial APIs or building language-specific resources.
  • Linguistic parity is achievable, not a pipe dream: the commercial models score about as well on Maltese, Estonian and Latvian as on English, so the gap reflects under-provisioning rather than intrinsic difficulty.
  • Public web crawls alone cannot close the gap: open-weight performance tracks crawled text volume, so parity requires resources beyond the open web, such as public broadcast archives.
  • Tokenisers charge a hidden price before inference: the median non-English language costs 31% more to run than English while scoring 10% lower, and low-resource EU languages use roughly twice the tokens per word.
  • The two commercial systems deliver markedly less value-per-dollar outside English, so a uniform per-token price does not buy equal service across languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct stress test would be to rerun all nine models over the EU-24 with fixed seeds; the paper's single-run protocol means the 5.6-point Greek margin is the point where the universal ordering claim is most exposed.
  • If the open-weight gap is a training-data effect, then publicly funded non-web corpora (broadcast archives, parliamentary records, local-government text) could narrow it; the paper gestures at this policy conclusion but does not test it.
  • The near-zero completion cells the paper attributes to parsing, rather than competence, imply the benchmark may underestimate ability in some language–game pairs; a native-speaker audit of all 30 localised game packs would settle this.
  • The strong LEF correlation for open-weight models suggests market incentives alone will not serve the smallest official languages, which is the implicit case for public intervention the paper leaves to the reader.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a multilingual extension of the clembench dialogue-game evaluation framework to 30 languages (the 24 official EU languages plus six others), and evaluates nine LLMs (two commercial, seven open-weight) as self-playing agents in 14 goal-directed games. Scores are computed programmatically from rule compliance and task success, without reference answers. The main empirical claims are that (i) in every EU-24 language both commercial models outscore every open-weight model; (ii) no open-weight model covers the EU-24 well; (iii) performance correlates with web-text availability and with a newly defined Linguistic Economic Footprint for open-weight but not commercial models; and (iv) non-English languages cost more and score lower than English, with a median cost premium of 31% and a score deficit of 10%.

Significance. If the central findings hold, this is a valuable contribution: it provides a systematic, multi-turn, programmatically scored benchmark covering the full EU-24 language set, with cost and tokenizer analysis, and it has direct policy relevance for the EU's language-equality commitments. The paper's strengths include its transparent localisation pipeline, public code and leaderboard, and the absence of fitted parameters or reference-answer dependency. The comparison with external benchmarks is descriptive rather than circular. However, the strongest universal claim—that both commercial models beat every open-weight model in every EU language—rests on single-run measurements with no variance estimates, and the tightest margin is small enough that the authors themselves caution against interpreting such differences. The cost-premium headline is similarly derived from single runs and list prices. These issues do not undermine the overall pattern, but they do require either additional evidence or a more qualified statement.

major comments (3)
  1. [§5.1, Table 4; Limitations] The abstract and §5.1 state that in every EU-24 language both commercial models outscore every open-weight model. The tightest margin is Greek: Claude Opus 4.8 at 75.5 vs. GLM-5.2 at 69.9, a 5.6-point gap. The Limitations explicitly state that 'each language is run once, at temperature 1 and without a fixed seed, so we report no variance estimate and small differences between adjacent cells should not be read as meaningful.' A 5.6-point difference is exactly the kind of small difference the authors tell readers not to interpret, and the per-game tables (Tables 11–15) show swings of 10–30 points, so run-to-run variance at temperature 1 could plausibly exceed this margin. The universal quantifier is therefore not supported by the reported measurements. I ask for repeated runs or per-episode bootstrap/CI estimates for at least the tightest cells, or for rewriting the claim as the observed r
  2. [§3.3, §5.1, Limitations] The benchmark's portability claim depends on localised game files being equally playable in all 30 languages, but only six languages were manually verified by native speakers. The Limitations acknowledge 'a small number of game–language pairs still show near-zero completion for reasons we attribute to parsing rather than to competence.' Several unverified EU languages are exactly the ones where open-weight models collapse (Irish, Maltese, Latvian), so a localisation or parsing artifact could depress scores and materially affect the coverage conclusion. Please report which pairs are affected, how the parsing attribution was established, and whether the main EU-24 conclusions survive when those cells are removed or corrected.
  3. [§5.3, Table 8] The headline cost claim ('the median non-English language costs 31% more to run than English, and scores 10% lower') is computed from single runs per language–model cell and from listed API prices. Token counts are objective, but generation at temperature 1 is stochastic, and the same variance caveat applies. Please provide a measure of run-to-run or episode-level variability for the cost ratios, or present the 31% and 10% figures as rough central tendencies rather than precise estimates.
minor comments (5)
  1. [§4.4] The definition of clemscore as a 'normalised product' of %Played and Quality should be made explicit with a formula, including how the scaling to [0,100] is applied.
  2. [Figure 8 / Appendix H] FineWeb-2 is referenced only by footnote URL; a full citation should be added to the reference list, as is done for other datasets.
  3. [§5.2] The text says tokeniser profiles are 'near-identical across providers' and then immediately reports that Claude Opus 4.8 averages roughly 50% more tokens per word than the median. Please reconcile these statements or clarify that Claude is the exception.
  4. [Appendix B] The two-to-six point inflation of Chinese scores due to dropping Wordle is reported in both the main text and Appendix B; consider consolidating to avoid redundancy.
  5. [Table 16] The caption lists sources for speaker shares and GDP only indirectly via Appendix G; state the specific data versions and access dates in the caption or immediately below the table.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the central claims are direct programmatic observations and self-citations are not load-bearing.

full rationale

The paper's derivation chain is self-contained. clemscore is defined as the product of %Played and Quality, both computed by rule-based game masters, so the headline result that both commercial models outscore every open-weight model in every EU-24 language is a direct reading of Table 4 rather than the output of fitted parameters. The localization pipeline uses models outside the evaluated set (GPT-5.2 and Claude Sonnet 4.5, whereas the evaluated models are GPT-5.4 and Claude Opus 4.8) and six languages were manually verified, so any localization bias is a validity concern, not circularity. The self-citations to clembench and Schlangen et al. justify the evaluation paradigm but are not load-bearing for the specific EU-24 empirical claim, which is newly measured here. The comparison with external benchmarks is explicitly descriptive (n=5, vendor-reported, τ values) and does not feed back into the scores. The limitations about single runs, temperature=1, and absence of variance estimates are measurement-uncertainty issues; they weaken the universal quantifier evidentially but do not make the observations equivalent to their inputs. Score 1 reflects only minor, non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The benchmark has no fitted parameters; all scores are measured outcomes. The main assumptions are about the validity of the game-based paradigm, the adequacy of machine-localized materials, the comparability of games across languages, the sufficiency of public crawls as a data proxy, and the accuracy of API pricing. LEF is an introduced metric, though it is a definition rather than a physical entity.

assumptions (5)
  • domain assumption Dialogue game self-play with programmatic scoring is a valid proxy for interactive language use.
    The entire benchmark rests on this, stated in Section 3.1, citing prior clembench work.
  • domain assumption Machine translation with cross-provider validation yields playable game files in all 30 languages.
    Section 3.2; only 6 of 30 languages were verified by native speakers, and the paper admits some near-zero completions attributed to parsing.
  • domain assumption The 14 games cover multiple capabilities and are language-agnostic.
    Section 3.1, Table 2; Wordle is dropped for Chinese, so Chinese scores are not strictly comparable to other languages.
  • domain assumption Public web crawled text (HPLT, FineWeb) is the relevant training-data proxy for open-weight models.
    Section 5.2 and Appendix H; the correlation between crawled volume and open-weight performance is interpreted causally.
  • domain assumption API list prices reflect true cost for the cost analysis.
    Appendix E; Apertus pricing is proxied by Qwen-3-8B, and several models use OpenRouter list prices rather than actual bills.
invented entities (1)
  • Linguistic Economic Footprint (LEF)
    purpose: Weights language importance by speaker-weighted GDP to correlate model performance with economic demand.
    A new composite index defined in Section 5.2 and Appendix G. It is a definition built from World Bank and Ethnologue data, not a falsifiable entity; its specific formulation affects the correlation analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+." pith.science (2026). https://pith.science/paper/QGUIOKH3

@misc{pith2026260801395,
  author       = {Pith},
  title        = {Pith review of: Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QGUIOKH3}},
  note         = {Machine review of arXiv:2608.01395}
}
read the original abstract

We evaluate large language models (LLMs) as language agents playing goal-directed dialogue games in self-play across 30 languages: the 24 official EU languages plus six others. Unlike static or preference-based evaluation, this paradigm is multi-turn, reference-free and programmatically scored, and because the game mechanics are language-agnostic it extends to a new language by localising a fixed set of prompt and word-list files. Evaluating nine open-weight and commercial LLMs, we find that no open-weight model covers the EU-24 well: in every official language both commercial systems outscore every open-weight model, and the two weakest average below 40 points across the EU-24. The commercial systems stay ahead even in languages with four orders of magnitude less public web text, showing that linguistic parity is achievable, but not from public crawls alone. A model's home region lifts it without closing the gap: Chinese is the strongest of all 30 languages for two Chinese-developed models, yet the best Chinese score of any model belongs to a US commercial system. Coverage is also not parity of service. Pooled over models and languages, the median non-English language costs 31% more to run than English, and scores 10% lower.

Figures

Figures reproduced from arXiv: 2608.01395 by the authors.

Figure 1
Figure 1. Overview of the multilingual benchmark construction and score aggregation across evaluated LLMs. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Model performance averages across languages [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. LEF (bars, left axis, log scale) against clemscore (lines, right axis) per language, grouped by family. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Token fertility per model and language. Chinese is character-based and not directly comparable. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Value—clemscore points per dollar—normalised against the same model’s English value: 1.0 matches [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Ranking differences between clemscore (English, left column of each panel) and external benchmarks [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Spearman correlation between each model’s clemscore and LEF across the 30 benchmark languages. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Web data availability per language (billions of words, log scale) in HPLT v3 (blue) and FineWeb-2 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 20 canonical work pages

  1. [1]

    Elmadany, and Muhammad Abdul-Mageed

    Ife Adebara, Hawau Olamide Toyin, Nahom Tesfu Ghebremichael, AbdelRahim A. Elmadany, and Muhammad Abdul-Mageed. 2025. https://doi.org/10.18653/v1/2025.acl-long.1572 Where are we? evaluating LLM performance on A frican languages . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32704...

    doi:10.18653/v1/2025.acl-long.1572 2025

  2. [2]

    Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, and 8 others

    David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En - Shiun Annie Lee, Chiamaka Ijeoma Chukwuneke, Happy Buzaaba, Blessing K. Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu N...

    doi:10.18653/v1/2025.naacl-long.139 2025

  3. [3]

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.614 Do all languages cost the same? tokenization in the era of commercial language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904--9923, ...

    doi:10.18653/v1/2023.emnlp-main.614 2023

  4. [4]

    Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Uttama Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.258 MEGA: multilingual evaluation of generative AI . In Proceedings of the 2023 Conference on Empirical Methods in Na...

    doi:10.18653/v1/2023.emnlp-main.258 2023

  5. [5]

    Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2024. https://doi.org/10.18653/V1/2024.NAACL-LONG.143 MEGAVERSE: benchmarking large language models across languages, modalities, models and tasks . In Proceedings of the 2024 Conferen...

    doi:10.18653/v1/2024.naacl-long.143 2024

  6. [6]

    Israel Abebe Azime, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Yonas Chanie, Bontu Fufa Balcha, Negasi Haile Abadi, Henok Biadglign Ademtew, Mulubrhan Abebe Nerea, Debela Desalegn Yadeta, Derartu Dagne Geremew, Assefa Atsbiha tesfau, Philipp Slusallek, Thamar Solorio, and Dietrich Klakow. 2025. https://doi.org/10.18653/V1/2025.FINDINGS-NAACL.350 Proverbe...

    work page doi:10.18653/v1/2025.findings-naacl.350 2025

  7. [7]

    Fabio Barth and Georg Rehm. 2025. https://doi.org/10.48550/ARXIV.2502.12895 Multilingual european language models: Benchmarking approaches and challenges . CoRR, abs/2502.12895

    work page Pith review arXiv doi:10.48550/arxiv.2502.12895 2025

  8. [8]

    Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. https://doi.org/10.18653/v1/2022.acl-long.376 Systematic inequalities in language technology performance across the world`s languages . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486--5505, Dublin, Ireland. Associ...

    doi:10.18653/v1/2022.acl-long.376 2022

Show all 46 references
  1. [9]

    Kranti Chalamalasetti, Jana G \" o tze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, and David Schlangen. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.689 clembench: Using game play to evaluate chat-optimized language models as conversational agents . In Proceeding...

    2023 doi

  2. [10]

    Marc-Alexandre C \^o t \'e , \'A kos K \'a d \'a r, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2019. Textworld: A learning environment for text-based games. In Computer Gam...

    2019

  3. [11]

    Christopher Zhang Cui, Xingdi Yuan, Ziang Xiao, Prithviraj Ammanabrolu, and Marc - Alexandre C \^ o t \' e . 2025. https://doi.org/10.48550/ARXIV.2504.14128 TALES: text adventure learning environment suite . CoRR, abs/2504.14128

    2025 doi arXiv

  4. [12]

    EU Charter 2012. 2012. https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:12012P/TXT Charter of fundamental rights of the European Union . Official Journal of the European Union, OJ C 326, 26.10.2012, p. 391--407. Art. 22: ``The Union shall respect cultural, religi...

    2012

  5. [13]

    Leon Guertler, Bobby Cheng, Simon Yu, Bo Liu, Leshem Choshen, and Cheston Tan. 2025. https://doi.org/10.48550/ARXIV.2504.11442 Textarena . CoRR, abs/2504.11442

    2025 doi arXiv

  6. [14]

    Wenhan Han, Yifan Zhang, Zhixun Chen, Binbin Li, Haobin Lin, Bingni Zhang, Taifeng Wang, Mykola Pechenizkiy, Meng Fang, and Yin Zheng. 2025. https://doi.org/10.48550/ARXIV.2506.19468 Mubench: Assessment of multilingual capabilities of large language models across 61 languages ...

    2025 doi arXiv

  7. [15]

    Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, Shruti Bhosale, Chenguang Zhu, Karthik Abinav Sankararaman, Eryk Helenowski, Melanie Kambadur, Aditya Tayade, Hao Ma, Han Fang, and Sinong Wang. 2024. https://doi....

    2024 doi arXiv

  8. [16]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In 9th International Conference on Learning Representations, ICLR 2021,...

    2021

  9. [17]

    Lanxiang Hu, Qiyu Li, Anze Xie, Nan Jiang, Ion Stoica, Haojian Jin, and Hao Zhang. 2025. https://openreview.net/forum?id=SeQ8l8xo1r Gamearena: Evaluating LLM reasoning through live computer games . In The Thirteenth International Conference on Learning Representations, ICLR 20...

    2025

  10. [18]

    Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, and Fei Yuan. 2025. https://aclanthology.org/2025.findings-emnlp.909/ Benchmax: A comprehensive multilingual evaluation suite for large language models . In Findings of the Association for Computational Linguis...

    2025

  11. [19]

    Dieuwke Hupkes and Nikolay Bogoychev. 2025. https://doi.org/10.48550/ARXIV.2504.10356 Multiloko: a multilingual local knowledge benchmark for llms spanning 31 languages . CoRR, abs/2504.10356

    2025 doi arXiv

  12. [20]

    Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili, Kavsar Huseynova, Dmitry Gaynullin, Anar Rzayev, Osman Tursun, Aizirek Turdubaeva, Ilshat Saetov, Rinat Kharisov, Saule Belginova, Ariana Kenbayeva, Amina Alisheva, Abdullatif K \"o ksal, Samir Rustamov, and Duygu Ataman. 2025...

    2025 doi

  13. [21]

    Khapra, and Pratyush Kumar

    Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.445 I ndic NLPS uite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual langua...

    2020 doi

  14. [22]

    Yekyung Kim, Jenna Russell, Marzena Karpinska, and Mohit Iyyer. 2025. https://doi.org/10.48550/ARXIV.2503.01996 One ruler to measure them all: Benchmarking multilingual long-context language models . CoRR, abs/2503.01996

    2025 doi

  15. [23]

    Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.878 Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual l...

    2023 doi

  16. [24]

    Zhenyu Li, Kehai Chen, Yunfei Long, Xuefeng Bai, Yaoyin Zhang, Xuchen Wei, Juntao Li, and Min Zhang. 2025. https://doi.org/10.48550/ARXIV.2503.07539 Xifbench: Evaluating large language models on multilingual instruction following . CoRR, abs/2503.07539

    2025 doi

  17. [25]

    Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Jann Railey Montalan, Ryan Ignatius Hadiwijaya, Joanito Agili Lopo, William Nixon, B \"o rje F

    Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James Validad Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P. Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Jann Railey ...

    2024 doi

  18. [26]

    Dan Nielsen. 2023. https://aclanthology.org/2023.nodalida-1.20/ S cand E val: A benchmark for S candinavian natural language processing . In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 185--201, T \'o rshavn, Faroe Islands. Universi...

    2023

  19. [27]

    Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman

    Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Haji c , Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. https://aclanthology.org/2020.lrec-1.497/ U niversal D ependencies v2: An evergrowing multilingual treebank c...

    2020

  20. [28]

    Jessica Ojo, Odunayo Ogundepo, Akintunde Oladipo, Kelechi Ogueji, Jimmy Lin, Pontus Stenetorp, and David Ifeoluwa Adelani. 2025. https://aclanthology.org/2025.findings-acl.976/ Afrobench: How good are large language models on african languages? In Findings of the Association f...

    2025

  21. [29]

    Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/74bb24dca8334adce292883b4b651eda-Abstract-Conference.html Language model tokenizers introduce unfairness between languages . In Advances in Neural I...

    2023

  22. [30]

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, and 3 others. 202...

    2025 doi arXiv

  23. [31]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. https://doi.org/10.48550/ARXIV.2311.12022 GPQA: A graduate-level google-proof q & a benchmark . CoRR, abs/2311.12022

    2023 doi arXiv

  24. [32]

    Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Imanol Schlag, Marzieh Fadaee, Sara Hooker, Antoine Bosselut, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat...

    2025

  25. [33]

    Phillip Rust, Jonas Pfeiffer, Ivan Vuli \'c , Sebastian Ruder, and Iryna Gurevych. 2021. https://doi.org/10.18653/v1/2021.acl-long.243 How good is your tokenizer? on the monolingual performance of multilingual language models . In Proceedings of the 59th Annual Meeting of the ...

    2021 doi

  26. [34]

    David Schlangen, Sherzod Hakimov, Jonathan Jordan, and Philipp Sadler. 2025. https://doi.org/10.48550/ARXIV.2507.08491 A third paradigm for LLM evaluation: Dialogue game-based evaluation using clembench . CoRR, abs/2507.08491

    2025 doi

  27. [35]

    L \"u tfi Kerem Senel, Benedikt Ebing, Konul Baghirova, Hinrich Schuetze, and Goran Glava s . 2024. https://aclanthology.org/2024.eacl-long.100/ Karde s - NLU : Transfer to low-resource languages with the help of a high-resource cousin -- a benchmark and evaluation for T urkic...

    2024

  28. [36]

    Shivalika Singh, Angelika Romanou, Cl \' e mentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila - Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei - Yin Ko, Antoine Bosselut, Alice...

    2025

  29. [37]

    Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. https://aaai.org/papers/11164-conceptnet-5-5-an-open-multilingual-graph-of-general-knowledge/ ConceptNet 5.5: An open multilingual graph of general knowledge . In Proceedings of the Thirty-First AAAI Conference on Artificia...

    2017

  30. [38]

    o hler, Ren \' e J \

    Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores - Herr, Joachim K \" o hler, Ren \' e J \" a kel, and Mehdi Ali. 2024. https://doi.org/10.48550/ARXIV.2410.08928 Towards multilingual LLM eva...

    2024 doi arXiv

  31. [39]

    Ahmet \" U st \" u n, Viraat Aryabumi, Zheng Xin Yong, Wei - Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui - Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. ...

    2024 doi

  32. [40]

    Menan Velayuthan and Kengatharaiyer Sarveswaran. 2025. https://aclanthology.org/2025.coling-main.400/ Egalitarian language representation in language models: It all begins with tokenizers . In Proceedings of the 31st International Conference on Computational Linguistics, pages...

    2025

  33. [41]

    Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, Qiqian Cang, Yichang Zhang, Fei Huang, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. https://doi.org/10.48550/ARXIV.2504.18428 Polymath: Evaluating ma...

    2025 doi

  34. [42]

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. https://doi.org/10.48550/ARXIV.2504.12516 Browsecomp: A simple yet challenging benchmark for browsing agents . CoRR, a...

    2025 doi arXiv

  35. [43]

    Minghao Wu, Weixuan Wang, Sinuo Liu, Huifeng Yin, Xintong Wang, Yu Zhao, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2025. https://doi.org/10.48550/ARXIV.2504.15521 The bitter lesson learned from 2,000+ multilingual benchmarks . CoRR, abs/2504.15521

    2025 doi arXiv

  36. [44]

    Chengxuan Xia, Qianye Wu, Hongbin Guan, Sixuan Tian, Yilun Hao, and Xiaoyu Wu. 2025. https://doi.org/10.48550/ARXIV.2511.10664 Evaluating modern large language models on low-resource and morphologically rich languages:a cross-lingual benchmark across cantonese, japanese, and t...

    2025 doi

  37. [45]

    Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, and 13 others. 2025. https://doi.org/10.18653/V1/2025....

    2025 doi

  38. [46]

    Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/117c5c8622b0d539f74f6d1fb082a2e9-Abstract-Datasets\_and\_Benchmarks.html M3exam: A multilingual, multimodal, multilevel benchmark for examining la...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.