REVIEW 4 cited by
MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present MultiLoKo, a new benchmark for evaluating multilinguality in LLMs covering 31 languages. MultiLoKo consists of three partitions: a main partition consisting of 500 questions per language, separately sourced to be locally relevant to the specific language, and two translated partitions, containing human-authored translations from 30 non-English languages to English and vice versa. For comparison, we also release corresponding machine-authored translations. The data is equally distributed over two splits: a dev split and a blind, out-of-distribution test split. MultiLoKo can be used to study a variety of questions regarding the multilinguality of LLMs as well as meta-questions about multilingual benchmark creation. We compute MultiLoKo scores for 11 base and chat models marketed to be multilingual and study their average performance, their performance parity across languages, how much their ability to answer questions depends on the question language, and which languages are most difficult. None of the models we studied performs well on MultiLoKo, as indicated by low average scores as well as large differences between the best and worst scoring languages. Furthermore, we find a substantial effect of the question language, indicating sub-optimal knowledge transfer between languages. Lastly, we find that using local vs English-translated data can result in differences more than 20 points for the best performing models, drastically change the estimated difficulty of some languages. For using machines instead of human translations, we find a weaker effect on ordering of language difficulty, a larger difference in model rankings, and a substantial drop in estimated performance for all models.
Forward citations
Cited by 4 Pith papers
-
Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+
Across 30 languages, commercial LLMs outscore all open-weight models in every EU language, and non-English service costs more and scores lower, suggesting equality requires resources beyond public web crawls.
-
MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark
Multilingual LLMs show a reproducible 'Illusion of Cultural Alignment': they can be fluent in a language while lacking the culture's factual knowledge, and confidence, sampling, and retrieval do not fix it.
-
MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
A native-authored French, Spanish, and Chinese reasoning benchmark shows current LLMs score below 50% and improve by about 10% on math when questions are in English.
-
Rethinking Cross-lingual Gaps from a Statistical Viewpoint
Cross-lingual accuracy gaps in LLMs are dominated by higher response variance in target languages, not missing knowledge; ensembling and variance-reduction prompts shrink the gap.
Discussion (0). Continue with ORCID to comment.