REVIEW 6 cited by
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders XTREME benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.
Forward citations
Cited by 6 Pith papers
-
KatotohananQA: Evaluating Truthfulness of Large Language Models in Filipino
KatotohananQA is a Filipino translation of TruthfulQA; seven LLMs scored 94.72% in English versus 83.87% in Filipino, with GPT-5 and GPT-5 mini showing the smallest gap.
-
When Alignment Hurts: Decoupling Representational Spaces in Multilingual Models
Projecting away the estimated Modern Standard Arabic subspace during fine-tuning improves generation across 25 Arabic dialects by up to +4.9 chrF++, evidence that subspace dominance by a high-resource variety restrict...
-
Beyond Literal Token Overlap: Token Alignability for Multilinguality
A new metric based on subword token alignment predicts cross-lingual transfer in multilingual models better than literal token overlap, especially for different-script language pairs.
-
Find Central Dogma Again: Leveraging Multilingual Transfer in Large Language Models
A GPT-2 model fine-tuned on multilingual sentence similarity achieves at best 81% accuracy on classifying matching vs non-matching DNA-protein pairs, but the result is highly seed-dependent and only with an easy test set.
-
Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages
Federated averaging of prompt embeddings from a frozen multilingual model improves accuracy on some low-resource tasks (XNLI) but not consistently on others (MasakhaNEWS).
- Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages
Discussion (0). Continue with ORCID to comment.