REVIEW 7 cited by
On the Cross-lingual Transferability of Monolingual Representations
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross-lingual setting. This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions. We evaluate this hypothesis by designing an alternative approach that transfers a monolingual model to new languages at the lexical level. More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective, freezing parameters of all other layers. This approach does not rely on a shared vocabulary or joint training. However, we show that it is competitive with multilingual BERT on standard cross-lingual classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD). Our results contradict common beliefs of the basis of the generalization ability of multilingual models and suggest that deep monolingual models learn some abstractions that generalize across languages. We also release XQuAD as a more comprehensive cross-lingual benchmark, which comprises 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 translated into ten languages by professional translators.
Forward citations
Cited by 7 Pith papers
-
TyDi QA-WANA: A Benchmark for Information-Seeking Question Answering in Languages of West Asia and North Africa
TyDi QA-WANA is a new 28,000-example QA benchmark covering 10 under-represented languages with long-context, information-seeking questions and baseline evaluations.
-
skLEP: A Slovak General Language Understanding Benchmark
A nine-task Slovak-language understanding benchmark with translated and newly curated datasets, plus the first broad fine-tuned model comparison for Slovak.
-
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models
JQL trains small multilingual quality scorers from LLM judgments and human annotations, and filtering pretraining data with them improves downstream multilingual model performance over heuristic baselines.
-
The UD-NewsCrawl Treebank: Reflections and Challenges from a Large-scale Tagalog Syntactic Annotation Project
The authors manually annotated a 15,619-sentence Tagalog news treebank following Universal Dependencies and evaluated transformer-based dependency parsers against it, plus quality and topic analyses.
-
Neural FOXP2 -- Language Specific Neuron Steering for Targeted Language Improvement in LLMs
A three-stage SAE-plus-SVD steering recipe claims to make Hindi or Spanish the default language of an LLM at inference time, but the visible manuscript reports only expected, not measured, outcomes.
-
Transfer of Structural Knowledge from Synthetic Languages
A new synthetic language, flat_shuffle, transfers more structure to English fine-tuning than earlier synthetic bracket languages, though still far short of training on English from scratch.
-
Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs
Selective pre-translation, translating only some prompt components into English, generally outperforms both full prompt translation and direct inference across tasks and languages, with the largest gains for low-resou...
Discussion (0). Continue with ORCID to comment.