REVIEW 24 cited by
mT5: A massively multilingual pre-trained text-to-text transformer
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.
Forward citations
Cited by 24 Pith papers
-
On the Fitness Landscape in the $NK$ Model
For the NK fitness landscape with K/N tending to alpha, exact limits for free energy and maximum fitness are identified, together with the geometry of near-fittest peaks.
-
Meta CLIP 2: A Worldwide Scaling Recipe
A data curation and training recipe that scales CLIP from English-only data to 300+ languages from scratch, breaking the curse of multilinguality at ViT-H/14 scale.
-
Fine-Grained Chinese Hate Speech Understanding: Span-Level Resources, Coded Term Lexicon, and Enhanced Detection Frameworks
The paper creates a span-level Chinese hate speech dataset and a 830-term coded hate lexicon, but its two-stage training method's reported superiority is contradicted by the paper's own COLD results.
-
Effective vocabulary expansion of multilingual language models for extremely low-resource languages
Using bilingual-dictionary-aligned embeddings to initialize expanded vocabulary improves mBERT continued pre-training for seven low-resource languages, with average gains of 0.54% (POS) and 2.60% (NER).
-
Evaluating LLMs on Chinese Idiom Translation
Across 900 annotated translation pairs from nine MT systems, the best system still mistranslates Chinese idioms in 28% of cases, and standard metrics miss these errors (Pearson correlation below 0.48).
-
Multi-task Adversarial Attacks against Black-box Model with Few-shot Queries
CEMA converts multi-task black-box text attacks into attacks on a binary classifier trained on cluster pseudo-labels, achieving high attack success with 100 queries.
-
Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwan
Multi-TW is the first Traditional Chinese benchmark to evaluate multimodal models on both image-text and audio-text questions while also measuring inference latency.
-
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
An adaptive, per-language data filtering and deduplication pipeline produces multilingual LLM pre-training corpora that beat prior public datasets on 11 of 14 evaluated languages, and a 20TB, 1,868 language-script dat...
-
Cost-Optimal Active AI Model Evaluation
The paper derives budget-optimal rules for mixing cheap weak raters and expensive strong raters so that the mean rating is estimated with minimum variance, and shows large cost savings when example difficulty varies.
-
OneSug: The Unified End-to-End Generative Framework for E-commerce Query Suggestion
A unified encoder-decoder model with prefix representation enhancement and reward-weighted DPO outperforms Kuaishou's online multi-stage query suggestion system in offline and live A/B evaluations.
-
Entity Image and Mixed-Modal Image Retrieval Datasets
A new benchmark, MMIR, built from Wikipedia and WIT by masking entity names and supplying canonical entity images, tests mixed-modal retrieval with single- and multi-entity queries.
-
MultimodalHugs: Enabling Sign Language Processing in Hugging Face
MultimodalHugs provides a standardized TSV-based dataset format, modular processors, and Hugging Face integration to enable reproducible sign language and multimodal translation experiments.
-
Two Spelling Normalization Approaches Based on Large Language Models
Fine-tuned mT5 and mBART perform competitively on Spanish and Slovene historical spelling normalization, but character-based SMT achieves the best overall scores, confirming its continued state-of-the-art status.
-
Pi-SQL: Enhancing Text-to-SQL with Fine-Grained Guidance from Pivot Programming Languages
Generating Python programs as intermediate guidance before SQL, then voting on Python execution results to select the fastest matching SQL, improves text-to-SQL execution accuracy and efficiency on BIRD and Archer.
-
Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages
MSumBench is a new English/Chinese benchmark that grades summaries across six domains on faithfulness, completeness, and conciseness, using domain-specific key-facts and multi-agent-debate-assisted human annotations, ...
-
Mutarjim: Advancing Bidirectional Arabic-English Translation with a Small Language Model
A compact 1.5B Arabic-English model beats GPT-4o mini only on the authors' own Tarjama-25 benchmark, while trailing large models on standard WMT24++ and IWSLT2017 tests.
-
SELF: Self-Extend the Context Length With Logistic Growth Function
SELF replaces Self-Extend's fixed token grouping with a logistic-growth grouping schedule, giving mixed but sometimes large gains on long-context benchmarks.
-
LAGO: Few-shot Crosslingual Embedding Inversion Attacks via Language Similarity-Aware Graph Optimization
LAGO shows that constraining alignment matrices of linguistically similar languages to be close improves few-shot cross-lingual embedding inversion accuracy over independent per-language baselines.
-
Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs
Selective pre-translation, translating only some prompt components into English, generally outperforms both full prompt translation and direct inference across tasks and languages, with the largest gains for low-resou...
-
DIRECT: Direct Decoding for Efficient and Aligned Sequence Labeling with Large Language Models
DPO after SFT plus constrained template-filling decoding improves LLM sequence-labeling accuracy and cuts inference time by reusing KV cache for non-label tokens.
-
Rethinking Indic AI from a Lens of Cultural Heritage Preservation
The paper surveys Indic NLP evolution and proposes 'Culture Sensing' to integrate indigenous oral knowledge into foundation models for cultural preservation.
-
SHAMI-MT: A Syrian Arabic Dialect to Modern Standard Arabic Bidirectional Machine Translation System
A bidirectional MSA-Syrian Arabic translation system built by fine-tuning AraT5v2 on the Nabra corpus; only the MSA-to-Shami direction is evaluated, with a GPT-4.1 score of 4.01/5.
-
Token Constraint Decoding Improves Robustness on Question Answering for Large Language Models
Forcing a language model to output only the allowed answer-letter tokens, with a tuned penalty, recovers accuracy lost to a single extra space in multiple-choice prompts.
- Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages
Discussion (0). Continue with ORCID to comment.