REVIEW 8 cited by
PolyLM: An Open Source Polyglot Large Language Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) demonstrate remarkable ability to comprehend, reason, and generate following nature language instructions. However, the development of LLMs has been primarily focused on high-resource languages, such as English, thereby limiting their applicability and research in other languages. Consequently, we present PolyLM, a multilingual LLM trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning. To assess the model's performance, we collect several existing multilingual tasks, including multilingual understanding, question answering, generation, and translation. Extensive experiments show that PolyLM surpasses other open-source models such as LLaMA and BLOOM on multilingual tasks while maintaining comparable performance in English. Our models, alone with the instruction data and multilingual benchmark, are available at: \url{https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation}.
Forward citations
Cited by 8 Pith papers
-
Cross-Lingual Transfer of Cultural Knowledge: An Asymmetric Phenomenon
Cross-lingual transfer of cultural knowledge is bidirectional for high-resource languages and asymmetric for low-resource ones, with corpus frequency correlating with transfer success.
-
Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian
Llama-GENBA-10B is a 10B-parameter trilingual model that reports top Bavarian scores among sub-10B models on a machine-translated benchmark the authors built.
-
Preperiodic points, finiteness, and structures of semigroups of algebraic morphisms
The paper proves finiteness and structural results for preperiodic points of algebraic morphisms, including Burnside-type and Northcott-type theorems.
-
Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model
A multilingual LLM training method that groups similar languages, converts high-deviation layers into mixture-of-experts layers, and assigns one expert per language group improves perplexity across 18 to 128 languages.
-
MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP
A new 11-task Maltese benchmark shows that 55 large language models lag behind small fine-tuned models, with prior Maltese exposure the strongest predictor.
-
CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning
CC-Tuning fuses English feed-forward activations into non-English inputs during multilingual supervised fine-tuning, using a trainable Decision Maker and a least-squares Transform Matrix to simulate the connection at ...
-
Assessing the Role of Data Quality in Training Bilingual Language Models
A quality filter trained only on English labels can select better French, German, and Chinese pretraining data, improving bilingual model performance and cutting the monolingual-bilingual gap to about 1%.
-
Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts
A layer-wise expert allocation algorithm based on hidden-state similarity, plus a routing classifier, improves parameter efficiency and reduces forgetting when expanding LLMs to new languages.
Discussion (0). Continue with ORCID to comment.