REVIEW 11 cited by
Extrapolating Large Language Models to Non-English by Aligning Languages
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Existing large language models show disparate capability across different languages, due to the imbalance in the training data. Their performances on English tasks are often stronger than on tasks of other languages. In this paper, we empower pre-trained LLMs on non-English languages by building semantic alignment across languages. We start from targeting individual languages by performing cross-lingual instruction-tuning (CoIT) on LLaMA, i.e. tuning it with translation task data and cross-lingual general task data to obtain cross-lingual models (x-LLaMAs), and formulate underlying scaling laws to investigate the advantages of using scalable translation data. Then we perform multilingual instruction-tuning (MuIT) with mixed resources to build multilingual m-LLaMA. We also illustrate how we leverage the scaling laws to optimize data allocation in a resource-constrained setting. Experiment results on cross-lingual benchmarks XQUAD and MLQA show that x-LLaMAs surpass the English instruction-tuned counterpart (Alpaca) by an average of 27.83% across six non-English languages. Evaluation results on translation dataset Flores-101 show that x-LLaMAs outperform previous LLaMA-based models by an average of 18.89%. Encouragingly, m-LLaMA achieves comparable performance to x-LLaMAs on individual languages and demonstrates the ability to follow multilingual instructions. Further analysis on response content and representation space reveals the alignment of the multilingual semantic space within the middle layers of m-LLaMA.
Forward citations
Cited by 11 Pith papers
-
ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling
Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.
-
TokAlign: Efficient Vocabulary Adaptation via Token Alignment
TokAlign aligns source and target BPE token vocabularies using GloVe co-occurrence embeddings and re-initializes LLM embeddings, recovering within 5k steps and enabling token-level distillation.
-
CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning
CC-Tuning fuses English feed-forward activations into non-English inputs during multilingual supervised fine-tuning, using a trainable Decision Maker and a least-squares Transform Matrix to simulate the connection at ...
-
BadLingual: A Novel Lingual-Backdoor Attack against Large Language Models
A backdoor attack that uses language as the trigger works on specific tasks, but the claimed task-agnostic generalization (BadLingual) is only demonstrated in a few settings and is contradicted by many of the paper's ...
-
CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation
A confidence-reward score for selecting preference pairs improves DPO-based machine translation fine-tuning over reward-only selection methods on ALMA-7B and NLLB-1.3B.
-
Text2Cypher Across Languages: Evaluating and Finetuning LLMs
A new multilingual Text2Cypher benchmark shows LLMs rank English highest, Spanish next, and Turkish lowest, and multilingual finetuning narrows the language gap more than English-only finetuning.
-
Facilitating large language model Russian adaptation with Learned Embedding Propagation
Russian-adapted embeddings from foundation LLMs are propagated into instruction-tuned variants, matching original benchmark performance without instruction-tuning data.
-
Self-Evolution Knowledge Distillation for LLM-based Machine Translation
A token-adaptive distillation method that mixes teacher and ground-truth targets only for hard tokens yields consistent BLEU gains in LLM translation.
-
Facts Do Care About Your Language: Assessing Answer Quality of Multilingual LLMs
A small evaluation of Llama 3.1 shows factuality in school-level question answering degrades with decreasing language speaker count, though the statistical support is weakened by methodological issues.
-
Improving Multilingual Language Models by Aligning Representations through Steering
A single learned steering vector added to one transformer layer improves multilingual task performance without fine-tuning, and transfers between related languages.
-
Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation
Bilingual embedding alignment plus instruction tuning improves Persian classification in Llama-2, while English-to-Persian transfer is marginal and task-dependent.
Discussion (0). Continue with ORCID to comment.