REVIEW 3 major objections 4 minor 48 references
Language verY Rare for All
T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A single-GPU fine-tuning recipe, LYRA, is claimed to make open LLMs match a 200-language model on Monégasque.
desk verdict The mo→fr results are likely inflated by oracle retrieval of the French reference, but the new Monégasque dataset and single-GPU recipe merit a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the LYRA recipe, a sequence of training and inference choices rather than a single new architecture. Data standardization fixes capitalization, punctuation, and quotation marks in the 10,794 parallel sentences. Transfer learning pre-fine-tunes each LLM on French-Italian parallel text before exposing it to Monégasque. Retrieval-augmented generation embeds the French side of every training sentence with a French-retrieval embedding model and, at test time, places the ten nearest French sentences in the prompt so the LLM translates with relevant examples in context. For the encoder-decoder baseline, the paper also uses NLLB's Ligurian tokenizer, since Ligurian is closer to Monégasque than Italian.
What would settle it
Take the released French-Monégasque corpus and split it by source so no book or document appears in both training and test, retrain LYRA-G, and compare BLEU and METEOR against NLLB-200; if the margin over NLLB vanishes or reverses, the claimed advantage came from test-train closeness.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a decoder-only LLM can be made competitive with a large dedicated translation model on a truly rare language by combining three interventions: cleaning the scarce parallel corpus, pre-fine-tuning on a related high-resource language pair (French-Italian), and adding test-time retrieval of similar examples. With all three in place, LYRA-G matches NLLB-200's BLEU on French-to-Monégasque and exceeds it on Monégasque-to-French. The authors read this as evidence that for rare-language translation the bottleneck is data organization and transfer from linguistically close languages, not model scale.
Load-bearing premise
The evaluation test set was hand-selected for high translation quality from the same literary sources used to build the training set, with no reported overlap check, so the reported gains could partly reflect retrieval of near-duplicate sentences rather than general translation skill.
Editorial extensions
If this is right
- Very low-resource languages can be added to machine translation systems without multi-GPU training, provided a related high-resource language and a cleaned parallel corpus exist.
- Translation into the high-resource language benefits most from retrieval; translation into the low-resource language benefits most from transfer learning, so the recipe's value is direction-dependent.
- Data standardization alone improves every tested model, making corpus cleaning a prerequisite rather than an optional refinement.
- RAG is not uniformly beneficial: one model's score dropped in the French-to-Monégasque direction when retrieval was added, so the strategy needs model-specific validation.
- The released French-Monégasque dataset gives the community a new testbed for rare-language translation methods.
Reading between the lines
- The Monégasque-to-French BLEU jumps could partly come from the LLM copying or lightly editing a retrieved French reference, since retrieval is done on the French side; ablating the retrieved French sentences would separate retrieval-as-memory from retrieval-as-example.
- The same recipe should transfer to other rare Romance languages close to a high-resource one, such as Occitan, Sardinian, or Corsican, though the authors do not test this.
- A domain-shift evaluation on Monégasque text outside the literary sources, such as transcribed speech or newswire, would test whether the standardization and transfer gains generalize beyond the corpus's own style.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces LYRA (Language verY Rare for All), a method for improving machine translation for a very low-resource language (Monégasque) by combining fine-tuning of open decoder-only LLMs, retrieval-augmented generation (RAG), and transfer learning from a related high-resource language (Italian). The authors created a French–Monégasque parallel dataset (about 10,794 sentence pairs plus dictionary/grammar entries), applied data standardization, and compared NLLB-200 1.3B with LoRA-fine-tuned Llama-3.1-8B, Gemma-2-9B, and Mistral-Nemo-12B. Table 1 shows that the best LYRA variant (LYRA-G) roughly ties NLLB in fr→mo BLEU (35.25 vs 35.27) and surpasses it in mo→fr BLEU (57.23 vs 52.18) when using RAG and Italian transfer learning. The paper claims LYRA 'frequently surpasses and consistently matches' state-of-the-art encoder-decoder models.
Significance. The paper makes a useful practical contribution by releasing a new parallel dataset for an extremely low-resource language and by demonstrating a single-GPU training pipeline that combines three techniques (data standardization, RAG, and transfer learning) explicitly targeted at low-resource conditions. The experimental design is clear, the training details (LoRA configuration, learning rates, early stopping) are transparent, and the data are publicly released. If the evaluation concerns are addressed, the approach could be informative for practitioners working on other low-resource languages. However, as presented, the central comparative claim is undermined by the oracle-retrieval issue in the mo→fr direction and by the lack of a properly held-out test set, so the significance of the reported results is conditional on a corrected evaluation.
major comments (3)
- [Section 4, RAG paragraph; Table 1]
- [Section 3, Data]
- [Section 5, Experiments; Table 1]
minor comments (4)
- [Section 3, Data; Appendix B]
- [Section 5, Models]
- [Figure 2 and Table 1]
- [Introduction, Related Works]
Circularity Check
mo→fr RAG evaluations use the gold French reference to retrieve nearest training examples, so the reported BLEU/METEOR gains over NLLB partly reduce to answer leakage.
-
self definitional
[Section 4, Retrieval Augmented Generation; results in Table 1 (Section 5)]
"Since the Monégasque language is unknown to the available embedding models, the French parts are used to generate an embedding for each instance. ... This is done for the train and test sets and the embeddings are used to improve test-time performance by retrieving, for each test sample, the 10 nearest train samples and including them in the prompt."
For the mo→fr direction, the model input is Monégasque and the French side of each test instance is the gold reference. Embedding each instance from its French side means the retrieval query for mo→fr is generated from the correct answer. Cosine-similarity retrieval then selects the 10 training pairs whose French sentences are nearest to the gold reference, seeding the prompt with near-answer translations. The reported mo→fr improvements from RAG (LYRA-G BLEU 52.12→58.10 and METEOR 69.40→74.31, vs NLLB 52.18/63.55) are therefore largely answer-neighbor retrieval, not independent translation quality. No masking of the reference for this direction is described.
full rationale
The paper's derivation chain is mostly empirical rather than theoretical, and there are no load-bearing self-citations or imported uniqueness theorems. However, the central 'surpassing' claim for mo→fr is partially circular by construction: Section 4 embeds each instance using its French side, and for mo→fr test items that French side is the gold translation. Retrieving the 10 nearest training pairs by French cosine similarity therefore uses the answer to select the prompt examples. Table 1 shows LYRA-G mo→fr BLEU rising from 52.12 without RAG to 58.10 with RAG, while NLLB-200 sits at 52.18; without RAG LYRA-G essentially matches NLLB. Thus the headline margin over NLLB in that direction is substantially an oracle-retrieval effect rather than translation skill. The fr→mo direction and the data-standardization and French-Italian transfer results are not affected by this leak and retain independent content. The test set is also described as 'constituted by selecting sentences with high quality translation' from the same literary sources used for training, with no reported overlap check, which is an additional external-validity risk but not a demonstrated formula-level circularity. Overall score 6: one key reported advantage reduces by construction, while other results remain independent.
Assumptions & free parameters
free parameters (4)
- Number of RAG retrievals =
10
- LoRA rank and alpha =
r=16, alpha=16
- Per-model learning rates =
1e-5 to 3e-5
- Early stopping epoch =
varies per model
assumptions (4)
- domain assumption Monégasque is grammatically similar to Italian and Ligurian, so transfer learning from French-Italian data and the Ligurian tokenizer are beneficial.
- domain assumption French embeddings from BAAI/bge-multilingual-gemma2 retrieve useful translation examples.
- domain assumption BLEU, METEOR, and chrF++ adequately measure translation quality for Monégasque.
- domain assumption The OCR-based corpus and manual annotation are accurate enough for training and evaluation.
Cite this review
Pith. "Pith review of Language verY Rare for All." pith.science (2026). https://pith.science/paper/CRNKLE2R
@misc{pith2026241213924,
author = {Pith},
title = {Pith review of: Language verY Rare for All},
year = {2026},
howpublished = {\url{https://pith.science/paper/CRNKLE2R}},
note = {Machine review of arXiv:2412.13924}
}
read the original abstract
In the quest to overcome language barriers, encoder-decoder models like NLLB have expanded machine translation to rare languages, with some models (e.g., NLLB 1.3B) even trainable on a single GPU. While general-purpose LLMs perform well in translation, open LLMs prove highly competitive when fine-tuned for specific tasks involving unknown corpora. We introduce LYRA (Language verY Rare for All), a novel approach that combines open LLM fine-tuning, retrieval-augmented generation (RAG), and transfer learning from related high-resource languages. This study is exclusively focused on single-GPU training to facilitate ease of adoption. Our study focuses on two-way translation between French and Mon\'egasque, a rare language unsupported by existing translation tools due to limited corpus availability. Our results demonstrate LYRA's effectiveness, frequently surpassing and consistently matching state-of-the-art encoder-decoder models in rare language translation.
Figures
Reference graph
Works this paper leans on
-
[1]
Duarte Miguel Alves, Jos \'e Pombal, Nuno M Guerreiro, Pedro Henrique Martins, Jo \ a o Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, Jos \'e G. C. de Souza, and Andre Martins. 2024. https://openreview.net/forum?id=EHPns3hVkj Tower: An open multilingual large language model for translation-related tasks ....
2024
-
[2]
Dzmitry Bahdanau. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473
arXiv 2014
-
[3]
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72
2005
-
[4]
Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. https://doi.org/10.18653/v1/2022.acl-long.376 Systematic inequalities in language technology performance across the world ' s languages . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486--5505, Dublin, Ireland. Asso...
-
[5]
Peter F Brown, Stephen A Della Pietra, Vincent J Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2):263--311
work page 1993
-
[6]
Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165
arXiv 2020
-
[7]
Kyunghyun Cho. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259
arXiv 2014
-
[8]
Marta R Costa-juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672
arXiv 2022
Show all 48 references
-
[9]
Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2020. A survey of multilingual neural machine translation. ACM Computing Surveys (CSUR), 53(5):1--38
2020
-
[10]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models . arXiv preprint arXiv:2407.21783
2024 arXiv
-
[11]
Ilshat Gibadullin, Aidar Valeev, Albina Khusainova, and Adil Khan. 2019. A survey of methods to leverage monolingual data in low-resource neural machine translation. arXiv preprint arXiv:1910.00373
2019 arXiv
-
[12]
Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jind r ich Helcl, and Alexandra Birch. 2022. Survey of low-resource machine translation. Computational Linguistics, 48(3):673--732
2022
-
[13]
Tahmid Hasan, Abhik Bhattacharjee, Kazi Samin, Masum Hasan, Madhusudan Basak, M Sohel Rahman, and Rifat Shahriyar. 2020. Not low-resource anymore: Aligner ensembling, batch filtering, and new datasets for bengali-english machine translation. arXiv preprint arXiv:2009.09359
2020 arXiv
-
[14]
Michael A Hedderich, Lukas Lange, Heike Adel, Jannik Str \"o tgen, and Dietrich Klakow. 2020. A survey on recent approaches for natural language processing in low-resource scenarios. arXiv preprint arXiv:2010.12309
2020 arXiv
-
[15]
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are GPT models at machine translation ? a comprehensive evaluation . arXiv preprint arXiv:2302.09210
2023 arXiv
-
[16]
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556
2022 arXiv
-
[17]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685
2021 arXiv
-
[18]
Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Dong Zhang, Zhehuai Chen, and Eng Siong Chng. 2024. GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators . arXiv preprint arXiv:2402.06894
2024 arXiv
-
[19]
John Hutchins. 1997. From first conception to first demonstration: the nascent years of machine translation, 1947--1954. a chronology. Machine Translation, 12:195--252
1997
-
[20]
William John Hutchins. 1986. Machine translation: past, present, future. Ellis Horwood Chichester
1986
-
[21]
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Asso...
2017
-
[22]
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. https://doi.org/10.18653/v1/2020.acl-main.560 The state and fate of linguistic diversity and inclusion in the NLP world . In Proceedings of the 58th Annual Meeting of the Association for Co...
2020 doi
-
[23]
Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press
2009
-
[24]
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf Large language models are zero-shot reasoners . In Advances in Neural Inf...
2022
-
[25]
Garry Kuwanto, Afra Feyza Aky \"u rek, Isidora Chara Tourni, Siyang Li, Alex Jones, and Derry Wijaya. 2023. Low-resource machine translation training curriculum fit for low-resource languages. In Pacific Rim International Conference on Artificial Intelligence, pages 453--458. Springer
2023
-
[26]
Alexandre Magueresse, Vincent Carles, and Evan Heetderks. 2020. Low-resource languages: A review of past work and future challenges. arXiv preprint arXiv:2006.07264
2020 arXiv
-
[27]
Mieradilijiang Maimaiti, Yang Liu, Huanbo Luan, and Maosong Sun. 2019. Multi-round transfer learning for low-resource nmt using multiple high-resource languages. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 18(4):1--26
2019
-
[28]
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196
2024 arXiv
-
[29]
Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Mere...
2020
-
[30]
Iroro Orife, Julia Kreutzer, Blessing Sibanda, Daniel Whitenack, Kathleen Siminyu, Laura Martinus, Jamiil Toure Ali, Jade Abbott, Vukosi Marivate, Salomon Kabongo, et al. 2020. Masakhane--machine translation for africa. arXiv preprint arXiv:2003.11529
2020 arXiv
-
[31]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...
2022
-
[32]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318
2002
-
[33]
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. https://proceedings.neurips.cc/paper_files/paper/2021/file/5c04925674920eb58467fb52ce4ef728-Paper.pdf True few-shot learning with language models . In Advances in Neural Information Processing Systems, volume 34, pages 11054--...
2021
-
[34]
Maja Popovi \'c . 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392--395
2015
-
[35]
Alec Radford. 2018. Improving language understanding by generative pre-training
2018
-
[36]
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446
2021 arXiv
-
[37]
Laria Reynolds and Kyle McDonell. 2021. https://doi.org/10.1145/3411763.3451760 Prompt programming for large language models: Beyond the few-shot paradigm . In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA '21, New York, NY, USA. A...
2021
-
[38]
Rico Sennrich. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709
2015 arXiv
-
[39]
J \"o rg Tiedemann. 2009. News from OPUS - A Collection of Multilingual Parallel Corpora with Tools and Interfaces, volume V, pages 237--248
2009
-
[40]
Yury Tokpanov, Beren Millidge, Paolo Glorioso, Jonathan Pilault, Adam Ibrahim, James Whittington, and Quentin Anthony. 2024. Zyda: A 1.3 T Dataset for Open Language Modeling . arXiv preprint arXiv:2406.01981
2024 arXiv
-
[41]
A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems
2017
-
[42]
Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. https://openreview.net/forum?id=yzkSU5zd...
2022
-
[43]
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024. https://openreview.net/forum?id=farT6XXntP A paradigm shift in machine translation: Boosting translation performance of large language models . In The Twelfth International Conference on Learning Representations
2024
-
[44]
Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023. Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages. arXiv preprint arXiv:2305.18098
2023 arXiv
-
[45]
Jiajun Zhang and Chengqing Zong. 2020. Neural machine translation: Challenges, progress and future. Science China Technological Sciences, 63(10):2028--2050
2020
-
[46]
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. arXiv preprint arXiv:1604.02201
2016 arXiv
-
[47]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[48]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.