A Family of Pretrained Transformer Language Models for Russian

Alena Fenogenova; Alexander Abramov; Andrey Kalmykov; Artem Snegirev; Danil Astafurov; Dmitry Zmitrovich; Ekaterina Taktasheva; Maria Tikhonova; Mark Baushenko; Sergey Markov

arxiv: 2309.10931 · v4 · pith:26QX5NJYnew · submitted 2023-09-19 · 💻 cs.CL

A Family of Pretrained Transformer Language Models for Russian

Dmitry Zmitrovich , Alexander Abramov , Andrey Kalmykov , Maria Tikhonova , Ekaterina Taktasheva , Danil Astafurov , Mark Baushenko , Artem Snegirev

show 5 more authors

Vitalii Kadulin Sergey Markov Tatiana Shavrina Vladislav Mikhailov Alena Fenogenova

This is my paper

classification 💻 cs.CL

keywords languagerussiantransformermodelspretrainingresearchabilitiesapplications

0 comments

read the original abstract

Transformer language models (LMs) are fundamental to NLP research methodologies and applications in various languages. However, developing such models specifically for the Russian language has received little attention. This paper introduces a collection of 13 Russian Transformer LMs, which spans encoder (ruBERT, ruRoBERTa, ruELECTRA), decoder (ruGPT-3), and encoder-decoder (ruT5, FRED-T5) architectures. We provide a report on the model architecture design and pretraining, and the results of evaluating their generalization abilities on Russian language understanding and generation datasets and benchmarks. By pretraining and releasing these specialized Transformer LMs, we aim to broaden the scope of the NLP research directions and enable the development of industrial solutions for the Russian language.

This paper has not been read by Pith yet.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Zero-shot Large Language Models for Automatic Readability Assessment
cs.CL 2026-04 unverdicted novelty 7.0

Zero-shot LLMs with a new prompting method outperform prior unsupervised approaches on 13 of 14 readability datasets, and the hybrid LAURAE method improves robustness across languages, lengths, and technical content.