Pith. sign in

REVIEW 4 major objections 7 minor 36 references

Facilitating large language model Russian adaptation with Learned Embedding Propagation

T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes Learned Embedding Propagation, which transfers continued-pretrained Russian embeddings into an instruction-tuned LLM via a linear projection and matches or exceeds the original model without instruction-tuning on…

desk verdict Useful method and benchmark for Russian LLM adaptation, but selection on the evaluation benchmark undermines the headline numbers. read the letter →

arxiv 2412.21140 v1 pith:DZMQTPXD submitted 2024-12-30 cs.CL cs.AI

Mikhail Tikhomirov , Daniil Chernyshev This is my paper · ORCID
classification cs.CLcs.AI
keywords largelanguagemodelsadaptationRussianNLPembeddingpropagationvocabularyextensioncontinuedpretraininginstructiontuningtextgenerationbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Learned Embedding Propagation (LEP), a way to adapt an instruction-tuned large language model to Russian without running instruction-tuning on Russian data. The method continued-pretrains only the input and output embeddings of a base model on Russian text, then propagates those trained embeddings into an existing instruct-tuned version of the same model using one of three linear projections computed from shared tokens. On the authors' Darumeru benchmark, the propagated models match or exceed the original instruction-tuned models on several Russian task groups, and additional calibration steps push them further. If the result holds, language adaptation no longer requires access to a high-quality instruction dataset in the target language, and the instruct-tuned model's proprietary tuning data can be left untouched.

What carries the argument

The load-bearing object is the embedding-propagation identity $LLM_{ru/cpt\to inst} = I_{ru/cpt} D_{inst}^{ru} W_{inst} U_{inst}^{ru} O_{ru/cpt}$, which states that an instruction-tuned variant of a language-adapted model can be obtained by sandwiching the instruction-tuned transformer body between two alignment matrices applied to the continued-pretrained embeddings. Those matrices come from least-squares pseudo-inverses of the overlap between the original and new vocabularies, or from vocabulary-conversion projections. The paper's key claim is that this linear correction is sufficient to skip the instruction-tuning step entirely.

What would settle it

Take a base model and its instruct-tuned version, run LEP with the vocabulary-conversion projection, then compare the propagated model's zero-shot Russian benchmark scores against the original instruct-tuned model's scores; if the propagated model does not at least match the original on tasks that do not require new Russian knowledge, the assumption that the transformer body is unchanged is falsified. A more direct check is to measure hidden-layer activation drift on a fixed Russian text set before and after the embedding swap and look for a large drop in cosine similarity.

Watch

Extended reading notes

Core claim

LEP's central claim is that the entire language-adaptation pipeline can be compressed to two stages: continue-pretraining only the new embedding matrices on target-language text, then transferring those embeddings into any instruct-tuned variant of the same base model through a learned linear transform. The transfer writes the model as $LLM = I W O$, regards the instruct-tuned body as $I_{base} D W_{inst} U O_{base}$, and after continued pretraining approximates $W_{ru\to inst}\approx W_{inst}$, yielding the propagated model $I_{ru/cpt} D W_{inst} U O_{ru/cpt}$. The alignment matrices $D$ and $U$ are estimated either as identity (direct swap), as least-squares projections over the common token subset, or as vocabulary-conversion projections. The paper reports that the vocabulary-conversion variant is the most reliable, and that after self-calibration or continued instruction-tuning on a Russian instruction dataset the adapted models reach or surpass the original instruct-tuned quality on the Darumeru benchmark.

Load-bearing premise

The central assumption is that replacing only the input and output embeddings of an instruction-tuned model with language-adapted embeddings, plus a linear correction estimated on shared tokens, leaves the model's learned reasoning layers intact and functional; if those internal layers shift when the vocabulary changes, the propagated model loses the instruction-tuned behavior.

Editorial extensions

If this is right

  • Russian adaptation of 7B-8B models can be done with continued pretraining of embeddings only, plus a projection step, so high-quality Russian instruction data is no longer a prerequisite.
  • The three propagation variants give a practical recipe: for Mistral-7B, vocabulary extension with conversion projection beats the original OpenChat 3.5; for LLaMa-3-8B-Instruct, no propagation variant fully recovers the original until calibration is applied.
  • Self-calibration with model-generated answers and copy tasks improves token utilization and summarization, but can hurt comprehension of rare concepts for weaker vocabulary adaptations.
  • Continued instruction-tuning on the Saiga dataset with an added 2000-example copying task boosts adapted models beyond their non-adapted counterparts on several Russian task groups.
  • The Darumeru benchmark provides an offline, train-time evaluation with open labels, separating understanding tasks, summarization, and token-copy reliability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same propagation step should work for any language pair with a reasonable token overlap, not only Russian, because the linear-correction machinery is language-agnostic; the paper only tests Russian.
  • Editorial inference: if the equal-performance result on OpenChat 3.5 reflects a general property, then instruction-following skill lives mostly in the transformer body, and LEP-style propagation could replace distillation pipelines for adapting proprietary instruct-tuned models to low-resource languages.
  • Editorial inference: the DaruCopy task, by measuring whether a model can reproduce input text token-for-token, gives a direct probe of output-embedding alignment that could be adopted as a standard regression test for any vocabulary-swapping method.
  • Editorial inference: a natural next experiment would be applying LEP to a language with near-zero token overlap, such as Japanese or Arabic, where the least-squares projection has little common-token support and the method's graceful-degradation boundary can be mapped.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Learned Embedding Propagation (LEP), a recipe for adapting 7B-8B open-weight LLMs to Russian while skipping the instruction-tuning step. The pipeline is: (i) train a Russian-optimized tokenizer (BPE, Unigram, Extended, or Optimized), (ii) initialize the new embeddings by averaging overlapping base-token embeddings, (iii) continue-pretrain only the input/output embeddings of the base (foundation) model on Russian text, and (iv) propagate these trained embeddings into the corresponding instruct-tuned model using one of three linear alignment transforms: direct swap, least-squares correction on overlapping tokens (Eqs. 8-11), or a vocabulary-conversion projection (Eqs. 12-14). The derivation is based on a pseudo-linear factorization LLM = I W O (Eq. 3) and the assumption W_ru_to_inst is approximately W_inst (Eq. 7). The authors also introduce Darumeru, a Russian benchmark assembled from MERA, MMLU, RuCoLA, Gazeta, and new copy tasks, and evaluate four vocabulary adaptations of Mistral-7B (with OpenChat 3.5) and LLaMa-3-8B (with its Instruct variant), plus case studies on self-calibration and continued instruction-tuning on Saiga data. Headline results are that LEP with calibration reaches micro-average scores of 0.632 (OpenChat-based) and 0.618 (LLaMa-3-based) versus 0.607 and 0.610 for the original instruct models.

Significance. The manuscript ships all artifacts (models, code, benchmark framework) under open licenses, compares four vocabulary-adaptation strategies systematically on two model families, and introduces a benchmark whose diagnostic tasks (DaruSum abstractive/extractive separation, DaruCopy token-utilization test) address a real gap in Russian LLM evaluation. If the results withstand a clean evaluation, LEP would be a genuinely cheaper alternative to full language adaptation: it needs no Russian instruction data, touches only embedding layers, and reuses existing instruct-tuned weights. The honest Limitations section (need for the foundation model, weak initialization for non-Cyrillic scripts, bounded transfer volume) is to the authors' credit. The concern is that the quantitative claims at the center of the abstract are currently supported only by single-seed runs whose selection protocol is entangled with the reporting benchmark; this is fixable in revision but is not a presentation issue.

major comments (4)
  1. [Sec. 2.4.1, Sec. 3.2, Tables 2-4] The same benchmark, Darumeru, is used both as the model-selection instrument and as the final evaluation. In Sec. 2.4.1 the learning rate is chosen 'based on benchmark results' on Darumeru, and in Sec. 3.2 the tables report 'the best performing checkpoints' from the Darumeru-scored dynamics of Figure 2. The baseline models in Table 1 receive no such selection: each is evaluated once. This asymmetry matters because the claimed advantages are tiny: Table 3 shows OpenChat Extended Conversion at 0.609 versus the original OpenChat at 0.607, and Table 4 shows 'original model + saiga d7' at 0.615 versus LEP Extended + saiga d7 at 0.614, with the +copy-task rows at 0.616 versus 0.618. These margins are within the range that checkpoint and learning-rate selection over a noisy benchmark can produce. The manuscript should separate validation from test: select hyperparameters and checkpoints on one portion of Darumeru (or on another criterion) and report final numbers on an untouched held-out portion or an external benchmark, with the baselines evaluated under the same protocol.
  2. [Sec. 3.2-3.5, Tables 1-4] All reported scores are single point estimates from single runs: there are no standard deviations, no repeated seeds, and no significance testing, and Darumeru's own run-to-run stability is never characterized. Despite this, the text makes strong comparative claims on very small margins: 'even outperforming the original OpenChat 3.5' for 0.609 versus 0.607 (Table 3), 'by a significant margin' in Sec. 1 for the Figure 1 comparison, and 'new benchmark records' in the Conclusion for differences around 0.01-0.02 (Table 4). Note also that the 0.632 OpenChat-based result does not exceed the 0.639 of Mistral Nemo in Table 1, so 'records' needs qualification. At minimum the paper should report variance (multiple seeds or multiple final checkpoints), state the benchmark's noise level, and temper the superiority language to what single-run differences support.
  3. [Sec. 2.2, Eqs. (3)-(7)] The derivation's two structural premises are not validated. First, the factorization LLM = I W O (Eq. 3) treats the entire transformer stack as a single pseudo-linear matrix, and the paper does not test how well this bilinear form approximates the actual models. Second, the step from Eq. (6) to Eq. (7), 'by assuming that the optimal W_ru_to_inst is approximately W_inst', is asserted without evidence or an ablation, although it is the load-bearing premise of the propagation idea. The paper's own results suggest the approximation is lossy: Table 3 shows DaruCopy (RU) falling from 0.972 (LLaMa-3-8B Instruct) to 0.921 under Extended Conversion, and no experiment isolates whether this loss comes from the W mismatch or from the D/U projections. A concrete validation would be to continue-pretrain the instruct-tuned model's own embeddings with frozen inner layers and compare against LEP, or to measure the drift of inner-layer representations after embedding replacement. If the equations are intended only as motivation, the paper should say so and instead justify the method empirically; as written, they are presented as the basis of the three alignment variants in Secs. 2.2.1-2.2.3.
  4. [Abstract, Sec. 3.5, Tables 3-4] The central comparison that would justify the abstract's claim of competitiveness with 'traditional instruction-tuning methods' (running the full language-adaptation pipeline on the same base models and the same vocabulary variants, then instruction-tuning) is not performed. The comparisons in Tables 1-4 are against third-party instruct-tuned models (OpenChat, Saiga, Vikhr, LLaMa-3-8B-Instruct) that were trained with different data and procedures, so the advantage of LEP over the pipeline it is designed to replace is never measured directly. Given the paper's stated purpose of skipping the instruction-tuning step, a within-study control that instruction-tunes a continued-pretrained base model of the same family is needed before the headline claim is fully supported.
minor comments (7)
  1. [Sec. 1, References] The citation 'Husain (2024)' in Sec. 1 has no corresponding entry in the reference list; please add it or remove the citation.
  2. [Sec. 5 vs. Conclusion] The Conclusion's claim of 'cost-efficient utilization of any instruction-tuned models regardless of openness of their fine-tuning data' overstates the method, since Sec. 5 requires the foundation model to be available; proprietary models without released base weights are out of scope. The phrasing should be tightened.
  3. [Secs. 3.1 and 4.1] The claims that 'some parts of MMLU and MERA datasets were leaked to training data of Vikhr-5.2 and Qwen-2 7B' and that Vikhr results 'clearly suggest the case of benchmark hacking' are serious allegations about third-party models supported only by a score pattern; they should be presented as hypotheses or accompanied by contamination evidence such as n-gram overlap analysis.
  4. [Sec. 2.1.1] The description of the Optimized tokenizer is incomplete: 'reducing it to the most common 50% tokens of Russian corpora and then subsequent' is followed by a stray bullet ('Extension to the original size') and an unclear parenthetical. Please describe the construction fully and unambiguously.
  5. [Secs. 2.4.1 and 3.2] Please clarify whether the learning-rate search on 20% of the data was followed by a fresh full-data training run for the final checkpoints, and how the checkpoint-selection points in Figure 2 relate to the training-data split; as written, the reader cannot tell whether the reported models were trained on 20% or on the full corpus.
  6. [Secs. 2.3.2-2.3.3, 2.4.2] LLaMa-3-70B-Instruct is used to rephrase MultiQ gold answers, to filter DaruSum examples, and to grade the self-calibration data; the benchmark's and calibration's dependence on LLaMa-3-family judgments should be discussed as a possible source of bias, given that the strongest reported final results are for LLaMa-3-8B-based models.
  7. [Throughout] Copy-editing is needed: 'don not' in the Abstract, comma-as-decimal in Tables 1-4 versus dot in the text, inconsistent 'LoRA'/'LoRa' in Secs. 2.4.2-2.4.3, unnatural breaks such as 'V ocabulary' in section titles, and several run-on sentences in Sec. 2.3.5.

Circularity Check

2 steps flagged · score 6.0 of 10

Core LEP derivation is self-contained, but the reported empirical gains are partially forced by hyperparameter/checkpoint selection on Darumeru and by calibrating on copy tasks that mirror the DaruCopy evaluation.

  1. fitted input called prediction [Section 2.4.1 (Continued Pre-training); Section 3.2, Tables 2-4]
    "We tested 4 different learning rates: 2e-5, 5e-5, 1e-4, 2e-4 for each model and tokenization on 20% of all continued pre-training dataset. Based on benchmark results, we chose a learning rate equal to 1e-4 for all Mistral-7B models, and learning rate equal to 2e-4 for LLaMa-3-8B models. ... In Table 2 we report the detailed results of the best performing checkpoints."

    The same Darumeru benchmark is used both to choose the learning rate (and, via Figure 2's benchmark-scored training dynamics, the best checkpoint) and as the final evaluation instrument in Tables 2-4. The reported scores are therefore maxima over an implicit hyperparameter and checkpoint search on the reporting benchmark, while the baseline models in Table 1 are evaluated once with no such selection. The headline claim of competitiveness with OpenChat 3.5 and LLaMa-3-8B-Instruct is thus a selected result rather than an independent prediction: the evaluation quantity is also the selection criterion.

  2. fitted input called prediction [Sections 2.3.4, 2.4.2, and 3.4]
    "We used Wikipedia articles of different genres to collect copy task datasets for English and Russian languages ... we supplemented the self-instruct dataset with 10000 article-copy task examples, obtained from the part of Wikipedia that has no overlap with our pre-training or benchmark datasets. ... As expected the performance of DaruCopy tasks improved substantially, practically reaching the perfect reliability levels."

    DaruCopy is defined as a Wikipedia article-copy task, and the self-calibration procedure trains on 10000 article-copy examples from Wikipedia. The reported DaruCopy improvement is therefore not independent evidence of general task-solving enhancement: the calibration data is drawn from the same task family as the evaluation metric, so the score increase is largely forced by training directly on the skill that the metric measures. Avoiding exact overlap prevents test-set leakage but not task-level circularity.

full rationale

The mathematical core of LEP (Eqs. 3-14) is not circular: it is an explicit construction using pseudo-inverse embeddings and the stated approximation W_ru_to_inst is approximately W_inst, with no benchmark labels or target model outputs used to fit the propagation matrices. The self-citations to the authors' earlier tokenization papers are background and do not carry the central derivation. The partially circular element is empirical: the headline comparison is weakened because learning rates and checkpoints are selected on Darumeru and then evaluated on Darumeru, and because the self-calibration case study trains on article-copy examples that directly mirror the DaruCopy benchmark component. These factors do not make the method definitionally equivalent to its inputs, but they do make several reported improvements statistically forced by the evaluation protocol rather than free-standing predictions. The open-source release and use of external datasets are genuine strengths, but they do not remove this selection-on-evaluation bias.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a linear factorization of a transformer and on unverified equivalence assumptions, plus several hyperparameters tuned on the evaluation benchmark. No new physical entities are introduced; the contributions are procedural and benchmark-related.

free parameters (6)
  • Continued pretraining learning rate = Mistral-7B: 1e-4; LLaMa-3-8B: 2e-4
    Selected from {2e-5, 5e-5, 1e-4, 2e-4} based on Darumeru scores; the paper notes performance depends strongly on learning rate (Section 2.4.1).
  • Embedding alignment transformations D and U = Not reported; estimated by least-squares pseudo-inverse in Eqs. (8)-(14)
    Central to LEP; computed as pseudo-inverses on overlapping token embeddings and not validated on held-out data.
  • Self-calibration example filtering threshold = Score >= 4 out of 5; 13531 examples retained
    Ad hoc threshold for example quality filtering in the self-calibration case study (Section 2.4.2).
  • LoRA calibration hyperparameters = Self-calibration: rank 8, alpha 1, lr 2.5e-5; Saiga tuning: rank 32, alpha 16, lr 5e-5
    Used to produce the final calibrated results in Table 4; no sensitivity analysis is reported.
  • Copy-task mixture size in calibration data = 10000 Wikipedia copy-task examples; 2000 special copy instructions in Saiga tuning
    Chosen by hand to prevent tokenization regression during calibration (Sections 2.4.2 and 2.4.3).
  • Darumeru filtering thresholds = 20% unsupported summary sentences; PARus reverse-order correctness; 99% LCS copy threshold
    Benchmark construction choices that affect reported scores but are not part of the LEP derivation.
assumptions (6)
  • ad hoc to paper LLM behavior can be represented as LLM = I W O, with W a pseudo-linear approximation of all intermediate layers.
    Invoked in Eq. (3); actual transformer layers are nonlinear and are not a single matrix, so this is an unvalidated modeling simplification.
  • ad hoc to paper Continued pretraining of embeddings leaves intermediate layer dynamics unchanged: W_ru_to_inst approximately equals W_inst.
    Key assumption in Eqs. (6)-(7); not measured or ablated in the paper.
  • domain assumption Instruction-following knowledge is language-agnostic and stored mainly in intermediate layers, while vocabulary knowledge is stored in input embeddings and the LM head.
    Borrowed from Artetxe et al. and prior adaptation work (Sections 2.2 and 4.2); needed for LEP to transfer knowledge.
  • domain assumption Overlapping token embeddings between base and instruct-tuned models can estimate alignment transforms D and U via least-squares pseudo-inversion.
    Eqs. (8)-(14); depends on non-invertible matrices and likely noisy least-squares solutions, with no validation.
  • ad hoc to paper Wikipedia-article copying during calibration recalls pretraining memory and propagates signals to new embeddings.
    Mechanism assumed in Section 2.4.2; no direct evidence that copy examples act specifically on embeddings.
  • domain assumption MERA and MMLU open splits are a fair evaluation target for Russian adaptation and no evaluated model trained on them.
    Underlies Darumeru validity; the paper itself suspects Vikhr and Qwen leakage, so the assumption is contested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Facilitating large language model Russian adaptation with Learned Embedding Propagation." pith.science (2026). https://pith.science/paper/DZMQTPXD

@misc{pith2026241221140,
  author       = {Pith},
  title        = {Pith review of: Facilitating large language model Russian adaptation with Learned Embedding Propagation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DZMQTPXD}},
  note         = {Machine review of arXiv:2412.21140}
}
read the original abstract

Rapid advancements of large language model (LLM) technologies led to the introduction of powerful open-source instruction-tuned LLMs that have the same text generation quality as the state-of-the-art counterparts such as GPT-4. While the emergence of such models accelerates the adoption of LLM technologies in sensitive-information environments the authors of such models don not disclose the training data necessary for replication of the results thus making the achievements model-exclusive. Since those open-source models are also multilingual this in turn reduces the benefits of training a language specific LLMs as improved inference computation efficiency becomes the only guaranteed advantage of such costly procedure. More cost-efficient options such as vocabulary extension and subsequent continued pre-training are also inhibited by the lack of access to high-quality instruction-tuning data since it is the major factor behind the resulting LLM task-solving capabilities. To address the limitations and cut the costs of the language adaptation pipeline we propose Learned Embedding Propagation (LEP). Unlike existing approaches our method has lower training data size requirements due to minimal impact on existing LLM knowledge which we reinforce using novel ad-hoc embedding propagation procedure that allows to skip the instruction-tuning step and instead implant the new language knowledge directly into any existing instruct-tuned variant. We evaluated four Russian vocabulary adaptations for LLaMa-3-8B and Mistral-7B, showing that LEP is competitive with traditional instruction-tuning methods, achieving performance comparable to OpenChat 3.5 and LLaMa-3-8B-Instruct, with further improvements via self-calibration and continued tuning enhancing task-solving capabilities.

Figures

Figures reproduced from arXiv: 2412.21140 by the authors.

Figure 1
Figure 1. Performance comparison of proposed adaptation method on Darumeru benchmark [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Micro average benchmark score dynamic throughout training [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. An example of generation using the OpenChat-3.5 model and its adapted versions. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 18 canonical work pages

  1. [1]

    Stanford alpaca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023

    2023

  2. [2]

    Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation

    Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation. arXiv preprint arXiv:2305.15011, 2023

    arXiv 2023

  3. [3]

    Polylm: An open source polyglot large language model

    Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. Polylm: An open source polyglot large language model. arXiv preprint arXiv:2307.06018, 2023

    arXiv 2023

  4. [4]

    rulm: A toolkit for training neural language models, 2023

    Ilya Gusev. rulm: A toolkit for training neural language models, 2023

    work page 2023

  5. [5]

    Teaching llama a new language through cross-lingual knowledge transfer

    Hele-Andra Kuulmets, Taido Purason, Agnes Luhtaru, and Mark Fishel. Teaching llama a new language through cross-lingual knowledge transfer. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3309–3325, 2024

    work page 2024

  6. [6]

    Extrapolating large language models to non-english by aligning languages

    Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Extrapolating large language models to non-english by aligning languages. arXiv preprint arXiv:2308.04948, 2023

    arXiv 2023

  7. [7]

    Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrations

    Leonardo Ranaldi, Giulia Pucci, and Andre Freitas. Empowering cross-lingual abilities of instruction-tuned large language models by translation-following demonstrations. arXiv preprint arXiv:2308.14186, 2023

    work page Pith review arXiv 2023

  8. [8]

    Improving in-context learning of multilingual generative language models with cross-lingual alignment

    Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Improving in-context learning of multilingual generative language models with cross-lingual alignment. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8051–8069, 2024

    work page 2024

Show all 36 references
  1. [9]

    xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning

    Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xiannian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, et al. xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning. arXiv preprint arXiv:2401.07037, 2024

    2024 arXiv

  2. [10]

    Romansetu: Efficiently unlocking multilingual capabilities of large language models via romanization

    J Jaavid, Raj Dabre, M Aswanth, Jay Gala, Thanmay Jayakumar, Ratish Puduppully, and Anoop Kunchukuttan. Romansetu: Efficiently unlocking multilingual capabilities of large language models via romanization. In Proceedings of the 62nd Annual Meeting of the Association for Comput...

    2024

  3. [11]

    Transfer learning in multilingual neural machine translation with dynamic vocabulary

    Surafel M Lakew, Aliia Erofeeva, Matteo Negri, Marcello Federico, and Marco Turchi. Transfer learning in multilingual neural machine translation with dynamic vocabulary. In International Workshop on Spoken Language Translation, 2018

    2018

  4. [12]

    Adaptation of deep bidirectional multilingual transformers for russian language

    Y Kuratov and M Arkhipov. Adaptation of deep bidirectional multilingual transformers for russian language. In Komp’juternaja Lingvistika i Intellektual’nye Tehnologii, pages 333–339, 2019. 7https://huggingface.co/RefalMachine 8https://github.com/RefalMachine/ruadapt 9https://g...

    2019

  5. [13]

    How good is your tokenizer? on the monolingual performance of multilingual language models

    Phillip Rust, Jonas Pfeiffer, Ivan Vuli´c, Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th I...

    2021

  6. [14]

    Cino: A chinese minority pre-trained language model

    Ziqing Yang, Zihang Xu, Yiming Cui, Baoxin Wang, Min Lin, Dayong Wu, and Zhigang Chen. Cino: A chinese minority pre-trained language model. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3937–3949, 2022

    2022

  7. [15]

    As good as new

    Wietse de Vries and Malvina Nissim. As good as new. how to successfully recycle english gpt-2 to make models for other languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 836–846, 2021

    2021

  8. [16]

    Impact of tokenization on llama russian adaptation

    Mikhail Tikhomirov and Daniil Chernyshev. Impact of tokenization on llama russian adaptation. In2023 Ivannikov Ispras Open Conference (ISPRAS), pages 163–168. IEEE, 2023

    2023

  9. [17]

    Improving large language model russian adaptation with preliminary vocabulary optimization

    MM Tikhomirov and DI Chernyshev. Improving large language model russian adaptation with preliminary vocabulary optimization. Lobachevskii Journal of Mathematics, 45(7):3211–3219, 2024

    2024

  10. [18]

    Efficient and effective text encoding for chinese llama and alpaca

    Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177, 2023

    2023 arXiv

  11. [19]

    Seallms–large language models for southeast asia

    Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, et al. Seallms–large language models for southeast asia. arXiv preprint arXiv:2312.00738, 2023

    2023 arXiv

  12. [20]

    Vikhr: The family of open-source instruction- tuned large language models for russian

    Aleksandr Nikolich, Konstantin Korolev, and Artem Shelmanov. Vikhr: The family of open-source instruction- tuned large language models for russian. arXiv preprint arXiv:2405.13929, 2024

    2024 arXiv

  13. [21]

    On the cross-lingual transferability of monolingual representations

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, 2020

    2020

  14. [22]

    Improving language plasticity via pretraining with active forgetting

    Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Adelani, Pontus Lars Erik Saito Stenetorp, Sebastian Riedel, and Mikel Artetxe. Improving language plasticity via pretraining with active forgetting. Advances in Neural Information Processing Systems, 36:31543–31557, 2023

    2023

  15. [23]

    Exploring design choices for building language-specific llms

    Atula Tejaswi, Nilesh Gupta, and Eunsol Choi. Exploring design choices for building language-specific llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10485–10500, 2024

    2024

  16. [24]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024

    2024

  17. [25]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  18. [26]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  19. [27]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2022

    2022

  20. [28]

    Dataset for automatic summarization of russian news

    Ilya Gusev. Dataset for automatic summarization of russian news. In Artificial Intelligence and Natural Language: 9th Conference, AINL 2020, Helsinki, Finland, October 7–9, 2020, Proceedings 9, pages 122–134. Springer, 2020

    2020

  21. [29]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  22. [30]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  23. [31]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  24. [32]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 16

    2023 arXiv

  25. [33]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  26. [34]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  27. [35]

    Mera: A comprehensive llm evaluation in russian

    Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetga- reeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, et al. Mera: A comprehensive llm evaluation in russian. arXiv preprint arXiv:2401.04531, 2024

    2024 arXiv

  28. [36]

    Rucola: Russian corpus of linguistic acceptability

    Vladislav Mikhailov, Tatiana Shamardina, Max Ryabinin, Alena Pestova, Ivan Smurov, and Ekaterina Artemova. Rucola: Russian corpus of linguistic acceptability. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5207–5227, 2022. 17

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.