Pith. sign in

REVIEW 5 major objections 6 minor 56 references

HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization

T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HYPEROFA claims that a hypernetwork trained on source-language tokens can generate target-language token embeddings that beat random initialization and match OFA's similarity-based initialization for expanding a pretrained model's…

desk verdict HYPEROFA is a modest, honest incremental method that roughly matches OFA rather than clearly beating it; the evidence is thin in places, and the BiLSTM's lack of permutation invariance is a real but addressable gap. read the letter →

arxiv 2504.21018 v2 pith:F7JAJKRD submitted 2025-04-21 cs.CL cs.LG

Enes Özeren , Yihong Liu , Hinrich Schütze This is my paper · ORCID
classification cs.CLcs.LG
keywords hypernetworkembeddinginitializationvocabularyexpansioncross-lingualtransfercontinualpre-trainingmultilinguallanguagemodelsfactorizedembeddingslow-resourcelanguages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a way to give pretrained language models embeddings for new tokens in languages they were not trained on: instead of designing similarity weights by hand, train a small hypernetwork to map external multilingual word vectors into the model's embedding space, then use it to generate the new embeddings. Trained only on source-language tokens (22K for RoBERTa, 103K for XLM-R), the hypernetwork is meant to transfer to unseen target-language tokens. The paper reports that this consistently beats random initialization and lands in the same range as OFA, the current similarity-based convex-combination method, on sentence retrieval, named-entity recognition, and part-of-speech tagging, both before and after continual pre-training. The practical stakes are that vocabulary expansion for mid- and low-resource languages could replace a similarity heuristic with a learned, more flexible mapping.

What carries the argument

The central object is the hypernetwork, a BiLSTM that takes a set of matched external word vectors $W_{\{s_i\}}$ for a source token $i$ and predicts that token's coordinate row $F_i^s$ in the SVD-factorized embedding matrix $E_s \approx F_s P$. Because the input is a set of variable length with no natural order, the BiLSTM is trained with random shuffling of the input word vectors as a form of order augmentation; the loss is $\lambda$ times a contrastive cosine term plus $(1-\lambda)$ times a normalized L1 term. After training, the same hypernetwork predicts coordinate rows for target tokens, with overlapping token rows copied directly from $F_s$ and unmatched tokens initialized from a Gaussian fitted to the source embedding statistics.

What would settle it

Take a held-out target language whose script has no overlap with the source vocabulary, generate its token embeddings with HYPEROFA, and continually pre-train the model from both the HYPEROFA initialization and a random initialization with identical data and compute; if random initialization converges to equal or better validation loss and downstream scores, the central claim fails. A cheaper check is to measure whether HYPEROFA's predicted embeddings for such unseen-script tokens have higher cosine similarity to the embeddings the model eventually learns than random initializations do.

Watch

Extended reading notes

Core claim

HYPEROFA argues that a BiLSTM hypernetwork can learn the nonlinear map from an external multilingual word-vector space to a pretrained language model's factorized embedding space, and that the embeddings it produces for target-language tokens are a better or equal starting point for continual pre-training than random initialization and than OFA's convex combinations of source embeddings. On the paper's 22-language continual pre-training runs, HYPEROFA matches or slightly exceeds OFA on multilingual XLM-R and on most monolingual RoBERTa metrics, with the clearest gains over random initialization appearing in the monolingual case. The differences between HYPEROFA and OFA are generally small, and the paper treats both as viable strategies while expecting HYPEROFA to improve more as hypernetwork and continual-pre-training data grow.

Load-bearing premise

The hypernetwork trained on source-language tokens generalizes to target-language tokens, including languages and scripts it never saw, even though its BiLSTM encoder is not permutation-invariant and relies on random shuffling to dampen order sensitivity.

Editorial extensions

If this is right

  • For monolingual models such as RoBERTa, wise initialization matters: both HYPEROFA and OFA converge faster and score higher after continual pre-training than random initialization, so vocabulary expansion without such initialization leaves clear performance on the table.
  • For a multilingual model such as XLM-R, the choice of initialization has little impact after continual pre-training; the model's multilingual pretraining largely absorbs randomly initialized new embeddings.
  • HYPEROFA's nonlinear mapping reaches OFA-level performance before continual pre-training, suggesting that the convex-combination constraint is not the main bottleneck limiting OFA in this experimental regime.
  • Because the hypernetwork is trained on only 22K to 103K source-language pairs, the paper expects its advantage over OFA to grow with more hypernetwork training data and longer continual pre-training, though that expectation is stated rather than demonstrated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test the paper does not run is to train on source tokens in one script family and evaluate the hypernetwork on completely unseen scripts, isolating whether the transfer is genuinely cross-lexical or just language-similarity based.
  • Because the BiLSTM is not permutation-invariant, replacing it with a set encoder such as a deep set or set transformer could improve robustness; the paper's own appendix reports that a transformer encoder without positional encodings underperformed on the limited data, so this is a data-scarcity hypothesis worth testing.
  • The method's cost scales steeply with embedding dimension (210M hypernetwork parameters for a factorized dimension of 400), so applying it to 4096-dimensional LLM embeddings would likely require shrinking the coordinate dimension or changing the hypernetwork architecture.
  • The paper only tests encoder-only models; the stated open direction is decoder-only and encoder-decoder LLMs, where vocabulary expansion for new languages is equally pressing and where the same initialization comparison could be run.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper proposes HYPEROFA, a hypernetwork-based method for initializing the embeddings of new target-language tokens when extending the vocabulary of a pretrained language model. The hypernetwork is a BiLSTM trained to map sets of external multilingual word vectors (from ColexNet+) to the factorized embedding space of a PLM (RoBERTa or XLM-R); at inference, it generates embeddings for new target tokens, which are then used as initialization before continual pre-training. The authors evaluate six configurations (three factorization dimensions for mono and multi setups) on sentence retrieval (SR-T, SR-B), NER, and POS before and after 4,000 steps of continual pre-training on 22 languages, comparing against random initialization and OFA. They report that HYPEROFA outperforms random initialization and performs comparably to OFA, with occasional gains.

Significance. If substantiated, HYPEROFA would provide a useful alternative to similarity-based convex combination for new token embedding initialization, with potentially higher expressiveness, and would be relevant to the growing area of vocabulary expansion for multilingual PLMs. The paper's strengths include making code public, following the established OFA evaluation protocol, covering many languages and tasks, and providing per-language breakdowns in the appendix. However, the current evidence is insufficient to support the stated claims: the comparison in Table 3 uses published OFA numbers rather than matched runs, no variance is reported, and the mono-100 results after continual pre-training contradict the 'matches or exceeds' claim. With additional experiments and a revised framing, the contribution could become a solid empirical study.

major comments (5)
  1. [Abstract and §5.2, Table 4] The claim that HYPEROFA 'matches or exceeds the performance of OFA' is contradicted by the paper's own results. In Table 4, after continual pre-training, HYPEROFA-mono-100 scores 11.3 on SR-T versus 15.2 for OFA, and 43.4 vs 45.7 on NER; Table 3 also shows HYPEROFA-mono-400 below OFA on all four tasks (e.g., 6.3 vs 7.2 on SR-T). The abstract and contribution bullet should be revised to a more accurate 'competitive' formulation, and the cases where HYPEROFA underperforms should be explained or at least explicitly acknowledged.
  2. [§3.4, Appendix A.3] The hypernetwork is a BiLSTM that does not satisfy the permutation-invariance requirement stated at the beginning of §3.4. The only mitigation is random shuffling during training, which does not guarantee that generated embeddings are independent of input order at inference. Because the paper specifies no canonical ordering of the set W{tj}, every embedding generated by the hypernetwork in §3.5, and hence every result in Tables 3 and 4 that depends on those embeddings, is potentially a function of an arbitrary implementation detail. Please add a permutation-sensitivity test (e.g., variance of generated embeddings across multiple random orderings) and either fix a canonical ordering or switch to a permutation-invariant architecture.
  3. [§5.1, Table 3] The OFA baseline scores in Table 3 are taken from Liu et al. (2024a) rather than reproduced in the same pipeline. This makes the headline before-continual-pretraining comparison non-controlled: HYPEROFA and Random are run under the authors' implementation while OFA numbers come from the original paper. Differences as small as 0.1–0.5 points cannot be attributed to the initialization method with confidence. The authors should reproduce OFA under identical conditions, even for a subset of configurations, or restrict the 'matches or exceeds' claim to the fully controlled Table 4 setting.
  4. [§4.2, §3.3] The hypernetwork is trained only on source-vocabulary token pairs (22K for RoBERTa, 103K for XLM-R) and then applied to target tokens from the Glot500 tokenizer covering many languages and scripts. The paper does not provide any per-script or per-language diagnostic of prediction quality; aggregated downstream averages (Tables 3 and 4) can hide systematic failures for particular scripts or languages not seen in hypernetwork training. Please report prediction accuracy or cosine similarity for target tokens broken down by script or by whether the target language was present in the hypernetwork training set.
  5. [Tables 3 and 4] No error bars, confidence intervals, or multiple seeds are reported anywhere in the empirical evaluation. Given that many HYPEROFA-vs-OFA differences in Table 3 are below one point (e.g., NER mono-100: 24.9 vs 25.0; SR-T mono-100: 6.4 vs 6.2), single-run results cannot support the claim that HYPEROFA 'consistently outperforms' the baselines. At minimum, the six continually pre-trained models in Table 4 should be run with at least three seeds and the mean and standard deviation reported; ideally the Table 3 comparisons should be re-run with matched OFA baselines.
minor comments (6)
  1. [Table 1] The second occurrence of 'HYPEROFA-multi-100' should be 'HYPEROFA-multi-200'; the corresponding hypernetwork HN-X-200 is listed in Appendix Table 6.
  2. [§3.4] The definition of NEG in the contrastive loss is not typeset cleanly; the summation over k should be written explicitly with the correct argument to avoid ambiguity.
  3. [Abstract and Conclusion] The abstract uses 'matches or exceeds the performance of OFA' while the conclusion says 'performs competitively with OFA'; align the wording to reflect the actual results.
  4. [§4.2] In the text, 'HYPEROFA-mutli-400' is a typo for 'HYPEROFA-multi-400'.
  5. [Appendix A.2] Appendix A.2 reports λ=0.1 and τ=0.5/0.25 but contains no sensitivity analysis for these hyperparameters; since they are free parameters chosen on validation cosine similarity, a brief sensitivity study would strengthen the method.
  6. [Figure 2] Figure 2 shows a single validation curve for one configuration; the claim that 'larger hypernetworks perform better' would benefit from quantitative results at convergence rather than a single curve.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: HYPEROFA trains a hypernetwork on source-token pairs and evaluates generated target embeddings on held-out tasks; self-citations are baseline/resource citations, not load-bearing reductions.

full rationale

The derivation chain is self-contained. The paper factorizes the source embedding matrix via SVD into Fs and P, constructs training pairs (W{s_i}, F^s_i) from source tokens, trains HNθ to minimize L = λ·Lc + (1−λ)·L1 against ground-truth source coordinate embeddings, and then generates target coordinate embeddings as HNθ(W{t_j}) without any fitting to target-language labels or to the downstream evaluation metrics. The prediction step is therefore a genuine held-out application of a learned mapping, not a renamed fit. The OFA scores in Table 3 are taken directly from Liu et al. (2024a), but that is benchmark reuse rather than a circular derivation: the HYPEROFA and Random rows are newly produced under the same setup, and Table 4 re-runs OFA. Citations to OFA, ColexNet+, and Glot500 are external resources and baselines; they do not supply the paper's central claim that the hypernetwork mapping improves initialization. The acknowledged BiLSTM permutation-invariance limitation is a potential robustness concern, not a case where the target embeddings reduce by construction to the training inputs. The stated limitations (encoder-only models, low embedding dimensions, small continual-pretraining corpus) affect scope and generalization but do not make the method circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several domain assumptions about external word vector quality, subword matching, transfer from source to target languages, and the factorization. The hypernetwork itself is a standard architecture, not a new physical entity. The free parameters are hyperparameters chosen during development without a systematic search on the target tasks.

free parameters (4)
  • Loss weight lambda = 0.1 for all hypernetworks
    Chosen by hand in Appendix A.2 to balance contrastive and L1 losses; affects the learned mapping quality.
  • Temperature tau = 0.5 for mono, 0.25 for multi
    Set in Appendix A.2; influences contrastive loss sharpness and embedding similarity distribution.
  • Factorized embedding dimension D' = 100, 200, 400
    Chosen to compare efficiency and performance; the central comparisons after continual pre-training use D'=100 for mono and D'=400 for multi.
  • BiLSTM hidden size and layers = 2-4 layers, 800-1600 hidden
    Selected to improve validation cosine similarity (Figure 2); larger hypernetworks used for higher embedding dimensions.
assumptions (4)
  • domain assumption The external multilingual word vectors (ColexNet+) are well-aligned across languages.
    The method relies on these vectors to create matched word sets for both source and target tokens; if the alignment is poor, the hypernetwork input is noisy. Invoked in Section 3.3.
  • domain assumption Token-to-word matching by subword containment is a sufficient signal for initialization.
    A target token is matched with an external word if the word contains the token as a subword; this heuristic may miss or wrongly match tokens for languages with different morphology. Invoked in Section 3.3.
  • domain assumption A hypernetwork trained on source tokens generalizes to target tokens from unseen languages.
    The entire method depends on this transfer; the paper does not provide direct evidence beyond downstream task performance. Stated in Section 3.4 and used in Section 3.5.
  • domain assumption The factorized embedding representation Fs P preserves enough information for the hypernetwork to learn a good mapping.
    The hypernetwork predicts the coordinate matrix Fs, not the full embeddings; if P and Fs are not well-behaved, the predicted F may not restore useful embeddings. Invoked in Section 3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization." pith.science (2026). https://pith.science/paper/F7JAJKRD

@misc{pith2026250421018,
  author       = {Pith},
  title        = {Pith review of: HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F7JAJKRD}},
  note         = {Machine review of arXiv:2504.21018}
}
read the original abstract

Many pre-trained language models (PLMs) exhibit suboptimal performance on mid- and low-resource languages, largely due to limited exposure to these languages during pre-training. A common strategy to address this is to introduce new tokens specific to the target languages, initialize their embeddings, and apply continual pre-training on target-language data. Among such methods, OFA (Liu et al., 2024a) proposes a similarity-based subword embedding initialization heuristic that is both effective and efficient. However, OFA restricts target-language token embeddings to be convex combinations of a fixed number of source-language embeddings, which may limit expressiveness. To overcome this limitation, we propose HYPEROFA, a hypernetwork-based approach for more adaptive token embedding initialization. The hypernetwork is trained to map from an external multilingual word vector space to the PLMs token embedding space using source-language tokens. Once trained, it can generate flexible embeddings for target-language tokens, serving as a good starting point for continual pretraining. Experiments demonstrate that HYPEROFA consistently outperforms random initialization baseline and matches or exceeds the performance of OFA in both continual pre-training convergence and downstream task performance. We make the code publicly available.

Figures

Figures reproduced from arXiv: 2504.21018 by the authors.

Figure 1
Figure 1. HYPEROFA pipeline. The source model (left) transfers weights to the target model (right). The target embeddings are initialized by first copying embeddings for matching tokens, then generating embeddings via a hypernetwork for tokens with matching external words, and finally randomly initializing the rest. matched word vectors and coordinate vector in F s for token i in V s , respectively.5 HNθ then takes W{si} as i… view at source ↗
Figure 2
Figure 2. Comparison of large (210M parameters) and [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Training loss curves during the continual pre [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 6 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee

    David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee. 2024. https://aclanthology.org/2024.eacl-long.14/ SIB -200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects . In Proceedings of the 18th Conference of the European Chap...

    2024

  4. [4]

    Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow

    Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. https://aclanthology.org/2022.coling-1.382/ Adapting pre-trained language models to A frican languages via multilingual adaptive fine-tuning . In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336--4349, Gyeongju, Republic of Korea....

    work page 2022

  5. [5]

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. https://doi.org/10.18653/v1/2020.acl-main.421 On the cross-lingual transferability of monolingual representations . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623--4637, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.421 2020

  6. [6]

    Mikel Artetxe and Holger Schwenk. 2019. https://doi.org/10.1162/tacl_a_00288 Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond . Transactions of the Association for Computational Linguistics, 7:597--610

    doi:10.1162/tacl_a_00288 2019

  7. [7]

    Gunjan Balde, Soumyadeep Roy, Mainack Mondal, and Niloy Ganguly. 2024. https://www.ijcai.org/proceedings/2024/683 MEDVOC: vocabulary adaptation for fine-tuning pre-trained language models on medical text summarization . In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, ...

    work page 2024

  8. [8]

    Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton. 2024. A brief review of hypernetworks in deep learning. Artificial Intelligence Review, 57(9):250

    2024

Show all 56 references
  1. [9]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning ...

    2020 doi

  2. [10]

    Yiming Cui, Ziqing Yang, and Xin Yao. 2024. https://arxiv.org/abs/2304.08177 Efficient and effective text encoding for chinese llama and alpaca . Preprint, arXiv:2304.08177

    2024 arXiv

  3. [11]

    Manning, Joakim Nivre, and Daniel Zeman

    Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. https://doi.org/10.1162/coli_a_00402 U niversal D ependencies . Computational Linguistics, 47(2):255--308

    2021 doi

  4. [12]

    Wietse de Vries and Malvina Nissim. 2021. https://doi.org/10.18653/v1/2021.findings-acl.74 As good as new. how to successfully recycle E nglish GPT -2 to make models for other languages . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 836-...

    2021 doi

  5. [13]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  6. [14]

    Konstantin Dobler and Gerard de Melo. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.829 FOCUS : Effective embedding initialization for monolingual specialization of multilingual models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process...

    2023 doi

  7. [15]

    Philip Gage. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23--38

    1994

  8. [16]

    Dai, and Quoc V

    David Ha, Andrew M. Dai, and Quoc V. Le. 2017. https://openreview.net/forum?id=rkpACe1lx Hypernetworks . In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net

    2017

  9. [17]

    John Hewitt. 2021. https:/nlp.stanford.edu/ johnhew//vocab-expansion.html Initializing new word embeddings for pretrained language models

    2021

  10. [18]

    Ayyoob Imani, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, Andr \'e Martins, Fran c ois Yvon, and Hinrich Sch \"u tze. 2023. https://doi.org/10.18653/v1/2023.acl-long.61 Glot500: Scaling multilingual corpora ...

    2023 doi

  11. [19]

    Taku Kudo. 2018. https://doi.org/10.18653/v1/P18-1007 Subword regularization: Improving neural network translation models with multiple subword candidates . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...

    2018 doi

  12. [20]

    Taku Kudo and John Richardson. 2018. https://doi.org/10.18653/v1/D18-2012 S entence P iece: A simple and language independent subword tokenizer and detokenizer for neural text processing . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processin...

    2018 doi

  13. [21]

    Anastasios Lamproudis, Aron Henriksson, and Hercules Dalianis. 2022. https://doi.org/10.5220/0010893800003123 Vocabulary modifications for domain-adaptive pretraining of clinical language models . In Proceedings of the 15th International Joint Conference on Biomedical Engineer...

    2022 doi

  14. [22]

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. https://openreview.net/forum?id=H1eA7AEtvS ALBERT: A lite BERT for self-supervised learning of language representations . In 8th International Conference on Learning Representat...

    2020

  15. [23]

    Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.813 XLM - V : Overcoming the vocabulary bottleneck in multilingual masked language models . In Proceedings of...

    2023 doi

  16. [24]

    Siyang Liu, Naihao Deng, Sahand Sabour, Yilin Jia, Minlie Huang, and Rada Mihalcea. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.944 Task-adaptive tokenization: Enhancing long-form text generation efficacy in mental health and beyond . In Proceedings of the 2023 Confer...

    2023 doi

  17. [25]

    Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schuetze. 2024 a . https://doi.org/10.18653/v1/2024.findings-naacl.68 OFA : A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining . In Findings of the Association fo...

    2024 doi

  18. [26]

    Yihong Liu, Chunlan Ma, Haotian Ye, and Hinrich Schuetze. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.136 T ransli C o: A contrastive learning framework to address the script barrier in multilingual pretrained language models . In Proceedings of the 62nd Annual Meeting ...

    2024 doi

  19. [27]

    Yihong Liu, Chunlan Ma, Haotian Ye, and Hinrich Sch \"u tze. 2025. https://aclanthology.org/2025.coling-main.32/ T rans MI : A framework to create strong baselines from multilingual pretrained language models for transliterated data . In Proceedings of the 31st International C...

    2025

  20. [28]

    Yihong Liu, Haotian Ye, Leonie Weissweiler, Renhao Pei, and Hinrich Schuetze. 2023 b . https://doi.org/10.18653/v1/2023.findings-emnlp.562 Crosslingual transfer learning for low-resource languages based on multilingual colexification graphs . In Findings of the Association for...

    2023 doi

  21. [29]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach

    2019 arXiv

  22. [30]

    Kelly Marchisio, Patrick Lewis, Yihong Chen, and Mikel Artetxe. 2023. https://doi.org/10.18653/v1/2023.findings-acl.338 Mini-model adaptation: Efficiently extending pretrained models to new languages via aligned shallow training . In Findings of the Association for Computation...

    2023 doi

  23. [31]

    Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. https://doi.org/10.18653/v1/2022.naacl-main.293 WECHSEL : Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models . In Proceedings of the 2022 Conference of the Nor...

    2022 doi

  24. [32]

    Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vuli \'c . 2024. Zero-shot tokenizer transfer. arXiv preprint arXiv:2405.07883

    2024

  25. [33]

    Ibraheem Muhammad Moosa, Mahmud Elahi Akhter, and Ashfia Binte Habib. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.50 Does transliteration help multilingual language modeling? In Findings of the Association for Computational Linguistics: EACL 2023, pages 670--685, Dubr...

    2023 doi

  26. [34]

    Benjamin Muller, Antonios Anastasopoulos, Beno \^i t Sagot, and Djam \'e Seddah. 2021. https://doi.org/10.18653/v1/2021.naacl-main.38 When being unseen from m BERT is just the beginning: Handling new languages with multilingual language models . In Proceedings of the 2021 Conf...

    2021 doi

  27. [35]

    Nandini Mundra, Aditya Nanda Kishore Khandavally, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, and Mitesh M Khapra. 2024. https://doi.org/10.18653/v1/2024.conll-1.8 An empirical comparison of vocabulary expansion and initialization approaches for language models . In Proc...

    2024 doi

  28. [36]

    Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. https://doi.org/10.18653/v1/P17-1178 Cross-lingual name tagging and linking for 282 languages . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (...

    2017 doi

  29. [37]

    Jonas Pfeiffer, Ivan Vuli \'c , Iryna Gurevych, and Sebastian Ruder. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.800 UNK s everywhere: A dapting multilingual language models to new scripts . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P...

    2021 doi

  30. [38]

    Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. https://doi.org/10.18653/v1/D17-1010 Mimicking word embeddings using subword RNN s . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 102--112, Copenhagen, Denmark. Associ...

    2017 doi

  31. [39]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  32. [40]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  33. [41]

    Timo Schick and Hinrich Sch \"u tze. 2020. https://doi.org/10.18653/v1/2020.acl-main.368 BERTRAM : Improved word embeddings have big impact on contextualized model performance . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3...

    2020 doi

  34. [42]

    Mike Schuster and Kaisuke Nakajima. 2012. https://doi.org/10.1109/ICASSP.2012.6289079 Japanese and korean voice search . In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2012, Kyoto, Japan, March 25-30, 2012 , pages 5149--5152. IEEE

    2012

  35. [43]

    Mike Schuster and Kuldip K. Paliwal. 1997. https://doi.org/10.1109/78.650093 Bidirectional recurrent neural networks . IEEE Transactions on Signal Processing , 45(11):2673--2681

    1997 doi

  36. [44]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  37. [45]

    Ke Tran. 2020. From english to foreign languages: Transferring pre-trained language models. arXiv preprint arXiv:2002.07306

    2020 arXiv

  38. [46]

    Ahmet \"U st \"u n, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D ' souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. ht...

    2024 doi

  39. [47]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  40. [48]

    Tomer Volk, Eyal Ben-David, Ohad Amosy, Gal Chechik, and Roi Reichart. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.610 Example-based hypernetworks for multi-source adaptation to unseen domains . In Findings of the Association for Computational Linguistics: EMNLP 2023...

    2023 doi

  41. [49]

    Johannes von Oswald, Christian Henning, Jo \ a o Sacramento, and Benjamin F. Grewe. 2020. https://openreview.net/forum?id=SJgwNerKvB Continual learning with hypernetworks . In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-...

    2020

  42. [50]

    Shijie Wu and Mark Dredze. 2020. https://doi.org/10.18653/v1/2020.repl4nlp-1.16 Are all languages created equal in multilingual BERT ? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120--130, Online. Association for Computational Linguistics

    2020 doi

  43. [51]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, and 1 others. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXi...

    2016 arXiv

  44. [52]

    Orgest Xhelili, Yihong Liu, and Hinrich Schuetze. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.659 Breaking the script barrier in multilingual pre-trained language models with transliteration-based post-training alignment . In Findings of the Association for Computati...

    2024 doi

  45. [53]

    Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. 2024 a . https://doi.org/10.18653/v1/2024.findings-emnlp.396 An empirical study on cross-lingual vocabulary adaptation for efficient language model inference . In Findings of the Association for Computational Linguis...

    2024 doi

  46. [54]

    Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. 2024 b . How can we effectively expand the vocabulary of llms with 0.01 gb of target language text? arXiv preprint arXiv:2406.11477

    2024

  47. [55]

    Haotian Ye, Yihong Liu, Chunlan Ma, and Hinrich Sch \"u tze. 2024. https://doi.org/10.18653/v1/2024.insights-1.1 M o SEC ro T : Model stitching with static word embeddings for crosslingual zero-shot transfer . In Proceedings of the Fifth Workshop on Insights from Negative Resu...

    2024 doi

  48. [56]

    Qingcheng Zeng, Lucas Garay, Peilin Zhou, Dading Chong, Yining Hua, Jiageng Wu, Yikang Pan, Han Zhou, Rob Voigt, and Jie Yang. 2023. https://doi.org/10.24963/IJCAI.2023/698 Greenplm: Cross-lingual transfer of monolingual pre-trained language models at almost no cost . In Proce...

    2023 doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.