REVIEW 3 major objections 5 minor 16 references
Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for Estonian
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read For Estonian, pruning mDeBERTa's unused vocabulary tokens cuts about 23% of model parameters with no NER performance loss, while a retrained 32K tokenizer drops accuracy.
desk verdict Solid empirical comparison of tokenizer retraining vs pruning for Estonian, but the 'no negative effects' claim for pruning outruns the statistics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is vocabulary surgery followed by continual training. Pruning keeps the original tokenizer and embeddings for retained tokens, removing only tokens never seen in the ENC train split, so no new vectors enter the model. Retraining builds a new 32K SentencePiece vocabulary, copies embeddings for shared tokens, initializes unseen tokens with the mean of the original embedding matrix, then tunes embeddings for one epoch and continues with LoRA-based MLM training. The contrast shows that preserving the original embedding and tokenization structure is what protects downstream NER performance.
What would settle it
Count how many tokens in EstNER's test set, especially named-entity spans, are absent from the pruned 169K vocabulary. If a material fraction of test entities no longer appears as a single token and must be split into subwords, the pruned model's parity with the base model should degrade; alternatively, re-run the pruning experiment using only a non-Wikipedia subset of ENC and check whether EstNER F1 drops.
Extended reading notes
Core claim
The paper's central claim is that for mDeBERTa v3 base applied to Estonian, removing vocabulary tokens that never appear in the Estonian National Corpus is a free lunch: it shrinks the model from 279M to 215M parameters (about 23%) and leaves EstNER F1 at 80.62 ± 0.12, statistically indistinguishable in practice from the unmodified model's 80.96 ± 0.19. By contrast, training a new 32K-token SentencePiece tokenizer on the same corpus and initializing new embeddings as the mean of the original embedding matrix degrades NER performance to 76.40 ± 0.23 after one epoch of embedding-only tuning, and to 77.58 ± 0.47 after LoRA continual training; the authors attribute the gap to insufficient embedding tuning and note that longer training might close it. The paper also reports that MLM accuracy improves substantially with LoRA even when NER F1 does not, so the two metrics are not aligned across vocabularies.
Load-bearing premise
The conclusion that pruning loses nothing assumes that the corpus used to decide which tokens to delete contains every word that matters in the evaluation data; if the test set has rare names or terms the corpus never saw, those words get broken into pieces and the pruned model could be worse.
Editorial extensions
If this is right
- A practitioner can cut about 23% of mDeBERTa v3 base parameters for an Estonian use case by pruning never-seen vocabulary tokens, with NER F1 staying at the baseline level.
- Embedding tuning after pruning is unnecessary: the pruned model with untouched embeddings already matches the base model, and tuning adds no gain.
- A retrained 32K Estonian tokenizer needs substantially more than one epoch of embedding tuning and one epoch of LoRA training before it can be judged; the paper's protocol is insufficient to realize its potential.
- MLM accuracy should not be used as a proxy for downstream NER quality when comparing models with different vocabularies, since LoRA raised MLM accuracy from 25.84 to 38.42 for the pruned model while NER F1 stayed flat.
- The pruned model retains the base tokenizer, so its gains come from memory and parameter savings, not from shorter input sequences; sequence-length savings require the retrained tokenizer.
Reading between the lines
- If the pruned vocabulary is applied to Estonian text outside the ENC distribution, the assumption of full coverage weakens: rare named entities absent from ENC would be decomposed into subwords, and the measured parity might not hold on domain-specific NER test sets.
- The 23% parameter reduction could translate into lower GPU memory and faster embedding lookups, but inference latency would barely change because token counts per word stay at 2.23; combining pruning with a distilled or quantized model would compound the savings.
- The retrained 32K tokenizer's 1.75 tokens per word suggests that, with more adaptation data, it could eventually beat the pruned model on both efficiency and accuracy; a direct test would be to train embeddings for several epochs or increase LoRA training before evaluating.
- The observed decoupling of MLM accuracy and NER F1 suggests that vocabulary-specific MLM scores are not comparable across tokenizers, and any future comparison should report tokenization-adjusted metrics or downstream task performance.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript compares two vocabulary-adaptation strategies for mDeBERTa v3 base on Estonian: retraining a 32K SentencePiece tokenizer on the Estonian National Corpus (ENC), and pruning every vocabulary token that never occurs in the ENC train split. It reports tokenizer efficiency (vocabulary size, parameter count, tokens per word), MLM accuracy, and EstNER F1 after embedding-only tuning and after LoRA-based continual training. The main findings are that retraining degrades NER performance (F1 76.40 ± 0.23 after embedding tuning; 77.58 ± 0.47 after LoRA) despite more compact tokenization, while pruning reduces parameters by about 23% and yields F1 scores close to the base model (80.62 ± 0.12 vs 80.96 ± 0.19).
Significance. If the pruning result holds, it is practically valuable: it would provide a cheap way to shrink a multilingual encoder for a target language without retraining. The study has concrete strengths: experiments are repeated three times with reported standard deviations, the comparisons use public datasets and publicly available models, and the authors explicitly acknowledge that MLM accuracy is not directly comparable across vocabularies. The retraining degradation is large and consistent, and the tokenization-efficiency measurements are useful. However, the paper's central 'no negative effects' claim currently rests on a small, untested F1 gap and on an implicit assumption that the ENC train split covers the tokens that matter in the EstNER evaluation text. Because these two points are load-bearing for the main conclusion, the current evidence is not yet sufficient for an unconditional claim.
major comments (3)
- [§5, Table 2] The claim that pruning has 'no negative effects' is stronger than the reported statistics. The base mDeBERTa-base has F1 80.96 ± 0.19 and the pruned model has F1 80.62 ± 0.12 with three runs each; the 0.34-point difference is not subjected to any statistical test. A pooled two-sample t-test would be borderline (roughly p ≈ 0.06), so the observed gap cannot simply be dismissed as noise, and a non-significant result would not by itself establish equivalence. The authors should report an appropriate significance test or confidence interval for the pruning-vs-base comparison, or alternatively soften the wording to 'no observable negative effect in this experiment' and explicitly characterize the precision of the estimate.
- [§3 (Tokenizer Pruning) and §4 (Models Developed)] The pruning criterion is based entirely on token occurrence in the ENC train split, and the paper never verifies that the EstNER evaluation text is covered by the retained vocabulary. If a named-entity token in EstNER was absent from ENC and was a single vocabulary item in the base tokenizer, pruning deletes it and forces retokenization into subwords on exactly the spans that the NER model must label. The observed parity between pruned and base models could then be specific to this particular train/evaluation split rather than a general property of pruning. To support the central claim, the authors should quantify coverage of the EstNER tokens (especially entity mentions) by the pruned vocabulary, or otherwise measure how often the pruned tokenizer changes the model input relative to the base tokenizer.
- [§6 (Discussion)] The statement that 'a single epoch of embedding tuning was insufficient' to restore performance after tokenizer retraining is offered as a likely explanation, but no training curve or checkpoint evaluation is shown to support it. Since the paper already reports that only one epoch was used, a direct test would be to evaluate the retrained-tokenizer model at intermediate embedding-tuning checkpoints, or at least to report the trajectory of NER F1 as a function of tuning steps. Without such evidence, the 'longer tuning would help' claim remains speculative rather than an empirical finding.
minor comments (5)
- [§5 (Results)] The text contains a typo: 'little bit worse' should be 'a little bit worse.'
- [§4 (Fine-tuning on NER)] The text says the model has to produce classification scores for individual 'tokes'; this should be 'tokens.'
- [§3 (Continuous Training with LoRA)] Calling the LoRA phase 'full model training' is imprecise, since LoRA updates only low-rank adapters while most parameters remain frozen; consider phrasing such as 'continual training with LoRA adapters.'
- [§4 (Fine-tuning on NER)] The paper does not specify whether the reported F1 is micro-averaged or macro-averaged, or whether it is computed at the token level or entity level; this should be stated for reproducibility.
- [Table 1] The row label 'mDeBERTa base Tuned' is ambiguous because both the retrained-tokenizer model and embedding-tuned models are 'tuned'; a caption or footnote should clarify that this row denotes the model with the retrained 32K tokenizer.
Circularity Check
No significant circularity: the paper's claims are empirical comparisons against external benchmarks, not derivations from fitted inputs.
full rationale
This paper is an empirical study comparing vocabulary retraining and pruning for mDeBERTa v3 on Estonian NER. There is no derivation chain in which a claimed prediction is constructed from its own target, and no fitted parameter is renamed as a prediction. The pruning method removes tokens absent from the ENC train split, and the resulting model is then evaluated on the independent EstNER benchmark; this is a standard train/evaluation separation, even if the coverage of EstNER by the pruned vocabulary could be examined more carefully. The retrained-tokenizer embeddings are initialized from the base model's embedding mean, and the model is then fine-tuned on EstNER; again, the evaluation is external. The paper explicitly acknowledges that MLM accuracy scores across different vocabularies are not directly comparable, which reduces rather than creates circularity. Self-citations, such as the EstNER dataset (Sirts, 2023) and EstBERT (Tanvir et al., 2020), are used as benchmarks or baselines, not as load-bearing justifications for the paper's conclusions. Even the worrisome 'no negative effects' claim is an empirical observation about measured F1, not a result made true by construction. Therefore, no circular step is present, and the appropriate score is 0.
Assumptions & free parameters
free parameters (5)
- Retrained tokenizer vocabulary size =
32,000 tokens
- Pruning coverage criterion =
keep all tokens appearing in ENC train split (no frequency threshold)
- Embedding tuning duration =
1 epoch over ENC (3.4B tokens)
- LoRA continuous training duration =
1 epoch over ENC
- LoRA rank and scaling factor =
rank=4, alpha=32
assumptions (4)
- domain assumption mDeBERTa v3 base is a representative state-of-the-art multilingual encoder for this study.
- domain assumption The ENC train split is representative of Estonian text needed for EstNER.
- domain assumption EstNER F1 is a sufficient downstream measure of vocabulary quality.
- domain assumption LoRA MLM training approximates full continual pretraining for vocabulary adaptation.
Cite this review
Pith. "Pith review of Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for Estonian." pith.science (2026). https://pith.science/paper/4FOCGFXK
@misc{pith2026250102631,
author = {Pith},
title = {Pith review of: Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for Estonian},
year = {2026},
howpublished = {\url{https://pith.science/paper/4FOCGFXK}},
note = {Machine review of arXiv:2501.02631}
}
read the original abstract
Adapting multilingual language models to specific languages can enhance both their efficiency and performance. In this study, we explore how modifying the vocabulary of a multilingual encoder model to better suit the Estonian language affects its downstream performance on the Named Entity Recognition (NER) task. The motivations for adjusting the vocabulary are twofold: practical benefits affecting the computational cost, such as reducing the input sequence length and the model size, and performance enhancements by tailoring the vocabulary to the particular language. We evaluate the effectiveness of two vocabulary adaptation approaches -- retraining the tokenizer and pruning unused tokens -- and assess their impact on the model's performance, particularly after continual training. While retraining the tokenizer degraded the performance of the NER task, suggesting that longer embedding tuning might be needed, we observed no negative effects on pruning.
Reference graph
Works this paper leans on
-
[1]
Amine Abdaoui, Camille Pradel, and Gr \'e goire Sigel. 2020. https://doi.org/10.18653/v1/2020.sustainlp-1.16 Load W hat Y ou N eed: S maller V ersions of M utililingual BERT . In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 119--123, Online. Association for Computational Linguistics
-
[2]
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 U nsupervised C ross-lingual R epresentation L earning at S cale . In Proceedings of the 58th Annual Meeting of the Association for...
-
[3]
Zoltan Csaki, Bo Li, Jonathan Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. 2024. Sambalingo: T eaching large language models new languages. arXiv preprint arXiv:2404.05829
arXiv 2024
-
[4]
Gautier Dagan, Gabriel Synnaeve, and Baptiste Roziere. 2024. https://openreview.net/forum?id=ZFYBnLljtT Getting the most out of your tokenizer for pre-training and domain adaptation . In Forty-first International Conference on Machine Learning
work page 2024
-
[5]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : P re-training of D eep B idirectional T ransformers for L anguage U nderstanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (...
-
[6]
Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. 2022. https://doi.org/10.18653/v1/2022.emnlp-industry.41 F ast V ocabulary T ransfer for L anguage M odel C ompression . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 409--416, Abu Dhabi, UAE. Association for Computationa...
-
[7]
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. https://arxiv.org/abs/2111.09543 DeBERTaV3 : I mproving DeBERTa using ELECTRA - S tyle P re- T raining with G radient- D isentangled E mbedding S haring . Preprint, arXiv:2111.09543
arXiv 2023
-
[8]
Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685
arXiv 2021
Show all 16 references
-
[9]
Yuri Kuratov and Mikhail Arkhipov. 2019. Adaptation of deep bidirectional multilingual transformers for R ussian language. arXiv preprint arXiv:1905.07213
2019 arXiv
-
[10]
Robert Mroczkowski, Piotr Rybak, Alina Wr \'o blewska, and Ireneusz Gawlik. 2021. H er BERT : E fficiently P retrained T ransformer-based L anguage model for P olish. In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pages 1--10
2021
-
[11]
Kairit Sirts. 2023. E stonian N amed E ntity R ecognition: N ew D atasets and M odels. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 752--761, T \'o rshavn, Faroe Islands. University of Tartu Library
2023
-
[12]
Hasan Tanvir, Claudia Kittask, and Kairit Sirts. 2020. https://arxiv.org/abs/2011.04784 EstBERT : A P retrained L anguage- S pecific BERT for E stonian . Preprint, arXiv:2011.04784
2020 arXiv
-
[13]
Atula Tejaswi, Nilesh Gupta, and Eunsol Choi. 2024. Exploring D esign C hoices for B uilding L anguage- S pecific LLM s. arXiv preprint arXiv:2406.14670
2024 arXiv
-
[14]
University of Tartu . 2018. https://doi.org/10.23673/PH6N-0144 UT R ocket
2018 doi
-
[15]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[16]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.