{"id":"a75893d7-446f-44bd-ad79-c34e57b32a77","arxiv_id":"2501.02631","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Pruning unused tokens from a multilingual encoder's vocabulary for Estonian preserves named-entity recognition F1 and cuts model size, while a retrained 32K tokenizer degrades performance under the tested training budget.","lead":"This paper tests two ways to shrink the vocabulary of a multilingual AI model for Estonian: training a new tokenizer or pruning unused tokens. Pruning cut model size by 23% without hurting named-entity recognition, while the retrained tokenizer lost performance even after extra training.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No-negative-effect conclusion for pruning is not yet established: the 0.34-F1 gap is untested for significance and the pruned tokenizer's coverage of EstNER entities is unreported.","rationale":"I read the paper as a focused empirical comparison: retraining the tokenizer to 32K and pruning to 169K, evaluated by EstNER F1. The retrained-tokenizer degradation is plausible, honestly caveated, and consistent with prior results on retraining cost. The pruned model's parameter reduction (23%) is real and useful. The single load-bearing gap is the pruning claim. The F1 difference is small, but absence of evidence is not evidence of absence; with three seeds and no significance test, the paper cannot distinguish 'no effect' from 'small negative effect.' The representativeness issue is the likely mechanism by which a small negative effect would appear: tokens absent from a 3.4B-token corpus are exactly rare named entities and domain terms, and those are the tokens NER systems must classify. The paper does not report any coverage statistic on EstNER, so the reader's weakest assumption remains open. I do not see an internal inconsistency or an unsupported retraining claim; the main issue is the overstrong wording of the pruning conclusion.","tokens_in":6129,"tokens_out":5519,"duration_ms":57041,"concrete_test":"On EstNER, compute the OOV rate of the 169K pruned tokenizer overall and specifically on gold entity mentions: for each EstNER token, check whether it is present in the pruned vocabulary; for missing tokens, compare the subword segmentation produced by the original 250K tokenizer and the pruned tokenizer. Separately, run a paired bootstrap over EstNER test examples (or a t-test over the three fine-tuning seeds) for base versus pruned F1. If the OOV/entity-retokenization rate is near zero and the 95% confidence interval for the F1 difference includes zero, the concern does not land; if either result goes the other way, the conclusion should be weakened to 'no negative effect observed on this split' or the claim revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim ('we observed no negative effects on pruning') depends on two things that are not shown. In Section 4, the pruned vocabulary is formed by deleting every mDeBERTa token that never occurs in the ENC train split. First, it is never checked whether EstNER contains tokens or entity mentions that were removed: if a rare named entity was a single vocabulary item in the base tokenizer but absent from ENC, pruning forces it to be re-tokenized into subwords, changing the model input for exactly the spans that NER must label. Second, even for the reported numbers, Table 2 shows base mDeBERTa at 80.96±0.19 and pruned at 80.62±0.12 (n=3 each); no statistical test is reported, and the 0.34-point gap is not obviously noise—a pooled t-test would be borderline (p≈0.06). Without either an EstNER coverage analysis or a significance test, the abstract's unconditional 'no negative effects' is stronger than the evidence supports.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript compares two vocabulary-adaptation strategies for mDeBERTa v3 base on Estonian: retraining a 32K SentencePiece tokenizer on the Estonian National Corpus (ENC), and pruning every vocabulary token that never occurs in the ENC train split. It reports tokenizer efficiency (vocabulary size, parameter count, tokens per word), MLM accuracy, and EstNER F1 after embedding-only tuning and after LoRA-based continual training. The main findings are that retraining degrades NER performance (F1 76.40 ± 0.23 after embedding tuning; 77.58 ± 0.47 after LoRA) despite more compact tokenization, while pruning reduces parameters by about 23% and yields F1 scores close to the base model (80.62 ± 0.12 vs 80.96 ± 0.19).","tokens_in":6304,"tokens_out":3561,"duration_ms":36571,"significance":"If the pruning result holds, it is practically valuable: it would provide a cheap way to shrink a multilingual encoder for a target language without retraining. The study has concrete strengths: experiments are repeated three times with reported standard deviations, the comparisons use public datasets and publicly available models, and the authors explicitly acknowledge that MLM accuracy is not directly comparable across vocabularies. The retraining degradation is large and consistent, and the tokenization-efficiency measurements are useful. However, the paper's central 'no negative effects' claim currently rests on a small, untested F1 gap and on an implicit assumption that the ENC train split covers the tokens that matter in the EstNER evaluation text. Because these two points are load-bearing for the main conclusion, the current evidence is not yet sufficient for an unconditional claim.","major_comments":[{"comment":"The claim that pruning has 'no negative effects' is stronger than the reported statistics. The base mDeBERTa-base has F1 80.96 ± 0.19 and the pruned model has F1 80.62 ± 0.12 with three runs each; the 0.34-point difference is not subjected to any statistical test. A pooled two-sample t-test would be borderline (roughly p ≈ 0.06), so the observed gap cannot simply be dismissed as noise, and a non-significant result would not by itself establish equivalence. The authors should report an appropriate significance test or confidence interval for the pruning-vs-base comparison, or alternatively soften the wording to 'no observable negative effect in this experiment' and explicitly characterize the precision of the estimate.","section":"§5, Table 2"},{"comment":"The pruning criterion is based entirely on token occurrence in the ENC train split, and the paper never verifies that the EstNER evaluation text is covered by the retained vocabulary. If a named-entity token in EstNER was absent from ENC and was a single vocabulary item in the base tokenizer, pruning deletes it and forces retokenization into subwords on exactly the spans that the NER model must label. The observed parity between pruned and base models could then be specific to this particular train/evaluation split rather than a general property of pruning. To support the central claim, the authors should quantify coverage of the EstNER tokens (especially entity mentions) by the pruned vocabulary, or otherwise measure how often the pruned tokenizer changes the model input relative to the base tokenizer.","section":"§3 (Tokenizer Pruning) and §4 (Models Developed)"},{"comment":"The statement that 'a single epoch of embedding tuning was insufficient' to restore performance after tokenizer retraining is offered as a likely explanation, but no training curve or checkpoint evaluation is shown to support it. Since the paper already reports that only one epoch was used, a direct test would be to evaluate the retrained-tokenizer model at intermediate embedding-tuning checkpoints, or at least to report the trajectory of NER F1 as a function of tuning steps. Without such evidence, the 'longer tuning would help' claim remains speculative rather than an empirical finding.","section":"§6 (Discussion)"}],"minor_comments":[{"comment":"The text contains a typo: 'little bit worse' should be 'a little bit worse.'","section":"§5 (Results)"},{"comment":"The text says the model has to produce classification scores for individual 'tokes'; this should be 'tokens.'","section":"§4 (Fine-tuning on NER)"},{"comment":"Calling the LoRA phase 'full model training' is imprecise, since LoRA updates only low-rank adapters while most parameters remain frozen; consider phrasing such as 'continual training with LoRA adapters.'","section":"§3 (Continuous Training with LoRA)"},{"comment":"The paper does not specify whether the reported F1 is micro-averaged or macro-averaged, or whether it is computed at the token level or entity level; this should be stated for reproducibility.","section":"§4 (Fine-tuning on NER)"},{"comment":"The row label 'mDeBERTa base Tuned' is ambiguous because both the retrained-tokenizer model and embedding-tuned models are 'tuned'; a caption or footnote should clarify that this row denotes the model with the retrained 32K tokenizer.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an applied NLP or language-resource venue, and I see no concerns about citation practice or novelty disclosure. The revision should focus on the two load-bearing points: a proper statistical treatment of the pruned-vs-base F1 gap and a coverage analysis of EstNER under the pruned vocabulary. If those are addressed, the pruning result could become a solid practical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something useful and mostly does it carefully: it takes a strong multilingual encoder (mDeBERTa v3 base), applies two vocabulary adaptation routes for Estonian - retraining a 32K tokenizer and pruning tokens absent from a 3.4B-token corpus - and evaluates both on the same NER benchmark with three runs and standard deviations. The retrained-tokenizer result is a clean negative finding: even after one epoch of embedding tuning plus LoRA continual training, F1 drops from 80.96 to 76.40, then only to 77.58. That is consistent with prior work on tokenizer replacement requiring much more data, and it is a genuinely useful data point for low-resource language projects. The authors also deserve credit for openly stating that MLM accuracy is not comparable across vocabularies, and for not overplaying the efficiency gains.\n\nThe soft spot is the pruning claim. The abstract says 'we observed no negative effects on pruning,' but the evidence is a 0.34-point F1 gap (80.62 +/- 0.12 vs 80.96 +/- 0.19) with no statistical test. That gap is borderline: a pooled t-test would give p approximately 0.06, so it is not clearly noise. The stress-test's second worry also lands: the pruned vocabulary is built from the ENC train split, and the paper never checks whether EstNER contains entity tokens that were pruned. A rare named entity that was a single vocabulary item and never appeared in ENC would be re-tokenized into subwords, changing the input for exactly the spans NER must label. This may be harmless in practice - subword fallback usually works - but the paper does not show it. These are not fatal flaws, but they mean the central claim is somewhat overstated.\n\nTwo smaller issues: no code or checkpoints are released, which makes the numbers hard to verify independently, and the embedding-tuning duration (one epoch) is a free parameter that could interact with the retraining negative result. The authors acknowledge this, so it is minor.\n\nOverall, this is a well-scoped empirical paper with a clear question, honest caveats, and a useful negative result. It is not a breakthrough, but it is exactly the kind of evidence that helps people decide whether to prune or retrain a tokenizer on a low-resource language. I would send it to peer review, and in revision require either a significance test or a coverage analysis on EstNER - or a softening of the no-negative-effects wording to 'no significant negative effect observed.'","headline":"Solid empirical comparison of tokenizer retraining vs pruning for Estonian, but the 'no negative effects' claim for pruning outruns the statistics.","tokens_in":769,"tokens_out":852,"would_cite":true,"duration_ms":24249,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For Estonian, pruning mDeBERTa's unused vocabulary tokens cuts about 23% of model parameters with no NER performance loss, while a retrained 32K tokenizer drops accuracy.","keywords":["Estonian NLP","vocabulary pruning","tokenizer retraining","multilingual language models","named entity recognition","mDeBERTa","continual training","Low-Rank Adaptation"],"falsifier":"Count how many tokens in EstNER's test set, especially named-entity spans, are absent from the pruned 169K vocabulary. If a material fraction of test entities no longer appears as a single token and must be split into subwords, the pruned model's parity with the base model should degrade; alternatively, re-run the pruning experiment using only a non-Wikipedia subset of ENC and check whether EstNER F1 drops.","tokens_in":5894,"feed_emoji":"✂️","tokens_out":6472,"duration_ms":55362,"temperature":0.7,"pith_summary":"This paper asks whether a multilingual encoder can be made cheaper and better for Estonian by changing its vocabulary. It compares two routes: pruning every vocabulary token that never occurs in a 3.4B-token Estonian corpus, and replacing the vocabulary with a newly trained 32K-token Estonian tokenizer. The pruning route keeps Estonian named-entity recognition (EstNER) F1 essentially unchanged at 80.62 versus the base model's 80.96 while cutting parameters by about 23%. The retraining route makes tokenization more compact but drops EstNER F1 to 76.40 after embedding tuning and to 77.58 after additional LoRA training, leading the authors to conclude that pruning is the safe optimization and that retraining needs much longer adaptation than one epoch.","feed_headline":"Pruning mDeBERTa's vocabulary cuts model size 23% with no NER loss","feed_subtitle":"Retraining a 32K Estonian tokenizer instead drops NER F1, despite more compact tokenization.","key_machinery":"The load-bearing mechanism is vocabulary surgery followed by continual training. Pruning keeps the original tokenizer and embeddings for retained tokens, removing only tokens never seen in the ENC train split, so no new vectors enter the model. Retraining builds a new 32K SentencePiece vocabulary, copies embeddings for shared tokens, initializes unseen tokens with the mean of the original embedding matrix, then tunes embeddings for one epoch and continues with LoRA-based MLM training. The contrast shows that preserving the original embedding and tokenization structure is what protects downstream NER performance.","core_discovery":"The paper's central claim is that for mDeBERTa v3 base applied to Estonian, removing vocabulary tokens that never appear in the Estonian National Corpus is a free lunch: it shrinks the model from 279M to 215M parameters (about 23%) and leaves EstNER F1 at 80.62 ± 0.12, statistically indistinguishable in practice from the unmodified model's 80.96 ± 0.19. By contrast, training a new 32K-token SentencePiece tokenizer on the same corpus and initializing new embeddings as the mean of the original embedding matrix degrades NER performance to 76.40 ± 0.23 after one epoch of embedding-only tuning, and to 77.58 ± 0.47 after LoRA continual training; the authors attribute the gap to insufficient embedding tuning and note that longer training might close it. The paper also reports that MLM accuracy improves substantially with LoRA even when NER F1 does not, so the two metrics are not aligned across vocabularies.","pith_inferences":["If the pruned vocabulary is applied to Estonian text outside the ENC distribution, the assumption of full coverage weakens: rare named entities absent from ENC would be decomposed into subwords, and the measured parity might not hold on domain-specific NER test sets.","The 23% parameter reduction could translate into lower GPU memory and faster embedding lookups, but inference latency would barely change because token counts per word stay at 2.23; combining pruning with a distilled or quantized model would compound the savings.","The retrained 32K tokenizer's 1.75 tokens per word suggests that, with more adaptation data, it could eventually beat the pruned model on both efficiency and accuracy; a direct test would be to train embeddings for several epochs or increase LoRA training before evaluating.","The observed decoupling of MLM accuracy and NER F1 suggests that vocabulary-specific MLM scores are not comparable across tokenizers, and any future comparison should report tokenization-adjusted metrics or downstream task performance."],"forward_implications":["A practitioner can cut about 23% of mDeBERTa v3 base parameters for an Estonian use case by pruning never-seen vocabulary tokens, with NER F1 staying at the baseline level.","Embedding tuning after pruning is unnecessary: the pruned model with untouched embeddings already matches the base model, and tuning adds no gain.","A retrained 32K Estonian tokenizer needs substantially more than one epoch of embedding tuning and one epoch of LoRA training before it can be judged; the paper's protocol is insufficient to realize its potential.","MLM accuracy should not be used as a proxy for downstream NER quality when comparing models with different vocabularies, since LoRA raised MLM accuracy from 25.84 to 38.42 for the pruned model while NER F1 stayed flat.","The pruned model retains the base tokenizer, so its gains come from memory and parameter savings, not from shorter input sequences; sequence-length savings require the retrained tokenizer."],"supporting_citations":[{"why":"introduces the vocabulary pruning idea this paper adapts by dropping the frequency threshold and using a larger corpus","marker":"Abdaoui et al. (2020)"},{"why":"provides mDeBERTa v3, the base multilingual encoder whose vocabulary is pruned or replaced","marker":"He et al. (2023)"},{"why":"supplies EstNER, the named-entity dataset used to measure downstream F1","marker":"Sirts (2023)"},{"why":"supplies Low-Rank Adaptation, the continual-training method applied after vocabulary changes","marker":"Hu et al. (2021)"},{"why":"defines fast vocabulary transfer, the embedding-initialization approach the retraining route diverges from","marker":"Gee et al. (2022)"},{"why":"provides XLM-RoBERTa base as a multilingual baseline that the modified models are compared against","marker":"Conneau et al. (2020)"},{"why":"provides EstBERT, a language-specific baseline that the retrained-tokenizer model falls below","marker":"Tanvir et al. (2020)"}],"fun_headline_variants":["Prune mDeBERTa vocab for Estonian: 23% smaller, NER intact","Retraining Estonian tokenizer drops NER; pruning doesn't","For Estonian NER, prune the vocabulary, not retrain","Estonian vocab pruning cuts mDeBERTa 23%, keeps NER score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that pruning loses nothing assumes that the corpus used to decide which tokens to delete contains every word that matters in the evaluation data; if the test set has rare names or terms the corpus never saw, those words get broken into pieces and the pruned model could be worse.","fun_headline_variants_meta":{"raw":{"variants":["Prune mDeBERTa vocab for Estonian: 23% smaller, NER intact","Retraining Estonian tokenizer drops NER; pruning doesn't","For Estonian NER, prune the vocabulary, not retrain","Estonian vocab pruning cuts mDeBERTa 23%, keeps NER score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1682,"prompt_tokens":899,"completion_tokens":783,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":710}},"tokens_in":515,"tokens_out":783,"duration_ms":6317,"temperature":1.0,"reasoning_tokens":710,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:08:01.794642+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count how many tokens in EstNER's test set, especially named-entity spans, are absent from the pruned 169K vocabulary. If a material fraction of test entities no longer appears as a single token and must be split into subwords, the pruned model's parity with the base model should degrade; alternatively, re-run the pruning experiment using only a non-Wikipedia subset of ENC and check whether EstNER F1 drops.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies EstNER, the named-entity dataset used to measure downstream F1"},{"cited_title":"EstBERT: A Pretrained Language-Specific BERT for Estonian","cited_arxiv_id":"2011.04784","evidence_quote":"provides EstBERT, a language-specific baseline that the retrained-tokenizer model falls below"}],"review_version":1}