Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

When Every Token Counts: Optimal Segmentation for Low-Resource Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Optimal token splits cut token counts 3-5%, lift accuracy up to 10%.

desk verdict The intrinsic token-saving catalogue is solid and reusable, but the downstream accuracy claims rest on a confounded fine-tuning setup and a table inconsistency. read the letter →

arxiv 2412.06926 v5 pith:33JDBFOU submitted 2024-12-09 cs.CL cs.AIcs.LG

Bharath Raj , Garvit Suri , Vikrant Dewangan , Raghav Sonavane This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords optimalsegmentationBPEtokenizationtokensavingratiolow-resourcelanguagesGPT-2dynamicprogrammingmorphologicalcomplexityefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the standard greedy segmentation used by BPE tokenizers is suboptimal, and that replacing it with a minimum-token segmentation of the same vocabulary—computed by a Viterbi-style dynamic programming pass over a reversed-token trie—cuts token counts by 3-5% and improves downstream accuracy by up to 10% on low-resource and morphologically rich languages. The point is that a deployment-ready tokenizer can benefit from shorter sequences and better task performance without retraining a new vocabulary or pretraining from scratch. This matters because token bloat inflates compute costs quadratically with sequence length and disproportionately harms low-resource languages.

What carries the argument

The central object is the dynamic-programming recurrence $dp[i] = \min_{0 \le j \le i,\ d[j..i] \in V} (dp[j-1] + 1)$, which finds the minimum number of tokens to cover a document prefix. It is implemented with a trie built on reversed vocabulary tokens so membership tests run in $O(M)$ per position, where $M$ is the longest token length; the whole algorithm runs in $O(N \cdot M)$, the same worst-case bound as greedy segmentation. The paper also introduces the Token Saving Ratio (TSR), defined as $(|S_B(d)| - |S_A(d)|)/|S_B(d)|$, to quantify compression gains between two segmentations.

What would settle it

Train two identical GPT-2-size models from scratch on the same corpus and vocabulary—one with greedy BPE segmentation and one with optimal (minimum-token) segmentation—and compare their perplexity and task accuracy. If the optimally segmented model does not match or beat the greedy model after equivalent training, the paper's claim that optimal segmentation alone improves performance would be refuted.

Watch

Extended reading notes

Core claim

The core claim, stated on the paper's own terms, is that optimal segmentation $S^* = \arg\min_S |S(d)|$ over a fixed vocabulary $V$ is not only well-defined but practically achievable at the same worst-case time complexity as greedy BPE inference, and that the resulting sequences are both shorter and linguistically more coherent. The paper demonstrates this empirically across 116 languages using OpenAI tokenizers with 50K, 100K, and 200K vocabularies, reporting 3-5% average token savings and up to 20% savings on rare and long words, plus accuracy gains of up to 10% on classification and generation tasks in Finnish, Indonesian, and Turkish while keeping English perplexity essentially unchanged.

Load-bearing premise

Fine-tuning a model that was pretrained with greedy tokenization on optimally tokenized text is a valid substitute for pretraining with optimal tokenization from the start; if that substitution fails, the reported accuracy gains could be artifacts of distribution shift rather than benefits of optimal segmentation.

Editorial extensions

If this is right

  • Replacing greedy decoding with the minimum-token segmentation of the same vocabulary yields 3-5% fewer tokens on average across 116 languages, with up to 20% compression on rare and long words.
  • On downstream classification and generation tasks, models using optimal segmentation show accuracy gains of up to 10%, with larger gains on the subset of examples where the two segmentations actually differ.
  • English-language perplexity stays essentially flat, so the gains on non-English languages do not come at the cost of English modeling quality.
  • Because the optimal segmentation algorithm has the same worst-case time complexity as greedy, it can be dropped into existing BPE-based tokenizers without changing the vocabulary or the model architecture.
  • Longer words and agglutinative morphologies (Finnish, Turkish, Indonesian) show the strongest token savings, suggesting the method targets exactly the languages where token bloat is worst.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the fine-tuning proxy holds, then any subword tokenizer (WordPiece, Unigram, SentencePiece, or future ones) could adopt the same min-token decoding to gain compression and accuracy, since the segmentation stage is decoupled from vocabulary construction.
  • The TSR* subset analysis implies that gains concentrate on morphologically complex tokens; a testable extension is to predict per-sample benefit from word-length and morphological features before deciding whether to use optimal or greedy decoding.
  • A direct pretraining-from-scratch comparison—optimal vs greedy segmentation on identical corpora and vocabularies—would settle whether the reported accuracy gains reflect true representation quality or merely adaptation to a shifted token distribution; the paper did not run it.
  • Combining optimal segmentation with vocabulary construction that is aware of the min-token objective could push compression beyond the 3-5% reported here, since the vocabulary is held fixed in the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes replacing the greedy segmentation step of BPE tokenizers with an optimal segmentation that minimizes the number of tokens for a given document, using a dynamic programming algorithm over a reversed-trie of the fixed vocabulary. The authors evaluate the token-saving ratio (TSR) across 116 languages on CC-100 and three OpenAI tokenizer vocabularies, and then fine-tune GPT-2 (120M and 350M) checkpoints on OpenWebText with either greedy or optimal tokenization, evaluating downstream tasks in English, Finnish, Indonesian, and Turkish. They report 3–5% average token savings, larger savings for long words, and accuracy improvements up to about 10% on non-English tasks.

Significance. If the downstream claims held, the work would have practical value: it proposes a drop-in replacement for the segmentation stage that requires no vocabulary change or retraining. The algorithm is clearly described and its token-count minimization is mathematically sound and verifiable. The intrinsic measurement of token savings across languages is a useful empirical contribution, especially the word-length correlation. However, the paper's headline claim of accuracy improvements rests on a confounded experimental setup, and the reported results contain an inconsistency that undermines confidence in the extrinsic evaluation. The central contribution is therefore not yet established.

major comments (4)
  1. [Section 5.3 and Table 7] The extrinsic evaluation is confounded by distribution shift between pretraining and fine-tuning. The models are initialized from GPT-2 checkpoints pretrained on greedy tokenization and then fine-tuned on OpenWebText with optimal tokenization. The authors justify this by asserting that the greedy-pretrained model 'only has to learn the difference in the distribution of tokens with optimal segmentation' (Section 5.3), but no argument or supporting experiment is provided. Under optimal segmentation, the model sees different token IDs, different sequence lengths, and different co-occurrence statistics; the fine-tuned accuracy differences in Table 7 therefore conflate the effect of segmentation with the model's ability to adapt to a new token distribution. Without pretraining from scratch (or at least continued pretraining on optimal-tokenized data before task fine-tuning), the 'up to a 10% increase in accuracy' claim in the Abstract is not supported.
  2. [Table 5 vs Table 7] There is a direct inconsistency between Table 5 and Table 7 for Turkish XNLI. Table 5 reports that 100% of Turkish XNLI samples have non-zero TSR, which means the 'All' and 'TSR*' subsets are identical. However, Table 7 lists different accuracies for 'All' and 'TSR*' in the Turkish XNLI rows (e.g., 120M Greedy: 64.35 vs 63.83; 120M Optimal: 64.91 vs 64.59). Either the TSR* subset construction is incorrect, or Table 7 contains reporting errors. This discrepancy affects a reported result central to the paper's conclusion and must be resolved.
  3. [Section 3.2, Eq. (1) and Section 4.1, Eq. (2)] The headline 'optimal segmentation reduces token count' is true by construction: Eq. (2) defines S* as the minimizer of |S(d)| over a fixed vocabulary, so comparing S* to any other feasible segmentation (e.g., greedy) yields a non-negative TSR by definition. The paper's framing of this as an 'improvement' is therefore partially circular. The empirical content lies in the magnitude of the savings across languages and vocabularies, and the paper would be more accurately framed as measuring that magnitude rather than as demonstrating that optimal segmentation beats greedy in principle. The abstract and introduction should be revised to avoid presenting a definitional property as a discovered result.
  4. [Section 5.2 and Table 7] The TSR* subset analysis is post-hoc and lacks statistical controls. Restricting to examples where greedy and optimal differ (Table 5) can amplify small differences through selection effects, yet the paper reports no error bars, no number of random seeds, and no significance tests. For example, the English Story Cloze 350M result shows a 7.83% gain in TSR* versus a 0.43% gain on the full dataset, which is a large swing on a very small subset (6.15% of the data, hence about 13 examples if the dataset has ~200 examples). Without confidence intervals or multiple runs, these differences may reflect noise rather than a systematic effect. The authors should either provide repeated fine-tuning runs with variance estimates or temper their conclusions.
minor comments (5)
  1. [Table 2 caption] The caption of Table 2 defines 'TSR' as 'Token Stability Ratio', but Section 3.2 defines TSR as 'Token Saving Ratio'. This inconsistency should be corrected.
  2. [Algorithm 1] The pseudocode initializes dp[n] to 0 and uses dp[j-1] for j=0; the base case dp[-1]=0 is described in the text but not clearly represented in the code. Clarify the indexing (e.g., use 1-indexed or explicitly define dp[-1]).
  3. [Appendix B] The proof of optimality contains typos and confusing index notation, such as 'dp[i] = dp[km-2] + 1' where the recurrence should refer to dp[km-2] as a candidate minimum, not an equality, and the line 'dp[i] = dp[ki] + m - 1 - i' appears to have off-by-one errors. The proof should be rewritten for clarity.
  4. [Section 5.3] The description of the fine-tuning setup is insufficient for reproducibility: no learning rate, batch size, number of epochs, or random seed is reported. Appendix A lists only model architecture dimensions, not training hyperparameters.
  5. [Section 6.1.2] The claim that word length 'has a strong correlation' with TSR is supported only by a figure (Figure 1); no correlation coefficient or confidence interval is reported. A quantitative correlation statistic would strengthen the claim.

Circularity Check

1 steps flagged · score 6.0 of 10

Intrinsic 'optimal saves tokens' claim is true by definition of S*, so the framing is partly circular; magnitude and downstream results remain empirical.

  1. self definitional [Section 3.2, Eq. (1); Section 4.1, Eq. (2); Abstract]
    "The Token Saving Ratio when using tokenizer TA compared to tokenizer TB is defined as: T SR= |SB(d)| − |SA(d)| |SB(d)| (1) ... S∗ = minimize S |S(d)| (2) ... we demonstrate through extensive experiments that an optimal BPE configuration significantly reduces token count compared to greedy segmentation"

    Equation (2) defines S* as the segmentation minimizing |S(d)|, and greedy BPE is a feasible segmentation using the same vocabulary; so |S*(d)| ≤ |S_greedy(d)| for every document, making the TSR in Equation (1) non-negative whenever the two outputs differ. The abstract's claim that 'an optimal BPE configuration significantly reduces token count compared to greedy segmentation' is therefore entailed by the optimization objective, not established by the experiments. The experiments can only measure the magnitude and distribution of the savings (3-5%, word-length correlation), not the existence of a saving; presenting the sign as a demonstrated result is circular framing.

full rationale

The only genuine circular step is the intrinsic token-count claim: Eq. (2) defines S* as the minimizer of |S(d)|, and greedy BPE is a feasible segmentation, so TSR ≥ 0 by construction whenever the segmentations differ. The paper's abstract presents this sign as something 'demonstrate[d] through extensive experiments,' which is circular framing; the empirical content lies in the 3-5% magnitudes and their language- and word-length distribution, not in the existence of savings. The extrinsic accuracy comparisons are not circular but are weakened by an unsupported adaptation assumption in Section 5.3 ('We did not do a complete pretraining from scratch as the model pre-trained with greedy segmentation only has to learn the difference in the distribution of tokens with optimal segmentation'), so the 'up to 10%' accuracy gain may reflect token-distribution shift rather than the causal effect of optimal segmentation. That is a correctness/validity concern, not a circularity. No load-bearing self-citation or imported uniqueness theorem appears; the extrinsic evaluation uses external benchmarks and the paper's own DP recurrence is an internal optimization procedure. Score 6 reflects one central 'prediction' that reduces by construction while the quantitative and extrinsic content retains independent empirical substance.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its central token-saving claim rests on the definition of optimal segmentation, and its downstream claim rests on the fine-tuning adaptation assumption and the comparability of perplexity across tokenizations. The tie-breaking rule is an unexamined free choice.

free parameters (1)
  • Tie-breaking rule in dynamic programming = When multiple j give the minimal dp[i], choose the largest j (smallest suffix)
    Algorithm 1 states that only the largest such j is considered. This choice shapes the actual token sequence but is not derived from the token-count objective, and it may influence linguistic quality and downstream results.
assumptions (4)
  • domain assumption Any sequence of strings from the fixed vocabulary V is a valid tokenizer output for the byte sequence
    Section 4.2 builds the DP over arbitrary concatenations of vocabulary items, ignoring GPT-2's merge-order constraints. This is likely true for decoding but is not argued.
  • domain assumption Continued fine-tuning on OpenWebText from a greedy-pretrained checkpoint approximates training from scratch with optimal segmentation
    Section 5.3 states that the model 'only has to learn the difference in the distribution of tokens'; this is load-bearing for all extrinsic results.
  • domain assumption Per-token perplexity is comparable across different segmentations of the same text
    Table 8 compares perplexity numbers for Greedy and Optimal tokenizations without discussing that the token-level unit changes; bits-per-character or sentence-level likelihood would be safer.
  • standard math The DP recurrence dp[i] = min(dp[j-1]+1) computes the minimum token count
    Appendix B provides a proof sketch; the result is a standard shortest-path argument.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Every Token Counts: Optimal Segmentation for Low-Resource Language Models." pith.science (2026). https://pith.science/paper/33JDBFOU

@misc{pith2026241206926,
  author       = {Pith},
  title        = {Pith review of: When Every Token Counts: Optimal Segmentation for Low-Resource Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/33JDBFOU}},
  note         = {Machine review of arXiv:2412.06926}
}
read the original abstract

Traditional greedy tokenization methods have been a critical step in Natural Language Processing (NLP), influencing how text is converted into tokens and directly impacting model performance. While subword tokenizers like Byte-Pair Encoding (BPE) are widely used, questions remain about their optimality across model scales and languages. In this work, we demonstrate through extensive experiments that an optimal BPE configuration significantly reduces token count compared to greedy segmentation, yielding improvements in token-saving percentages and performance benefits, particularly for smaller models. We evaluate tokenization performance across various intrinsic and extrinsic tasks, including generation and classification. Our findings suggest that compression-optimized tokenization strategies could provide substantial advantages for multilingual and low-resource language applications, highlighting a promising direction for further research and inclusive NLP.

Figures

Figures reproduced from arXiv: 2412.06926 by the authors.

Figure 1
Figure 1. TSR and Word length correlation across seven [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Frequency vs Word Length: Comparison across seven languages with a vocab size of m = 100K In-context evaluation [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. In-Context Comparison: Percentage of ex [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Tokenization Matters: Improving Zero-Shot NER for Indic Languages

    cs.CL 2025-04 reject novelty 4.0 of 10

    Indic NER with SentencePiece tokenization beats BPE in zero-shot cross-lingual transfer, but the paper's evidence is limited by internal contradictions and undisclosed implementation details.

Reference graph

Works this paper leans on

36 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Mortensen, Noah A

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. 2023. https://arxiv.org/abs/2305.13707 Do all languages cost the same? tokenization in the era of commercial language models . Preprint, arXiv:2305.13707

    arXiv 2023

  2. [2]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  3. [3]

    Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki

    Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics

    work page 2020

  4. [4]

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. https://doi.org/10.18653/v1/D18-1269 XNLI : Evaluating cross-lingual sentence representations . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475--2485, Brussels, Belgium. Association...

    doi:10.18653/v1/d18-1269 2018

  5. [5]

    Gautier Dagan, Gabriele Synnaeve, and Baptiste Rozi \`e re. 2024. https://api.semanticscholar.org/CorpusID:267406279 Getting the most out of your tokenizer for pre-training and domain adaptation . ArXiv, abs/2402.01035

    arXiv 2024

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

    arXiv 2019

  7. [7]

    Pratibha Dongare. 2024. https://aclanthology.org/2024.wildre-1.8 Creating corpus of low resource I ndian languages for natural language processing: Challenges and opportunities . In Proceedings of the 7th Workshop on Indian Language Data: Resources and Evaluation, pages 54--58, Torino, Italia. ELRA and ICCL

    work page 2024

  8. [8]

    G.D. Forney. 1973. https://doi.org/10.1109/PROC.1973.9030 The viterbi algorithm . Proceedings of the IEEE, 61(3):268--278

    arXiv 1973

Show all 36 references
  1. [9]

    Matthias Gall \'e . 2019. https://doi.org/10.18653/v1/D19-1141 Investigating the effectiveness of BPE : The power of shorter sequences . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu...

    2019 doi

  2. [10]

    Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. https://doi.org/10.18653/v1/2024.findings-acl.134 Unpacking tokenization: Evaluating text compression and its correlation with model performance . In Findings of the Association for Comp...

    2024 doi

  3. [11]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  4. [12]

    Taku Kudo. 2018. https://api.semanticscholar.org/CorpusID:13753208 Subword regularization: Improving neural network translation models with multiple subword candidates . ArXiv, abs/1804.10959

    2018 arXiv

  5. [13]

    Taku Kudo and John Richardson. 2018. https://api.semanticscholar.org/CorpusID:52051958 Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing . In Conference on Empirical Methods in Natural Language Processing

    2018

  6. [14]

    Alexandre Magueresse, Vincent Carles, and Evan Heetderks. 2020. https://arxiv.org/abs/2006.07264 Low-resource languages: A review of past work and future challenges . Preprint, arXiv:2006.07264

    2020 arXiv

  7. [15]

    Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini

    Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: the penn treebank. Comput. Linguist., 19(2):313–330

    1993

  8. [16]

    Nikita Moghe, Evgeniia Razumovskaia, Liane Guillou, Ivan Vuli \'c , Anna Korhonen, and Alexandra Birch. 2023. https://doi.org/10.18653/v1/2023.findings-acl.230 M ulti3 NLU ++: A multilingual, multi-intent, multi-domain dataset for natural language understanding in task-oriente...

    2023 doi

  9. [17]

    Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. https://arxiv.org/abs/1604.01696 A corpus and evaluation framework for deeper understanding of commonsense stories . Preprint, arXiv:1604.01696

    2016 arXiv

  10. [18]

    Rozina Myoya, Fiskani Banda, Vukosi Marivate, and Abiodun Modupe. 2023. https://api.semanticscholar.org/CorpusID:259311336 Fine-tuning multilingual pretrained african language models . In AfricaNLP

    2023

  11. [19]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  12. [20]

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. https://arxiv.org/abs/1606.06031 The lambada dataset: Word prediction requiring a broad discourse context . Prepri...

    2016 arXiv

  13. [21]

    Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. https://arxiv.org/abs/2305.15425 Language model tokenizers introduce unfairness between languages . Preprint, arXiv:2305.15425

    2023 arXiv

  14. [22]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Comput. Linguist

    2019

  15. [23]

    Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.802 XTREME - R : Towards more challenging and nuanced multilingual eva...

    2021 doi

  16. [24]

    Phillip Rust, Jonas Pfeiffer, Ivan Vuli \'c , Sebastian Ruder, and Iryna Gurevych. 2021. https://doi.org/10.18653/v1/2021.acl-long.243 How good is your tokenizer? on the monolingual performance of multilingual language models . In Proceedings of the 59th Annual Meeting of the ...

    2021 doi

  17. [25]

    Jonne Saleva and Constantine Lignos. 2023. https://doi.org/10.18653/v1/2023.insights-1.7 What changes when you randomly choose BPE merge operations? not much. In Proceedings of the Fourth Workshop on Insights from Negative Results in NLP, pages 59--66, Dubrovnik, Croatia. Asso...

    2023 doi

  18. [26]

    Mei Silviana Saputri, Rahmad Mahendra, and Mirna Adriani. 2018. Emotion classification on indonesian twitter dataset. In Proceedings of the 2018 International Conference on Asian Language Processing(IALP), pages 90--95. IEEE

    2018

  19. [27]

    Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner

    Craig W. Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. https://api.semanticscholar.org/CorpusID:268041516 Tokenization is more than compression . ArXiv, abs/2402.18376

    2024 arXiv

  20. [28]

    Mike Schuster and Kaisuke Nakajima. 2012. https://api.semanticscholar.org/CorpusID:22320655 Japanese and korean voice search . 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149--5152

    2012

  21. [29]

    Ken Nabila Setya and Rahmad Mahendra. 2018. Semi-supervised textual entailment on indonesian wikipedia data. In Proceedings of the 2018 International Conference on Computational Linguistics and Intelligent Text Processing (CICLing)

    2018

  22. [30]

    Schmidt, Chris Tanner, and Yuval Pinter

    Omri Uzan, Craig W. Schmidt, Chris Tanner, and Yuval Pinter. 2024. https://api.semanticscholar.org/CorpusID:268230397 Greed is all you need: An evaluation of tokenizer inference methods . ArXiv, abs/2403.01289

    2024 arXiv

  23. [31]

    Menan Velayuthan and Kengatharaiyer Sarveswaran. 2024. https://arxiv.org/abs/2409.11501 Egalitarian language representation in language models: It all begins with tokenizers . Preprint, arXiv:2409.11501

    2024 arXiv

  24. [32]

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzm \'a n, Armand Joulin, and Edouard Grave. 2020. https://aclanthology.org/2020.lrec-1.494 CCN et: Extracting high quality monolingual datasets from web crawl data . In Proceedings of the Twel...

    2020

  25. [33]

    Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. https://aclanthology.org/2020.aacl-main.85 I ndo NLU : Benchmark and resources for evaluati...

    2020

  26. [34]

    Shiyue Zhang, Vishrav Chaudhary, Naman Goyal, James Cross, Guillaume Wenzek, Mohit Bansal, and Francisco Guzman. 2022. https://aclanthology.org/2022.amta-research.8 How robust is neural machine translation to language imbalance in multilingual tokenizer training? In Proceeding...

    2022

  27. [35]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.