Pith. sign in

REVIEW 2 major objections 5 minor 44 references

BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis

T0 review · 2 major / 5 minor · reviewed 2026-07-31 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A purpose-built tokenizer for Indian languages yields ~90% shorter sequences than GPT-2 on classical text.

desk verdict Useful tokenizer resource, but the headline generalisation claim is untested because the 'held-out' set is sampled from training corpora without a stated exclusion. read the letter →

arxiv 2607.23319 v1 pith:BZORYYHY submitted 2026-07-25 cs.CL cs.CYcs.ETcs.LG

Poornima Kumaresan , Pavithra Muruganantham , Lakshmi Rajendran , Santhosh Sivasubramani This is my paper · ORCID
classification cs.CLcs.CYcs.ETcs.LG
keywords subwordtokenizationSentencePiecebyte-pairencodingIndianlanguagesSanskritTamilmorphologyfertilityanalysisKnowledgeSystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard subword tokenizers, trained mostly on modern English, shred Brahmic-script text into byte fragments, wasting the context windows of small language models. This paper builds BHARATI v3, a SentencePiece BPE tokenizer with a 32,000-token vocabulary trained on a deliberately balanced 781 MB corpus across English, Hindi, Sanskrit, Tamil, Telugu, Kannada, and Malayalam, adding sandhi resolution for Sanskrit and soft suffix-boundary marking for Tamil. On a 490-sentence test set of Indian Knowledge System text, v3 produces sequences roughly 25% shorter than the mBART-50 multilingual baseline and about 90% shorter than GPT-2/byte-level encoding, and it reserves four high-frequency IKS terms as single tokens. The paper argues this directly translates into longer effective context and faster inference for downstream models, citing a 4.3x reduction in Tamil generation steps and a drop in training loss from 4.21 to 3.12 for a 15M-parameter model. The main load-bearing caveat: the test set was sampled from the same released corpora used to train the tokenizer, and the paper does not state that those sentences were excluded from that training.

What carries the argument

The central object is a SentencePiece byte-pair-encoding tokenizer with a 32,000-token vocabulary, trained on a balanced 781 MB corpus across seven languages with three preprocessing steps: Unicode normalization, rule-based sandhi resolution for Sanskrit (correct on about 87% of junctions), and a soft suffix-boundary-marking step for Tamil that doubles the frequency of productive suffixes without forcing hard splits. The fertility-analysis framework — tokens per word, tokens per domain term, and tokens per sentence, plus distributional spread and vocabulary utilization — is what turns the tokenizer into a measurable claim. Reserved IKS terms placed in the vocabulary (the four Patha variants)

What would settle it

Run v3 and the mBART-50 tokenizer on a fresh set of IKS sentences drawn from sources outside the Sangraha, Gutenberg, and classical-text corpora (for example, newly published critical editions or oral commentaries), and compare mean tokens per sentence. If the v3 advantage over mBART-50 shrinks from ~25% toward zero or reverses, the central claim fails. Alternatively, a reader can scan the released training corpus for the 490 benchmark sentences: if they appear, the held-out claim is disproven directly.

Watch

Extended reading notes

Core claim

The central claim is that a tokenizer trained on balanced, script-native, domain-supplemented data can close most of the tokenization gap for classical Indian languages without specialized morphological analyzers at inference time. BHARATI v3 averages 2.6 tokens per IKS term versus 5.25 for GPT-2 and 3.75 for the multilingual SentencePiece baseline; for the six Indic languages it lowers mean tokens per word to between 1.04 (Hindi) and 2.62 (Tamil), versus 7.6–23.9 under GPT-2. On the 490-sentence benchmark, average sequence length falls by approximately 90% relative to GPT-2 and byte-level encoding and by roughly 25% relative to mBART-50, with the largest per-language reductions over mBART f

Load-bearing premise

The load-bearing premise is that the 490-sentence test set is genuinely held out from tokenizer training; the paper says the set was sampled from the same released corpora used to build v3 but never states that those sentences were excluded from training, so the reported ~25% reduction over mBART-50 could be an in-sample estimate.

Editorial extensions

If this is right

  • For the six Indic languages, v3 lowers mean tokens per word to 1.04–2.62, versus 7.59–23.88 under GPT-2, so a fixed context window holds several times more Indic text.
  • A 15M-parameter transformer trained with v3 reaches training loss 3.12 at 50K steps versus 4.21 for v1, and uses 81.4% of its vocabulary versus 34.2% for GPT-2 on the same test corpus.
  • A 100-word Tamil response needs about 205 generation steps with v3 versus roughly 890 with GPT-2, a 4.3x reduction in inference latency.
  • The four reserved Patha terms tokenize to single units only in their exact space-separated Latin spelling; lowercase, hyphenated, or Devanagari renderings are multi-token.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the four Patha terms are single-token by construction, the 2.6 average IKS fertility is partly a design artifact; excluding the reserved terms would bring BHARATI close to the mBART baseline, so the paper's headline term fertility should be read as a ceiling for its reserved vocabulary, not a general linguistic gain.
  • The reported ~25% edge over mBART-50 is measured on text drawn from the same corpora used for training; a fair out-of-domain test on unseen IKS texts (new editions, commentaries, oral-transcript data) would likely show a smaller margin, and if those 490 sentences were not excluded from v3 training, the margin would be an in-sample estimate.
  • A direct way to test the morphological-awareness claim is to check whether v3's BPE merges align with gold Paninian sandhi splits and Tamil suffix boundaries on a separately annotated corpus; the paper only reports that the preprocessing splitter is 87% accurate, not that the tokenizer itself segments unsplit forms morphologically.
  • The paper's own 48K-vocabulary pilot (8% fertility reduction for Tamil, 6% for Telugu at 50% larger embedding) suggests an easy extension: measure fertility versus vocabulary size across 32K, 48K, 64K, and 96K to find the Pareto frontier for Indo-Aryan versus Dravidian languages.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents BHARATI, a family of SentencePiece BPE tokenizers for seven languages (English, Hindi, Sanskrit, Tamil, Telugu, Kannada, Malayalam), culminating in v3 with a 32K vocabulary and native subword coverage. The authors describe corpus construction, Sanskrit sandhi resolution, Tamil suffix-boundary marking, and the evolution from v1 to v3. They report three levels of fertility evaluation: token-level (tokens per word), term-level (tokens per IKS technical term), and sentence-level (tokens per sentence on a 490-sentence set). The headline results are a roughly 90% sequence-length reduction over GPT-2 and byte-level encoding, and approximately 25% over mBART-50 on the six Indic languages, plus downstream NanoGPT training-loss improvements. Models, scripts, and benchmarks are released.

Significance. If the sentence-level result is robust, BHARATI v3 would be a useful resource for context-limited multilingual and classical-language NLP, and the released models, training scripts, and benchmark would allow independent verification. The paper also gives a clear, honest account of the version history and of which IKS terms are single-token by construction. The main quantitative claims are plausible in direction, but the evaluation protocol currently leaves open a key question about whether the reported ~25% advantage over mBART-50 reflects genuine generalization or training-set overlap. The released artifacts are a real strength, and the authors' practice of disclosing caveats in footnotes and section text is commendable.

major comments (2)
  1. [§5.3, Fig. 4, Table 1] The central sentence-level claim (≈90% vs GPT-2/byte-level, ≈25% vs mBART-50) is measured on a test set described as 'held-out' but 'sampled from released corpora.' Table 1 shows the v3 tokenizer was trained on the same Sangraha/Sangam/classical corpora. The paper never states that the 490 sentences, or their component n-grams, were excluded from SentencePiece BPE training. Because BPE merges are frequency-derived, test text present in the training stream can receive inflated merge counts, artificially lowering measured fertility. This is not an accusation of misconduct; it is an unverified assumption in the evaluation protocol. Please either confirm the exclusion and include it in the released measurement script, or re-run the sentence-level comparison on a truly external IKS test set (e.g., newly digitized texts not in the training corpora) and report both within-corpus and out-of-doma
  2. [§5.2, Table 4, Abstract] The headline term-level fertility of 2.6 tokens per IKS term is achieved only because the four Patha terms are reserved in the vocabulary and tokenize to a single unit by construction. For the four non-reserved terms, BHARATI v3 averages 4.25 tokens, slightly worse than mBART's 4.0. The text is transparent about the by-construction status, but the Abstract and Conclusion repeat the 2.6 figure without the reserved/non-reserved breakdown. Please report the two averages separately whenever the 2.6 claim is made, and frame the term-level contribution as applying specifically to the reserved subset rather than to IKS terms generally.
minor comments (5)
  1. [Table 3] The baselines are not described with their vocabulary sizes. mBART-50 uses a much larger SentencePiece vocabulary than BHARATI's 32K; stating this in the caption or text would put the comparison in proper context.
  2. [§4.2, Table 2] The phrase 'approximate 10.6 tokens per word' for v2 collapses across languages with different fallback status. Clarify whether this is an unweighted mean of the seven per-language fertilities.
  3. [§5.2, Table 4 note] The note states that the reserved terms are the four Patha variants, but the total number and selection criteria for reserved terms are not given. A brief sentence on how many terms were reserved and why would help readers interpret the term-level results.
  4. [§1, Fig. 1] Figure 1 labels the corpus as '850 MB balanced corpus' while Table 1 and §1 give 781 MB after deduplication. The discrepancy is explained in a footnote, but the figure label could cause confusion for readers who skip the footnote.
  5. [§7.1, Table 5] The 'identical data' comparison across v1/v2/v3 is helpful, but the training loss at 50K steps depends on effective tokens per example; state whether the number of steps is kept constant while the amount of text per step varies, so readers can interpret the loss difference correctly.

Circularity Check

2 steps flagged · score 4.0 of 10

Term-level fertility headline is partly by-construction (four reserved Patha terms are single tokens because they were placed in the vocabulary), and the sentence-level 'held-out' 490-sentence set is sampled from the same released corpora used to train v3 without any stated exclusion; token-level and NanoGPT comparisons remain independent measurements.

  1. self definitional [Section 5.2 / Table 4 (IKS Term-Level Fertility); also abstract and Section 9]
    "The four Patha terms (†) are reserved in the vocabulary and tokenize to a single unit by construction; ... For non-reserved technical terms, such as Vishishtadvaita and Shankaracharya, BHARATI v3 is on par with the multilingual baseline (marginally behind) and ahead of GPT-2, indicating that its advantage is specific to the reserved IKS vocabulary rather than a general property of the training corpus."

    The headline 2.6-token average is computed over Table 4, and four of the eight terms score 1 only because they were inserted into the vocabulary during construction. Excluding those four by-construction terms, v3's average on the remaining terms is (2+7+2+6)/4 = 4.25, versus 4.0 for mBART, so the reported 1.4x reduction over the multilingual baseline is not an emergent property of the trained BPE; it is largely the output of the vocabulary-construction step. The paper explicitly discloses this, but the abstract and conclusion still present the aggregate 2.6 figure as a demonstrated fertility gain.

  2. other [Section 5.3 / Figure 4 and footnote 2; cf. Table 1]
    "Figure 4 presents sentence-level fertility on a held-out test set of 490 IKS-domain sentences (70 per language across seven languages, sampled from released corpora with reserved-term sentences excluded; the set and measurement script are released for exact reproduction). 2 A single post-registration amendment was made before any Tamil results were used: Thirukkural couplets, which the initial sentence splitter had separated into sub-five-word half-lines, are joined into whole couplets; this is logged in the released measurement-script header."

    The sentence-level 'held-out' claim is unsupported as an independence claim: the 490 sentences are 'sampled from released corpora,' while Table 1 shows v3 was trained on those same Sangraha/classical corpora (e.g., Tamil: Sangraha + Sangam + Thirukkural). The paper never states that the 490 sentences, or their component n-grams, were excluded from SentencePiece BPE training. If they were not excluded, the reported ~25% reduction over mBART and ~90% over GPT-2/byte-level are measured on the tokenizer's own training distribution, so the 'held-out' framing and implied generalization are not established. The disclosed post-registration amendment also shows the exact test-set definition changed during measurement.

full rationale

No load-bearing self-citation chain was found; the cited prior work appears to be external, and the tokenizer training pipeline is described with reproducible configuration. The token-level results and NanoGPT training experiments are independent comparisons against external baselines and retain substantial evidentiary value. The main construction-level circularity is the term-level fertility aggregate, where four reserved Patha terms are single-token by vocabulary insertion rather than by learned segmentation; this is disclosed in the paper, so it is a partial rather than hidden circularity. The sentence-level benchmark also carries a material validity risk because the 'held-out' set is sampled from the same released corpora used for training, with no stated exclusion from the training stream. Weighing these together, the central sentence-level and downstream claims still have independent content, so a score of 4 is appropriate rather than a higher score reserved for cases where the central result reduces entirely to a fit or self-citation chain.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper's central measures rest on standard BPE machinery plus several domain choices: corpus balancing, manual reservation of IKS terms, and the assumption that the released test sentences are genuinely held out. No new scientific entities are posited; the design choices (vocab size, 2x suffix boosting, reserved terms) are the effective free parameters.

free parameters (5)
  • vocabulary_size = 32,000
    SentencePiece vocabulary size is a design choice; the paper notes 48K would give ~8% lower Tamil fertility at 50% larger embeddings, but 32K is the shipped choice affecting all reported fertilities.
  • character_coverage = 0.9995
    Standard SentencePiece coverage threshold; determines byte fallback rate and affects the vocabulary composition.
  • corpus_balance = 50–140 MB per language (781 MB total)
    The balancing targets 'roughly equal' sizes per language, a hand-chosen allocation that determines vocabulary share per script.
  • reserved_iks_terms = 4 terms (Samhita Patha, Pada Patha, Krama Patha, Ghana Patha)
    These four terms are manually inserted into the vocabulary as single tokens; the Table 4 term-fertility average (2.6) depends on this construction.
  • tamil_soft_boundary_ratio = doubles suffix subword frequency
    The soft-boundary heuristic duplicates suffix tokens to bias BPE; the exact ratio (2x) is chosen by hand in §3.4.
assumptions (5)
  • standard math BPE merge statistics over the training corpus determine the vocabulary; the algorithm itself is taken as given.
    SentencePiece BPE is used as a black-box standard method (§4.1).
  • domain assumption The Sangraha corpora, Gutenberg, and classical digital texts are representative of the IKS domain in which BHARATI will be used.
    Corpus composition in Table 1 is the foundation for all fertility results; no external validation of domain representativeness is given.
  • domain assumption The rule-based sandhi splitter (Heritage Sanskrit Platform) provides correct boundaries for ~87% of junctions; the remaining 13% of unsplit forms do not materially degrade the tokenizer.
    §3.3 states this accuracy from manual evaluation of 500 junctions; the tokenizer relies on pre-split input.
  • domain assumption The 490-sentence test set is a fair held-out sample of IKS text.
    §5.3 says the set is sampled from released corpora and 'held-out', but the paper does not demonstrate exclusion from training; the weakest premise for the sentence-level claims.
  • domain assumption Fertility is an adequate proxy for downstream model quality.
    Acknowledged in §8.4 as 'necessary but not sufficient'; all headline conclusions are fertility-based.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis." pith.science (2026). https://pith.science/paper/BZORYYHY

@misc{pith2026260723319,
  author       = {Pith},
  title        = {Pith review of: BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BZORYYHY}},
  note         = {Machine review of arXiv:2607.23319}
}
read the original abstract

Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages. Sanskrit, Tamil, and other classical Indic languages exhibit agglutinative morphology, productive sandhi (phonological fusion at word boundaries), and domain-specific vocabularies absent from general-purpose training data. This paper presents BHARATI, a set of SentencePiece BPE tokenizers trained on a balanced 781 MB corpus spanning seven languages (English, Hindi, Sanskrit, Tamil, Telugu, Kannada, and Malayalam) with native script support for all languages. We describe three successive tokenizer versions: v1 (English and Sanskrit only, with broken byte-fallback for Tamil), v2 (four-language support with byte-level fallback for southern languages), and v3 (full seven-language native subword coverage). Subword fertility analysis demonstrates that v3 averages 2.6 tokens per Indian Knowledge System (IKS) technical term, compared to 5.25 tokens per term with GPT-2's tokenizer and 3.75 tokens with the multilingual SentencePiece baseline, with the largest gains on a set of reserved IKS terms that are represented as single tokens by construction. On a held-out test set of 490 IKS-domain sentences (70 per language across seven languages, released with the measurement script), v3 reduces sequence length by roughly 90% relative to GPT-2 and byte-level encoding (which lack native Indic subwords) and by approximately 25% relative to the mBART-50 multilingual baseline, averaged across the six Indic languages, directly translating to increased effective context length for downstream language models. The tokenizer models (32,000 vocabulary), training scripts, and evaluation benchmarks are released under open licenses.

Figures

Figures reproduced from arXiv: 2607.23319 by the authors.

Figure 1
Figure 1. Top-level overview of the BHARATI tokenizer pipeline. Seven languages are drawn from balanced [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Tokenizer version evolution: mean tokens per word (log scale) for each language across BHARATI [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Heatmap of tokens per IKS technical term across four tokenizers. Darker cells indicate higher [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Average tokens per sentence (log scale) for 490 IKS-domain test sentences across seven languages [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Distribution of 32,000 vocabulary entries across scripts in BHARATI v3. English receives the [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Left: NanoGPT training loss curves for 15M-parameter models using BHARATI v1, v2, and [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 13 canonical work pages

  1. [17]

    Pre- 30 training via paraphrasing

    Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Asli Celikyilmaz, and Luke Zettlemoyer. Pre- 30 training via paraphrasing. InAdvances in Neural Information Processing Systems, volume 33, pages 18470–18481, 2020

    2020

  2. [1]

    Neural machine translation of rare words with subword units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725. Association for Computational Linguistics, 2016. doi: 10.18653/v1/ P16-1162

    doi:10.18653/v1/ 2016

  3. [2]

    BERT: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. InProceedings of NAACL-HLT 2019, pages 4171–4186. Association for Computational Linguistics, 2019

    2019

  4. [3]

    SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

    Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71. Association for Computational Linguistics, 2018. doi: 10.18653/v1/D18-2012

    doi:10.18653/v1/d18-2012 2018

  5. [4]

    Torr, and Adel Bibi

    Aleksandar Petrov, Emanuele La Malfa, Philip H.S. Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages.arXiv preprint arXiv:2305.15425, 2024

    arXiv 2024

  6. [5]

    A distributed platform for Sanskrit processing

    Pawan Goyal and Gérard Huet. A distributed platform for Sanskrit processing. InProceedings of COLING 2012, pages 1011–1028. Association for Computational Linguistics, 2012

    2012

  7. [6]

    Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V . Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation.arXiv preprint arXiv:1609.08144, 2016

    arXiv 2016

  8. [7]

    Language models are unsupervised multitask learners.OpenAI blog, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.OpenAI blog, 2019

    2019

Show all 44 references
  1. [8]

    Research on morphological knowledge-guided low-resource agglutinative languages- chinese translation.Complex and Intelligent Systems, 11, 2025

    Abudouwaili G. Research on morphological knowledge-guided low-resource agglutinative languages- chinese translation.Complex and Intelligent Systems, 11, 2025. doi: 10.1007/s40747-025-01780-5

    2025 doi

  2. [9]

    Analysing unified embedding with morphological insight for multilingual text representation

    Phadte A. Analysing unified embedding with morphological insight for multilingual text representation. Engineered Science, 40, 2026. doi: 10.30919/es2102. 29

    2026 doi

  3. [10]

    Unsupervised cross-lingual representation learning at scale

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. InProceedings of the 58th Annual Meeting of ...

    2020

  4. [11]

    Khapra, and Pratyush Kumar

    Anoop Kunchukuttan, Divyanshu Kakwani, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. Ai4bharat-indicnlp corpus: Monolingual corpora and word embeddings for indic languages. InProceedings of the Twelfth Language Resources and Evaluation Con...

    2020

  5. [12]

    Khapra, and Pratyush Kumar

    Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. Indicnlpsuite: Monolingual corpora, evaluation benchmarks and pre- trained multilingual language models for indian languages. InFindings of EMNLP 2020, pa...

    2020

  6. [13]

    Tokenization and representation biases in multilingual models on dialectal nlp tasks

    Vani Kanjirangat. Tokenization and representation biases in multilingual models on dialectal nlp tasks. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 24003–24021, 2025. doi: 10.18653/v1/2025.emnlp-main.1224

    2025 doi

  7. [14]

    Kren-ne: A multilingual tokenization framework for northeast indian languages

    Badal Nyalang. Kren-ne: A multilingual tokenization framework for northeast indian languages. TechRxiv Preprint, 2025. doi: 10.36227/techrxiv.176184636.69664258/v1

    2025

  8. [15]

    Domain-specific language model pretraining for biomedical natural language processing.ACM Transactions on Computing for Healthcare, 3(1):1–23, 2021

    Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing.ACM Transactions on Computing for Healthcare, 3(1):1–23, 2021. doi:...

    2021

  9. [16]

    LEGAL-BERT: The muppets straight out of law school

    Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. LEGAL-BERT: The muppets straight out of law school. InFindings of EMNLP 2020, pages 2898–2904. Association for Computational Linguistics, 2020

    2020

  10. [18]

    Word segmentation for classical Chinese: New standards and a study on using pre-training

    Chao Li and Ji Xing. Word segmentation for classical Chinese: New standards and a study on using pre-training. InProceedings of the Sixth Workshop on NLP for Similar Languages, Varieties, and Dialects, pages 174–182. Association for Computational Linguistics, 2019

    2019

  11. [19]

    Tomasz Limisiewicz. Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages.Findings of the Association for Computational Linguistics: ACL 2023, pages 5661–5681, 2023. doi: 10.18653/v1/2023.findings-acl.350

    2023 doi

  12. [20]

    Tokenization with factorized subword encoding.Findings of the Association for Computational Linguistics: ACL 2023, pages 14143–14161, 2023

    David Samuel. Tokenization with factorized subword encoding.Findings of the Association for Computational Linguistics: ACL 2023, pages 14143–14161, 2023. doi: 10.18653/v1/2023.findings-acl. 890

    2023 doi

  13. [21]

    Bpe gets picky: Efficient vocabulary refinement during tokenizer training.Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16587–16604,

    Pavel Chizhov. Bpe gets picky: Efficient vocabulary refinement during tokenizer training.Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16587–16604,

    2024

  14. [22]

    Tokenization efficiency in code-switched text: Comparing sentencepiece and byte-pair encoding on taglish.TechRxiv Preprint, 2025

    Hyacinth Nathalie Salve. Tokenization efficiency in code-switched text: Comparing sentencepiece and byte-pair encoding on taglish.TechRxiv Preprint, 2025. doi: 10.36227/techrxiv.175756338.86755792/ v1

    2025

  15. [23]

    Tokenization matters: Improving zero-shot ner for indic languages.IEEE International Conference on Electro Information Technology, pages 456–462, 2025

    Pattnayak P. Tokenization matters: Improving zero-shot ner for indic languages.IEEE International Conference on Electro Information Technology, pages 456–462, 2025. doi: 10.1109/eIT64391.2025. 11103625

    2025

  16. [24]

    Haoran Lian. Scaffold-bpe: Enhancing byte pair encoding for large language models with simple and effective scaffold token removal.Proceedings of the AAAI Conference on Artificial Intelligence, 39: 24539–24548, 2025. doi: 10.1609/aaai.v39i23.34633

    2025 doi

  17. [25]

    Jivnesh Sandhan. Sanskritshala: A neural sanskrit nlp toolkit with web-based interface for pedagogical and annotation purposes.Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 103–112, 2023. doi: 1...

    2023 doi

  18. [26]

    Morphology-aware segmentation and tokenization for turkic languages: A cse-guided framework (the kazakh case).Information Switzerland, 17, 2026

    Tukeyev U. Morphology-aware segmentation and tokenization for turkic languages: A cse-guided framework (the kazakh case).Information Switzerland, 17, 2026. doi: 10.3390/info17020128

    2026 doi

  19. [27]

    Multilingual tokenization efficiency in large language models: A study on indian languages.SSRN Electronic Journal, 2025

    Azhar Mohamed. Multilingual tokenization efficiency in large language models: A study on indian languages.SSRN Electronic Journal, 2025. doi: 10.2139/ssrn.5201887

    2025 doi

  20. [28]

    Yadav S. A preliminary exploration of phrase-based smt and multi-bpe segmentations through con- catenated tokenised corpora for low-resource indian languages.Conference on Machine Translation Proceedings, pages 1253–1258, 2025. doi: 10.18653/v1/2025.wmt-1.103

    2025 doi

  21. [29]

    Madhavaraj A. Subword dictionary learning and segmentation for expanding the vocabulary of automatic speech recognition in tamil and kannada.ACM Transactions on Asian and Low Resource Language Information Processing, 24, 2025. doi: 10.1145/3705312

    2025 doi

  22. [30]

    One model is all you need: Byt5-sanskrit, a unified model for sanskrit nlp tasks

    Sebastian Nehrdich. One model is all you need: Byt5-sanskrit, a unified model for sanskrit nlp tasks. Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13742–13751, 2024. doi: 10.18653/v1/2024.findings-emnlp.805

    2024 doi

  23. [31]

    Gustaf Gren. Efficient low-resource language models using tokenizer transfer.Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 639–648, 2026. doi: 10.18653/v1/2026.eacl-srw.49

    2026 doi

  24. [32]

    Arnab Sen. Architectural evaluation of subword tokenization and compact language models (clms) for resource-constrained nlp deployment.International Journal of Innovative Science and Research Technology, pages 602–609, 2025. doi: 10.38124/ijisrt/25nov578

    2025 doi

  25. [33]

    Georgy Andryushchenko. Evaluating tokenizer adaptation methods for large language models on low-resource programming languages.Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 823–833, 2025. do...

    2025 doi

  26. [34]

    Multilingual denoising pre-training for neural machine translation

    Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. InTransactions of ACL, volume 8, pages 726–742, 2020. 32

    2020

  27. [35]

    Dávid Držík. Dictionary-based byte-pair encoding tokenizer for morphologically rich languages.2025 International Conference on Electrical and Computer Engineering Researches (ICECER), pages 1–7,

    2025

  28. [36]

    Nlp - tokenization and subword models.IDA-CL Working Papers, 2025

    Joerg Osterrieder. Nlp - tokenization and subword models.IDA-CL Working Papers, 2025. doi: 10.24818/ida-cl/2025.77

    2025 doi

  29. [37]

    Narzary S. Bpe and morphologically segmented phrase based statistical machine translation system for indian languages to resource constrained language bodo.Multimedia Tools and Applications, 84: 29715–29732, 2025. doi: 10.1007/s11042-024-20277-w

    2025 doi

  30. [38]

    A morphological tokenizer for generating vocabularies for large language models in spanish.Procesamiento del Lenguaje Natural, 75:29–40, 2025

    Óscar García-Sierra, Ana Fernández-Pampillón Cesteros, and Miguel Ortega-Martín. A morphological tokenizer for generating vocabularies for large language models in spanish.Procesamiento del Lenguaje Natural, 75:29–40, 2025. URL http://journal.sepln.org/sepln/ojs/ojs/index.php/...

    2025

  31. [39]

    Maibert: A pre-training corpus and language model for low-resourced maithili language

    Sumit Yadav. Maibert: A pre-training corpus and language model for low-resourced maithili language. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026), pages 444–452, 2026. doi: 10.18653/v1/2026.loreslm-1.38

    2026 doi

  32. [40]

    Spade-bert: Multilingual bert-based model with trigram-sensitive tokenization, tuned for depression detection in spanish texts.AI Switzerland, 7, 2026

    Reyes-Vera A. Spade-bert: Multilingual bert-based model with trigram-sensitive tokenization, tuned for depression detection in spanish texts.AI Switzerland, 7, 2026. doi: 10.3390/ai7020048

    2026 doi

  33. [41]

    Durga Prasad Manukonda. Enhancing multilingual natural language processing with custom subword tokenization: Subword2vec and bilstm integration for lightweight and streamlined approaches.2024 6th International Conference on Natural Language Processing (ICNLP), pages 366–371, 2...

    2024

  34. [42]

    Analysis of subword tokenization approaches for turkish language.2023 31st Signal Processing and Communications Applications Conference (SIU), pages 1–4, 2023

    Erencan Erkaya. Analysis of subword tokenization approaches for turkish language.2023 31st Signal Processing and Communications Applications Conference (SIU), pages 1–4, 2023. doi: 10.1109/ siu59756.2023.10223973. 33

    2023

  35. [2024]

    doi: 10.18653/v1/2024.emnlp-main.925

    2024 doi

  36. [2025]

    doi: 10.1109/icecer65523.2025.11401129

    2025

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.