Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Comparative analysis of subword tokenization approaches for Indian languages

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports a three-tokenizer, three-model comparison across 11 Indian languages and finds SentencePiece best for SMT and NMT while BPE wins for multilingual NMT.

desk verdict A useful tokenizer comparison undermined by reporting errors; the SMT majority claim does not survive its own tables. read the letter →

arxiv 2505.16868 v1 pith:6IBHAX5Z submitted 2025-05-22 cs.CL

Sudhansu Bala Das , Samujjal Choudhury , Tapas Kumar Mishra , Bidyut Kr. Patra This is my paper
classification cs.CL
keywords machinetranslationsubwordtokenizationSentencePieceBytePairEncodingWordIndianlanguagesmultilingualneuralBLEUscore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to settle a practical question: which subword tokenizer—SentencePiece, Byte Pair Encoding (BPE), or WordPiece—should be used when building machine translation systems for Indian languages. By running all three in statistical, neural, and multilingual neural models on the Samanantar corpus for 11 Indian languages and scoring with BLEU and five other metrics, the paper reports that SentencePiece wins for most language pairs in the first two model types, while BPE wins for the multilingual model. The authors also report that translations from Indian languages into English score higher than the reverse direction regardless of tokenizer. If the reported rankings are accurate, the paper supplies a concrete default choice for Indic machine translation practitioners.

What carries the argument

The load-bearing components are the three tokenizers and the evaluation metric. SentencePiece is a language-independent subword tokenizer that treats the input as a raw character stream with spaces as ordinary characters; BPE iteratively merges the most frequent adjacent character pairs to build a subword vocabulary; WordPiece builds a vocabulary by merging character pairs according to a likelihood-based score. The models that carry the comparison are a statistical machine translation pipeline and Transformer-based neural and multilingual neural models, with BLEU as the primary ranking metric. The claimed advantage of SentencePiece is that it avoids relying on pre-existing word boundaries, which matters for agglutinative Indian languages.

What would settle it

Re-run the same three tokenizers on the same Samanantar training and Flores200 test splits with identical model configurations, and check whether SentencePiece attains the highest BLEU for a majority of SMT and NMT language pairs and BPE for a majority of MNMT pairs. A faster check is to verify the WordPiece MNMT metric columns in Table 4, where the reported COMET range of 591 to 619, METEOR range of 129 to 391, and TER range of 119 to 319 lie outside the standard scales for those metrics.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is an empirical ranking: among the three subword tokenizers, SentencePiece achieves the best BLEU scores for a majority of the 22 English–Indic language pairs when training statistical and neural translation models, whereas BPE achieves the best BLEU scores for the multilingual neural model. The paper attributes SentencePiece's edge to its treatment of text as a raw character stream, which suits languages that do not use spaces between words, and BPE's edge in the multilingual setting to its segmentation strategy across many languages. It further finds that, with the same tokenizer and dataset, Indic-to-English translations outperform English-to-Indic translations for all three model types. The paper also claims to be the first attempt to compare these three tokenizers across 11 Indian languages.

Load-bearing premise

The entire conclusion is a ranking of numbers, so the one load-bearing premise is that the reported BLEU, TER, METEOR, CHRF, RIBES, and COMET values are correctly calculated and comparable across tokenizers; if those numbers are wrong, the stated winner for each model type is unsupported.

Editorial extensions

If this is right

  • SMT and NMT systems for Indian languages should default to SentencePiece if the goal is BLEU-maximizing translation quality.
  • Multilingual NMT systems covering several Indian languages should consider BPE, since it won for the MNMT configuration tested.
  • The directionality result implies that English-to-Indic translation needs more work than Indic-to-English, even with the same tokenizer and data.
  • Because the paper uses one dataset and one architecture per model type, the ranking is a practical baseline rather than a theoretical guarantee.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper ranks tokenizers only through aggregate metrics, so a different metric weighting or human evaluation might change the practical choice.
  • The consistent Indic-to-English advantage could reflect the Samanantar corpus composition rather than a general property of Indian languages, and would be worth testing on other parallel corpora.
  • An extension the authors do not test is whether the same ranking holds for Indic-to-Indic translation, which they list as future work.
  • The reported metric anomalies in the WordPiece tables suggest that the MNMT ranking in particular should be re-verified before being adopted as a guideline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper compares three subword tokenizers (BPE, WordPiece, SentencePiece) across three machine-translation frameworks (SMT, NMT, MNMT) for 11 Indian languages, using Samanantar for training and Flores200 for testing, and reports BLEU, TER, METEOR, CHRF, RIBES, and COMET scores. It claims that SentencePiece gives the best BLEU results for the majority of language pairs in SMT and NMT, while BPE is best for MNMT, and that Indic-to-English translations generally outperform English-to-Indic translations.

Significance. If the reported numbers are reliable, the paper would provide a practically useful empirical guideline for tokenizer selection in Indic machine translation, covering a broad matrix of 3 tokenizers, 3 model families, 22 translation directions, and 6 evaluation metrics on public benchmarks. The study is empirical and comparative rather than theoretical, and it does not release code or trained models; its value depends entirely on the accuracy and comparability of the tabulated scores, which currently contain several internal inconsistencies.

major comments (4)
  1. [Abstract, Section 5.2, Section 6, Tables 2-4] The central SMT claim is not supported by the reported BLEU scores. In the BLEU columns of Tables 2-4, SentencePiece is the highest-scoring tokenizer in only 10 of the 22 SMT language-pair rows, with the MR-EN row tied with WordPiece; WordPiece is also highest in 10 rows. Thus SentencePiece does not 'continuously perform better' for a majority of SMT language pairs, as stated in the abstract and conclusion. The SMT conclusion must either be corrected after verifying the tables or be substantially qualified.
  2. [Section 5.2, Table 4] The prose ranges for WordPiece MNMT are internally impossible: the text reports COMET scores of 591 to 619, METEOR of 129 to 391, and TER of 119 to 319, while Table 4's WordPiece MNMT rows have COMET roughly 0.57-0.87, METEOR roughly 0.08-0.63, and TER roughly 57.92-103.8. Such an inconsistency in the reported ranges must be resolved from the original evaluation outputs before any numerical conclusion can be trusted.
  3. [Table 4] Table 4 appears to contain transcription errors in load-bearing rows. The SMT EN-TE and TE-EN rows are identical (BLEU 5.74, TER 101.42, METEOR 0.25, CHRF 44.19, RIBES 0.4), which is implausible for two translation directions. In the NMT WordPiece block, EN-BN has BLEU 6.62 and BN-EN has BLEU 0.42, values that deviate sharply from the corresponding SentencePiece rows (17.43 and 28.61) and BPE rows (3.06 and 28.22); these entries need to be recomputed or corrected.
  4. [Section 4 and Section 5] The experimental setup is underspecified in ways that affect the rankings. The manuscript does not report subword vocabulary sizes, the number of BPE merges, SentencePiece training details, validation splits, random seeds, early stopping criteria, or whether any configuration was tuned per tokenizer. Section 5 also reports no significance tests or confidence intervals, so the small BLEU differences that determine the paper's conclusions may be within configuration or run-to-run variability.
minor comments (5)
  1. [Section 5.2] The WordPiece prose also says 'RIBES scores range from 0.08 to four,' which is impossible for a metric typically bounded above by 1; this should be corrected.
  2. [Section 2 and References] The text cites 'Ding et al [30]' for work on BPE merge counts, but reference [30] is Park et al.; the citation numbering should be checked throughout the related-work section.
  3. [Section 4.3] In the WordPiece MNMT paragraph, 'Fairseq[25]' should likely be Fairseq [40], and 'Transformer model[37]' should likely be [39]; the reference markers need to be verified.
  4. [Table 4 and Section 5.2] There are typographical errors in the table caption and prose ('MMMT', 'BLUE') that should be fixed before publication.
  5. [Section 1] The 'first attempt ever' contribution claim is difficult to verify and is stronger than the surrounding evidence warrants; it should be softened or supported by a more systematic related-work search.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: tokenizer ranking is an empirical benchmark comparison against Flores200, not a reduction to fitted inputs or self-citations.

full rationale

The paper is an empirical evaluation: it trains SMT, NMT, and MNMT systems on Samanantar and tests on Flores200, then ranks BPE, SentencePiece, and WordPiece by BLEU, TER, METEOR, CHRF, RIBES, and COMET. The central claim that SentencePiece performs best for most SMT/NMT pairs and BPE for MNMT is a reading of the reported metric tables, not a mathematical derivation from an assumed definition. No equation in the paper defines one tokenizer's performance in terms of another, and no fitted parameter is renamed as a prediction. The self-citations (refs. [10], [11], [12], [47]) are prior MT system papers by the same group, but they are cited as background and are not used as evidence for the tokenizer rankings; the rankings are determined by the new Flores200 results in Tables 2-4. The paper's internal numerical inconsistencies (e.g., COMET values reported as 591-619 in Table 4 prose, and duplicated or implausible rows in the tables) are serious correctness and reproducibility concerns, but they are not circularity: a wrong or misreported empirical result is still an empirical result, not a self-referential derivation. The 'first attempt ever' novelty claim is unsupported by a literature search in the text, but an unsupported priority claim is not circular. Therefore no circular step meets the evidentiary bar of quoting a specific reduction of the conclusion to its own input, and the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No mathematical derivation is involved, so the ledger is dominated by unstated experimental choices and domain assumptions. The missing vocabulary sizes and data splits are the most serious because they directly affect whether the tokenizer comparison is fair.

free parameters (2)
  • subword vocabulary size / number of BPE merges = not reported
    Tokenizer performance depends strongly on vocabulary size and merge counts. Section 4.3 describes using the tokenizer libraries but never states these values, so the comparison may confound tokenizer choice with vocabulary capacity.
  • training data subset per language = not reported
    Samanantar contains 49.6M sentence pairs, but the paper never states how many pairs per language were used for SMT, NMT, or MNMT. This affects low-resource language results and any claim about corpus-size effects.
assumptions (3)
  • domain assumption Flores200 is a valid held-out multilingual test set for the 11 Indian languages.
    Section 4.1 states Flores200 is used for testing, but the paper does not justify that it is representative or that all metrics are computed under identical tokenization settings.
  • domain assumption Samanantar parallel corpus is clean and sufficient for training all models after the described preprocessing.
    Section 4.2 describes cleaning steps, but no corpus statistics, noise analysis, or domain match with Flores200 is provided.
  • domain assumption The Fairseq Transformer and Moses configurations are appropriate and equivalent across tokenizer conditions.
    Section 4.3 says the same configuration is used, but key hyperparameters such as learning rate, warmup, epochs, early stopping, and beam size are not specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Comparative analysis of subword tokenization approaches for Indian languages." pith.science (2026). https://pith.science/paper/6IBHAX5Z

@misc{pith2026250516868,
  author       = {Pith},
  title        = {Pith review of: Comparative analysis of subword tokenization approaches for Indian languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6IBHAX5Z}},
  note         = {Machine review of arXiv:2505.16868}
}
read the original abstract

Tokenization is the act of breaking down text into smaller parts, or tokens, that are easier for machines to process. This is a key phase in machine translation (MT) models. Subword tokenization enhances this process by breaking down words into smaller subword units, which is especially beneficial in languages with complicated morphology or a vast vocabulary. It is useful in capturing the intricate structure of words in Indian languages (ILs), such as prefixes, suffixes, and other morphological variations. These languages frequently use agglutinative structures, in which words are formed by the combination of multiple morphemes such as suffixes, prefixes, and stems. As a result, a suitable tokenization strategy must be chosen to address these scenarios. This paper examines how different subword tokenization techniques, such as SentencePiece, Byte Pair Encoding (BPE), and WordPiece Tokenization, affect ILs. The effectiveness of these subword tokenization techniques is investigated in statistical, neural, and multilingual neural machine translation models. All models are examined using standard evaluation metrics, such as the Bilingual Evaluation Understudy (BLEU) score, TER, METEOR, CHRF, RIBES, and COMET. Based on the results, it appears that for the majority of language pairs for the Statistical and Neural MT models, the SentencePiece tokenizer continuously performed better than other tokenizers in terms of BLEU score. However, BPE tokenization outperformed other tokenization techniques in the context of Multilingual Neural Machine Translation model. The results show that, despite using the same tokenizer and dataset for each model, translations from ILs to English surpassed translations from English to ILs.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 32 canonical work pages

  1. [1]

    Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311–318, 2002

    work page 2002

  2. [2]

    An awkward disparity between BLEU/RIBES scores and human judgements in machine translation

    Tan, L., Dehdari, J., & van Genabith, J. An awkward disparity between BLEU/RIBES scores and human judgements in machine translation. In Proceedings of the 2nd Workshop on Asian Translation (WAT2015), 74–81, 2015

    work page 2015

  3. [3]

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments

    Banerjee, S., & Lavie, A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 65–72, 2005

    work page 2005

  4. [4]

    A study of translation edit rate with targeted human annotation

    Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, 223–231, 2006

    work page 2006

  5. [5]

    chrF: character n -gram F-score for automatic MT evaluation

    Popović, M. chrF: character n -gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, 392–395, 2015

    work page 2015

  6. [6]

    C., & Lavie, A

    Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. COMET: A neural framework for MT evaluation. arXiv preprint arXiv:2009.09025, 2020

    arXiv 2009

  7. [7]

    Ramesh, G. et al. Samanantar: The largest publicly available parallel corpora collection for 11 Indic languages. Transactions of the Association for Computational Linguistics, 10, 145–162, 2022

    work page 2022

  8. [8]

    Costa-jussà, M. R. et al. No language left behind: Scaling human entered machine translation. arXiv preprint arXiv:2207.04672, 2022

    arXiv 2022

Show all 47 references
  1. [9]

    Neural machine translation of rare words with subword units

    Sennrich, R., Haddow, B., & Birch, A. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  2. [10]

    K., & Patra, B

    Bala Das, S., Biradar, A., Mishra, T. K., & Patra, B. K. Improving multilingual neural machine translation system for Indic languages . ACM Transactions on Asian and Low -Resource Language Information Processing, 22(6), 1–24, 2023

    2023

  3. [11]

    B., Panda, D., Mishra, T

    Das, S. B., Panda, D., Mishra, T. K., Patra, B. K., & Ekbal, A. Multilingual Neural Machine Translation for Indic to Indic Languages. ACM Transactions on Asian and Low -Resource Language Information Processing, 2024

    2024

  4. [12]

    B., Biradar, A., Mishra, T

    Das, S. B., Biradar, A., Mishra, T. K., & Patra, B. K. NIT Rourkela machine translation (MT) system submission to WAT 2022 for MultiIndicMT: An Indic language multilingual shared task. 2022

    2022

  5. [13]

    Wu, Y . et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016

    2016 arXiv

  6. [14]

    Brown, P . F. et al. A statistical approach to machine translation. Computational Linguistics, 16(2), 79–85, 1990

    1990

  7. [15]

    Forcada, M. L. et al. Apertium: a free/open -source platform for rulebased machine translation. Machine Translation, 25, 127–144, 2011

    2011

  8. [16]

    Statistical machine translation

    Lopez, A. Statistical machine translation. ACM Computing Surveys (CSUR), 40(3), 1–49, 2008

    2008

  9. [17]

    Neural machine translation by jointly learning to align and translate

    Bahdanau, D., Cho, K., & Bengio, Y . Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014

    2014 arXiv

  10. [18]

    Johnson, M. et al. Google’s multilingual neural machine translation system: Enabling zero -shot translation. Transactions of the Association for Computational Linguistics, 5, 339–351, 2017

    2017

  11. [19]

    Massively multilingual neural machine translation

    Aharoni, R., Johnson, M., & Firat, O. Massively multilingual neural machine translation. arXiv preprint arXiv:1903.00089, 2019

    1903 arXiv

  12. [20]

    Domingo, M. et al. How much does tokenization affect neural machine translation? In International Conference on Computational Linguistics and Intelligent Text Processing , 545 –554, Springer, 2019

    2019

  13. [21]

    Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

    Kudo, T., & Richardson, J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018

    2018 arXiv

  14. [22]

    J., Anwar, W., & Durrani, N

    Khan, N. J., Anwar, W., & Durrani, N. Machine translation approaches and survey for Indian languages. arXiv preprint arXiv:1701.04290, 2017

    2017 arXiv

  15. [23]

    Chai, C. P . Comparison of text preprocessing methods. Natural Language Engineering , 29(3), 509–553, 2023

    2023

  16. [24]

    Morphology: Indian languages and European languages

    Vikram, S. Morphology: Indian languages and European languages . International Journal of Scientific and Research Publications, 3(6), 1–5, 2013

    2013

  17. [25]

    Fast wordpiece tokenization

    Song, X., Salcianu, A., Song, Y ., Dopson, D., & Zhou, D. Fast wordpiece tokenization . arXiv preprint arXiv:2012.15524, 2020

    2012 arXiv

  18. [26]

    NLTK documentation

    Bird, S., Klein, E., & Loper, E. NLTK documentation. Online: accessed April, 2008

    2008

  19. [27]

    D., Tetreault, J., & Stent, A

    Choi, J. D., Tetreault, J., & Stent, A. It depends: Dependency parser comparison using a web - based evaluation tool. In Proceedings of ACLIJCNLP, 387–396, 2015

    2015

  20. [28]

    Mielke, S. J. et al. Between words and characters: A brief history of open -vocabulary modeling and tokenization in NLP . arXiv preprint arXiv:2112.10508, 2021

    2021 arXiv

  21. [29]

    Ling, W., Trancoso, I., Dyer, C., & Black, A. W. Character-based neural machine translation. arXiv preprint arXiv:1511.04586, 2015

    2015 arXiv

  22. [30]

    An empirical study of tokenization strategies for various Korean NLP tasks

    Park, K., Lee, J., Jang, S., & Jung, D. An empirical study of tokenization strategies for various Korean NLP tasks. arXiv preprint arXiv:2010.02534, 2020

    2010 arXiv

  23. [31]

    BPE -dropout: Simple and effective subword regularization

    Provilkov, I., Emelianenko, D., & Voita, E. BPE -dropout: Simple and effective subword regularization. arXiv preprint arXiv:1910.13267 , 2019

    1910 arXiv

  24. [32]

    Byte pair encoding is suboptimal for language model pretraining

    Bostrom, K., & Durrett, G. Byte pair encoding is suboptimal for language model pretraining. arXiv preprint arXiv:2004.03720, 2020

    2004 arXiv

  25. [33]

    Nayak, A., & Timmapathini, H. P . Using integrated gradients and constituency parse trees to explain linguistic acceptability learnt by BERT. arXiv preprint arXiv:2106.07349, 2021

    2021 arXiv

  26. [34]

    HAN: hierarchical association network for computing semantic relatedness

    Gong, X., Xu, H., & Huang, L. HAN: hierarchical association network for computing semantic relatedness. In Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018

    2018

  27. [35]

    Meaningless yet meaningful: Morphology grounded subword- level NMT

    Banerjee, T., & Bhattacharyya, P . Meaningless yet meaningful: Morphology grounded subword- level NMT. In Proceedings of the Second Workshop on Subword/Character Level Models, 55–60, 2018

    2018

  28. [36]

    Gaser, M. A. S. A., & Vu, T. Subword-level Segmentation for Neural Machine Translation of Code- switched Dialectal Egyptian ArabicEnglish Text

  29. [37]

    W., Lee, K., & Toutanova, K

    Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. BERT: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  30. [38]

    AI as the next GPT: a Political -Economy Perspective

    Trajtenberg, M. AI as the next GPT: a Political -Economy Perspective. National Bureau of Economic Research, 2018

    2018

  31. [39]

    Vaswani, A. et al. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017

    2017

  32. [40]

    Ott, M. et al. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038, 2019

    1904 arXiv

  33. [41]

    Nayak, A., Timmapathini, H., Ponnalagu, K., & Venkoparao, V. G. Domain adaptation challenges of BERT in tokenization and sub-word representations of out-of-vocabulary words. In Proceedings of the First Workshop on Insights from Negative Results in NLP, 1–5, 2020

    2020

  34. [42]

    Creutz, M. et al. Morph -based speech recognition and modeling of out -of-vocabulary words across languages. ACM Transactions on Speech and Language Processing (TSLP), 5(1), 1–29, 2007

    2007

  35. [43]

    Moses-Statistical Machine Translation System

    Koehn, P ., & Hoang, H. Moses-Statistical Machine Translation System. User Manual and Code Guide, 2010

    2010

  36. [44]

    P ., & Ba, J

    Kingma, D. P ., & Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  37. [45]

    Post, A call for clarity in reporting BLEU scores, in Proceedings of the Third Conference on Machine Translation: Research Papers

    M. Post, A call for clarity in reporting BLEU scores, in Proceedings of the Third Conference on Machine Translation: Research Papers. Belgium, Brussels: Association for Computational Linguistics , Oct. 2018, pp. 186–191. [Online]. Available: https://www.aclweb.org/anthology/ W18-6319

    2018

  38. [46]

    T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P . Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P . von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, Transformers: Stateof -the-art natura...

    2020

  39. [47]

    B., Choudhury, S., Mishra, T

    Das, S. B., Choudhury, S., Mishra, T. K., & Patra, B. K. (2025, January). Investigating the Effect of Backtranslation for Indic Languages. In Proceedings of the First Workshop on Natural Language Processing for Indo-Aryan and Dravidian Languages (pp. 152-165)

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.