Pith. sign in

REVIEW 3 major objections 5 minor 30 references

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SUTRA's tokenizer beats GPT-4o and 10 other LLM tokenizers in 14 of India's 22 official languages.

desk verdict The SUTRA-wins-in-14-languages claim is not supported by the paper's own numbers: the NSL metric is misimplemented, every language is one hand-picked sentence, and the summary figure contradicts the results table. read the letter →

arxiv 2411.12240 v2 pith:ZC3FCFZW submitted 2024-11-19 cs.CL cs.AI

S. Tamang , D. J. Bora This is my paper · ORCID
classification cs.CLcs.AI
keywords tokenizerevaluationNormalizedSequenceLengthIndiclanguagesmultilingualLLMsSUTRAGPT-4obytepairencodinglanguagemodelefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks a narrow, practical question: among 12 large language models, whose tokenizer splits Indian-language text into the fewest pieces? It compares each model's tokenizer on one example sentence per official language, using Normalized Sequence Length (NSL), the ratio of a tokenizer's sequence length to a reference tokenizer's. The central claim is that SUTRA wins in 14 languages, more than any other tokenizer, including Indic-specific models, and that GPT-4o clearly improves on GPT-4 for Indian languages. If true, this means a general multilingual model can be more token-efficient for India than models built specifically for Indic text, and tokenizer choice deserves explicit design attention.

What carries the argument

The load-bearing instrument is Normalized Sequence Length (NSL), defined as $c_{\lambda\beta}=\frac{\sum_i \mathrm{length}(T_\lambda(D_i))}{\sum_i \mathrm{length}(T_\beta(D_i))}$ for a target tokenizer $T_\lambda$, a baseline tokenizer $T_\beta$, and examples $D_i$; lower NSL means fewer tokens per text. The paper pairs NSL with a deliberately minimal data setup: a single curated sentence in each of the 22 Eighth-Schedule languages, tokenized through each model's tokenizer and ranked. This lets a language-level leaderboard be built from per-sentence token counts.

What would settle it

Re-run the same NSL comparison on a corpus of at least 100 sentences per language drawn from independent news and literature sources. If SUTRA's count of lowest-NSL languages falls below 14, or GPT-4o's lead over GPT-4 reverses, the paper's central claim is refuted.

Watch

Extended reading notes

Core claim

The paper reports an empirical leaderboard: using NSL as the yardstick, SUTRA produces the shortest token sequences in 14 of the 22 official languages, while GPT-4o is best in 5 languages and Nanda in 6, with Project Indus, OpenHathi, MahaMarathi, Indic Gemma, Phi-3.5-MoE, and Airavata each winning in a handful (totals exceed 22 because multiple tokenizers can tie at the best value). The same data show GPT-4o's NSL values consistently below its predecessor GPT-4's, and Project Indus's tokenizer below 1.0 only for the eight Devanagari-script languages. The paper takes these results as evidence that general multilingual tokenizers can outperform Indic-specific ones, and that tokenizer engineering should be a first-class concern for Indic language support.

Load-bearing premise

The entire ranking assumes that one hand-picked sentence per language is representative enough that the resulting NSL values describe the language, not just that sentence.

Editorial extensions

If this is right

  • For users and providers, SUTRA-style token efficiency implies faster decoding and lower per-request cost on Indic text, since fewer tokens means fewer generation steps and smaller context.
  • The GPT-4o versus GPT-4 comparison implies tokenization for Indian languages was substantially retooled between model generations, and the improvement is visible without any fine-tuning.
  • The Project Indus pattern implies single-script training yields efficient tokenization only inside that script, and poor coverage elsewhere.
  • The leaderboard method can rank tokenizers before any benchmark evaluation, making token economy a cheap screening signal for multilingual model quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the evaluation uses exactly one hand-picked sentence per language, the per-language winner counts are best read as rankings of those sentences, not yet of whole languages.
  • The reported win totals exceed 22 because of ties, so a tie-breaking rule would make the headline 'SUTRA wins 14 languages' unambiguous.
  • If token economy tracks training-data coverage, NSL could serve as a diagnostic for which languages a tokenizer's vocabulary actually serves, though the paper does not test that correlation.
  • A direct extension would measure end-to-end latency and cost with the winning tokenizers to verify that lower NSL translates into real throughput gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper evaluates the tokenizers of 12 large language models across India's 22 official languages, using a metric called Normalized Sequence Length (NSL) defined in Eq. (1) as the ratio of token counts of a candidate tokenizer to a baseline tokenizer. The authors report NSL values in Table 2 and conclude that SUTRA outperforms all other models, 'excelling in 14 languages,' with secondary observations about GPT-4o versus GPT-4 and Project Indus. The evaluation is based on one hand-picked example sentence per language, listed in Appendix A.2, and the results are summarized in Table 2 and Figure 3.

Significance. The question of tokenizer efficiency for Indian official languages is timely and practically important, and the scope—12 tokenizers across 22 languages—is in principle valuable. If the central claim were supported by a sound, reproducible evaluation, the comparison of Indic-specific and multilingual tokenizers would be a useful reference for the community. However, the current manuscript does not deliver such an evaluation: the reported metric is not the defined NSL, the data are single sentences with no variance estimation, and the summary figure contradicts the detailed table. The positive contribution is therefore limited to a list of token counts for 22 hand-picked sentences, which is not sufficient to establish the stated conclusions.

major comments (3)
  1. [Section 3.3 and Table 2]
  2. [Section 3.1 and Appendix A.2]
  3. [Section 4, Figure 3 versus Table 2]
minor comments (5)
  1. [Throughout]
  2. [Section 4]
  3. [References]
  4. [Appendix A.2]
  5. [Section 3.3]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical token-count benchmark whose rankings are read directly from observed tokenizer outputs.

full rationale

The paper is an empirical measurement study, not a derivation. Section 3.3 defines NSL as a ratio of encoded-sequence lengths, and the values in Table 2 are computed directly from token counts produced by each tokenizer on the fixed example texts listed in Appendix A.2. The central claim that SUTRA "excels in 14 languages" is a count of the minimum-NSL entries in Table 2, i.e., a direct reading of measured data rather than a quantity constructed from the conclusion itself. There is no fitted parameter, no predicted quantity that is statistically forced by a fit, no uniqueness theorem, and no ansatz smuggled in through a citation. The only author self-citation, reference [3] (Tamang and Bora 2024), appears in Section 3.3 where the authors say "our work is extending the previous works by [3], we have chosen to go with the NSL metric"; it is used only to motivate the choice of metric, while the formal NSL definition is independently attributed to Dagan et al. [12]. Thus no load-bearing premise rests on the authors' own prior work. The one-example-per-language design and the internal mismatches between Table 2 minima and the win counts in Figure 3 are substantive methodological and reporting concerns, but they are not circularity: the rankings still reduce to direct token-count observations rather than to the assertions being evaluated. The paper is therefore self-contained as an empirical comparison, and no circular step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three unstated assumptions about the evaluation design. First, that one example sentence per language is representative of that language. Second, that the tokenizer APIs were used in a way that reflects production tokenization, with no special tokens or wrappers. Third, that the NSL values in Table 2 are computed with a well-defined normalization, although the stated formula and the actual numbers diverge. No free parameters are fitted, and no new entities are introduced.

assumptions (3)
  • domain assumption A single hand-picked sentence per language is representative of tokenizer efficiency for that language
    Section 3.1 compiles exactly one example text per language; all per-language NSL values in Table 2 derive from that one sentence, so the ranking is a one-sample measurement.
  • domain assumption The tokenizers were invoked in their production form without extra tokens or wrappers
    The paper does not describe the API calls, special-token handling, or model versions; it assumes the token counts reflect the raw tokenizer, Section 3.2.
  • domain assumption The NSL values correspond to a well-defined ratio
    Equation (1) defines NSL as a ratio of two tokenizers, but the reported values match tokens-per-character; the baseline tokenizer T-beta is never named, so the reader cannot tell which normalization was used, Section 3.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages." pith.science (2026). https://pith.science/paper/ZC3FCFZW

@misc{pith2026241112240,
  author       = {Pith},
  title        = {Pith review of: Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZC3FCFZW}},
  note         = {Machine review of arXiv:2411.12240}
}
read the original abstract

Large Language Models (LLMs) based on transformer architectures have revolutionized a variety of domains, with tokenization playing a pivotal role in their pre-processing and fine-tuning stages. In multilingual models, particularly those tailored for Indic languages, effective tokenization is crucial for optimizing performance. This paper presents a comprehensive evaluation of tokenizers used by 12 LLMs across all 22 official languages of India, with a focus on comparing the efficiency of their tokenization processes. We employed the Normalized Sequence Length (NSL) as a key metric in our analysis. Our findings reveal that the SUTRA tokenizer outperforms all other models, including several Indic-specific models, excelling in 14 languages. Notable insights include the SUTRA tokenizer's superior handling of Indic languages, GPT-4o's advancement over its predecessor GPT-4 in processing Indian languages, and the limited performance of Project Indus in certain languages. This study underscores the critical importance of developing targeted tokenization strategies for multilingual and Indic-centric models, laying the groundwork for future improvements in tokenizer design to enhance linguistic coverage and model efficiency.

Figures

Figures reproduced from arXiv: 2411.12240 by the authors.

Figure 1
Figure 1. Evaluation pipeline: (1) We collect example texts for all 22 languages. (2) We send the example texts to the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Assamese text used for evaluating tokenizer performance. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Number of Best Performances Achieved by Each Tokenizer Across 22 Languages. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (22 more)
Figure 4
Figure 4. Figure 4: Number of tokens required for a single example text in Assamese. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Number of tokens required for a single example text in Bengali. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Number of tokens required for a single example text in Bodo. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Number of tokens required for a single example text in Dogri. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Number of tokens required for a single example text in Gujarati. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Number of tokens required for a single example text in Hindi. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Number of tokens required for a single example text in Kannada. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Number of tokens required for a single example text in Kashmiri. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Number of tokens required for a single example text in Konkani. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 13
Figure 13. Figure 13: Number of tokens required for a single example text in Maithili. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 14
Figure 14. Figure 14: Number of tokens required for a single example text in Malayalam. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]
Figure 15
Figure 15. Figure 15: Number of tokens required for a single example text in Manipuri. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p010_15.png]
Figure 16
Figure 16. Figure 16: Number of tokens required for a single example text in Marathi. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]
Figure 17
Figure 17. Figure 17: Number of tokens required for a single example text in Nepali. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p011_17.png]
Figure 18
Figure 18. Figure 18: Number of tokens required for a single example text in Odia. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p011_18.png]
Figure 19
Figure 19. Figure 19: Number of tokens required for a single example text in Punjabi. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p011_19.png]
Figure 20
Figure 20. Figure 20: Number of tokens required for a single example text in Sanskrit. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p012_20.png]
Figure 21
Figure 21. Figure 21: Number of tokens required for a single example text in Santali. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p012_21.png]
Figure 22
Figure 22. Figure 22: Number of tokens required for a single example text in Sindhi. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p012_22.png]
Figure 23
Figure 23. Figure 23: Number of tokens required for a single example text in Tamil. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p012_23.png]
Figure 24
Figure 24. Figure 24: Number of tokens required for a single example text in Telugu. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p013_24.png]
Figure 25
Figure 25. Figure 25: Number of tokens required for a single example text in Urdu. Lower values are better. [PITH_FULL_IMAGE:figures/full_fig_p013_25.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 12 canonical work pages

  1. [1]

    Future applications of generative large language models: A data-driven case study on ChatGPT,

    F. Chiarello, V . Giordano, I. Spada, S. Barandoni, and G. Fantoni, "Future applications of generative large language models: A data-driven case study on ChatGPT," Technovation, vol. 133, p. 103002, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S016649722400052X. [Accessed: Nov. 12, 2024]

    work page 2024

  2. [2]

    A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges,

    Y . Nie, Y . Kong, X. Dong, J. M. Mulvey, H. V . Poor, Q. Wen, and S. Zohren, "A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges," arXiv preprint arXiv:2406.11903, 2024. [Online]. Available: https://arxiv.org/abs/2406.11903. [Accessed: Nov. 12, 2024]

    arXiv 2024

  3. [3]

    Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language,

    S. Tamang and D. J. Bora, "Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language," arXiv preprint arXiv:2410.03718, 2024. [Online]. Available: https://arxiv.org/abs/2410. 03718

    arXiv 2024

  4. [4]

    Large Language Model Tokenizer Bias: A Case Study and Solution on GPT-4o,

    J. Yang, Z. Wang, Y . Lin, and Z. Zhao, "Large Language Model Tokenizer Bias: A Case Study and Solution on GPT-4o," arXiv preprint arXiv:2406.11214, 2024. [Online]. Available: https://arxiv.org/abs/2406.11214. [Accessed: Nov. 13, 2024]

    arXiv 2024

  5. [5]

    Fast WordPiece Tokenization,

    X. Song, A. Salcianu, Y . Song, D. Dopson, and D. Zhou, “Fast WordPiece Tokenization,” 2021. [Online]. Available: https://arxiv.org/abs/2012.15524

    arXiv 2021

  6. [6]

    Better Than Whitespace: Information Retrieval for Languages without Custom Tokenizers

    O. Ogundepo, X. Zhang, and J. Lin, “Better Than Whitespace: Information Retrieval for Languages without Custom Tokenizers,” 2022. [Online]. Available:https://arxiv.org/abs/2210.05481

    work page Pith review arXiv 2022

  7. [7]

    Theoretical Analysis of Byte-Pair Encoding,

    L. Kozma and J. V oderholzer, “Theoretical Analysis of Byte-Pair Encoding,” 2024. [Online]. Available:https: //arxiv.org/abs/2411.08671

    arXiv 2024

  8. [8]

    A Formal Perspective on Byte-Pair Encoding

    V . Zouhar, C. Meister, J. L. Gastaldi, L. Du, T. Vieira, M. Sachan, and R. Cotterell, “A Formal Perspective on Byte-Pair Encoding,” 2024. [Online]. Available: https://arxiv.org/abs/2306.16837

    work page Pith review arXiv 2024

Show all 30 references
  1. [9]

    IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages,

    H. Singh, N. Gupta, S. Bharadwaj, D. Tewari, and P. Talukdar, "IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages," arXiv preprint arXiv:2404.16816, 2024. [Online]. Available: https://arxiv.org/abs/2404.16816

    2024 arXiv

  2. [10]

    EU Tokenizer Performance,

    Occiglot, "EU Tokenizer Performance," [Online]. Available: https://occiglot.eu/posts/eu_tokenizer_ perfomance/. Accessed: Nov. 17, 2024

    2024

  3. [11]

    Tokenizer performance on EU languages,

    Occiglot, "Tokenizer performance on EU languages," Occiglot Blog, Sep. 26, 2023. [Online]. Available: https: //occiglot.eu/posts/eu_tokenizer_perfomance/

    2023

  4. [12]

    Getting the most out of your tokenizer for pre-training and domain adaptation,

    G. Dagan, G. Synnaeve, and B. Rozière, "Getting the most out of your tokenizer for pre-training and domain adaptation," arXiv preprint arXiv:2402.01035 , 2024. [Online]. Available: https://arxiv.org/abs/2402. 01035. 6 Evaluating Tokenizer Performance of Large Language Models i...

    2024 arXiv

  5. [13]

    Exploring the New Frontier of AI: OpenAI’s GPT-4-O for Indic Languages,

    Microsoft, "Exploring the New Frontier of AI: OpenAI’s GPT-4-O for Indic Languages," Azure AI Blog, Oct. 30,

  6. [14]

    GPT-4 Technical Report,

    OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, et al. , "GPT-4 Technical Report," arXiv preprint arXiv:2303.08774, 2024. [Online]. Available: https://arxiv.org/abs/2303.08774

    2024 arXiv

  7. [15]

    SUTRA: Scalable Multilingual Language Model Architecture,

    A. Bendale, M. Sapienza, S. Ripplinger, S. Gibbs, J. Lee, and P. Mistry, "SUTRA: Scalable Multilingual Language Model Architecture," arXiv preprint arXiv:2405.06694, 2024. [Online]. Available: https://arxiv.org/abs/ 2405.06694

    2024 arXiv

  8. [16]

    Available at: https: //arxiv.org/abs/2401.12345

    Gemini Team et al., Gemini: A Family of Highly Capable Multimodal Models , 2024. Available at: https: //arxiv.org/abs/2401.12345

    2024 arXiv

  9. [17]

    The Llama 3 Herd of Models

    Dubey, Abhimanyu, et al. The Llama 3 Herd of Models. arXiv preprint, 2024. Available at: https://arxiv. org/abs/2407.21783

    2024 arXiv

  10. [18]

    Large Language Models: A Survey,

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao, "Large Language Models: A Survey," arXiv preprint arXiv:2402.06196, 2024. [Online]. Available: https://arxiv.org/abs/2402.06196

    2024 arXiv

  11. [19]

    A Comprehensive Overview of Large Language Models,

    Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian, "A Comprehensive Overview of Large Language Models," arXiv preprint arXiv:2307.06435, 2024. [Online]. Available: https://arxiv.org/abs/2307.06435

    2024 arXiv

  12. [20]

    Multilingual Tokenization Efficiency in Large Language Models: A Study on Indian Languages,

    AdaSci, "Multilingual Tokenization Efficiency in Large Language Models: A Study on Indian Languages," [Online]. Available: https://adasci.org/ multilingual-tokenization-efficiency-in-large-language-models-a-study-on-indian-languages/

  13. [21]

    Impact of Tokenization on Language Models: An Analysis for Turkish,

    Cagri Toraman, Eyup Halit Yilmaz, Furkan ¸ Sahinüç, and Oguzhan Ozcelik, "Impact of Tokenization on Language Models: An Analysis for Turkish," ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), vol. 22, no. 4, article 116, pp. 1–21, Mar. 2023....

    2023 doi

  14. [22]

    Generative Models For Indic Languages: Evaluating Content Generation Capabilities

    Bhat, Savita, Vasudeva Varma, and Niranjan Pedanekar. "Generative Models For Indic Languages: Evaluating Content Generation Capabilities." In *Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing*, edited by Ruslan Mitkov and Galia...

    2023

  15. [23]

    IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages

    Singh, Harman, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. "IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages." In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis...

    2024

  16. [24]

    Indic QA Benchmark: A Multilingual Benchmark to Evaluate Question Answering Capability of LLMs for Indic Languages

    Singh, Abhishek Kumar, Rudra Murthy, Vishwajeet Kumar, Jaydeep Sen, and Ganesh Ramakrishnan. "Indic QA Benchmark: A Multilingual Benchmark to Evaluate Question Answering Capability of LLMs for Indic Languages." arXiv preprint, 2024. https://arxiv.org/abs/2407.13522

    2024 arXiv

  17. [25]

    IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages

    Kumar, Aman, Himani Shrotriya, Prachi Sahu, Amogh Mishra, Raj Dabre, Ratish Puduppully, Anoop Kunchukut- tan, Mitesh M. Khapra, and Pratyush Kumar. "IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages." In *Proceedings of the 2022 Conference on E...

    2022

  18. [26]

    Evaluating Various Tokenizers for Arabic Text Classification

    Alyafeai, Zaid, Maged S. Al-shaibani, Mustafa Ghaleb, and Irfan Ahmad. "Evaluating Various Tokenizers for Arabic Text Classification." arXiv preprint, 2021. https://arxiv.org/abs/2106.07540

    2021 arXiv

  19. [27]

    Eighth Schedule,

    Government of India, "Eighth Schedule," [Online]. Available: https://www.mea.gov.in/Images/pdf1/S8. pdf. Accessed: Dec. 5, 2023

    2023

  20. [28]

    PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data,

    I. Watts, V . Gumma, A. Yadavalli, V . Seshadri, M. Swaminathan, and S. Sitaram, “PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data,” arXiv preprint arXiv:2406.15053, 2024. [Online]. Available: https://arxiv.org/abs/...

    2024 arXiv

  21. [29]

    জীৱনৰ পিৰসেৰ মািহত হাৱােটা বানীয়।

    DeepLearning.ai, Startup TWO AI Launches SUTRA: A Multilingual Model for South Asian Markets , https://www.deeplearning.ai/the-batch/ startup-two-ai-launches-sutra-a-multilingual-model-for-south-asian-markets/ , Accessed: 2024-11-18. 7 Evaluating Tokenizer Performance of Large...

    1930

  22. [2024]

    Available: https://techcommunity.microsoft.com/blog/azure-ai-services-blog/ exploring-the-new-frontier-of-ai-openais-gpt-4-o-for-indic-languages/4142383

    [Online]. Available: https://techcommunity.microsoft.com/blog/azure-ai-services-blog/ exploring-the-new-frontier-of-ai-openais-gpt-4-o-for-indic-languages/4142383

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.