Paper Citation Record · LEDGER

Comparative analysis of subword tokenization approaches for Indian languages

As of 17 August 2026, this Paper Citation Record lists 47 of 47 outbound references and 0 inbound Pith citation observations for arXiv:2505.16868.

A citation records a reference. It does not transfer a finding from one paper to another.

pith.paper-citation-record.v1
2505.16868 v1

Coverage vector

measured 47 of 47 reference resolution

Typed states for the displayed outbound observations.

Source: paper_references, paper_reference_links, observed 2026-08-07T14:56:45.919746Z

measured 47 of 47 standing notices

One-hop event checks from named stored sources.

Source: scholarly_work_events, retraction_status_cache, observed 2026-08-17T06:30:58.91139+00:00

measured 0 of 0 inbound itemization

Pith citing papers itemized under the disclosed page cap.

Source: paper_references, paper_reference_links

measured 0 of 1 external citation measurements

A source-named dated measurement, never combined with another source.

Source: cited_works

Reference resolution

47 of 47 outbound references displayed

verified exact2
verified fuzzy17
unresolved27
parse uncertain0
malformed identifier0
metadata mismatch1

External citation measurements

No source-named external measurement is stored.

Outbound references

Observation fb1ca4a8-9723-42c3-a201-cce50e45c2b7 · outbound

This paper cites an unresolved cited work.

Comparative analysis of subword tokenization approaches for Indian languages Unresolved cited work

Reference 1

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:56:53.598816Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 0156cc44-19ec-44c7-940b-1662bb841aaf · outbound

This paper cites An awkward disparity between BLEU/RIBES scores and human judgements in machine translation.

Comparative analysis of subword tokenization approaches for Indian languages An awkward disparity between BLEU/RIBES scores and human judgements in machine translation

Reference 2

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:53.378184Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation c4150d97-819e-4845-9aaf-13846d41016b · outbound

This paper cites METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.

Comparative analysis of subword tokenization approaches for Indian languages METEOR: An automatic metric for MT evaluation with improved correlation with human judgments

Reference 3

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:53.114813Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation bb282389-3dcd-4731-9c10-a63a2318367b · outbound

This paper cites A study of translation edit rate with targeted human annotation.

Comparative analysis of subword tokenization approaches for Indian languages A study of translation edit rate with targeted human annotation

Reference 4

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:52.849174Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 585289e0-bb35-4ec9-81ae-36141b49a028 · outbound

This paper cites chrF: character n -gram F-score for automatic MT evaluation.

Comparative analysis of subword tokenization approaches for Indian languages chrF: character n -gram F-score for automatic MT evaluation

Reference 5

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:52.584017Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation e6ea1fe1-48fa-4e29-9ab5-29f925f3497f · outbound

This paper cites COMET: A Neural Framework for MT Evaluation.

Comparative analysis of subword tokenization approaches for Indian languages COMET: A Neural Framework for MT Evaluation

Reference 6

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:41.199975Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 1ebcd9e6-1d42-4f66-ad66-bf03338a7f47 · outbound

This paper cites an unresolved cited work.

Comparative analysis of subword tokenization approaches for Indian languages Unresolved cited work

Reference 7

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:56:52.396246Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 58ae7b42-df44-452a-afe3-5a15d57f399f · outbound

This paper cites No Language Left Behind: Scaling Human-Centered Machine Translation.

Comparative analysis of subword tokenization approaches for Indian languages No Language Left Behind: Scaling Human-Centered Machine Translation

Reference 8

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:41.422461Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation d5bd4a5b-0163-4bf4-8052-2839630a70dc · outbound

This paper cites Neural Machine Translation of Rare Words with Subword Units.

Comparative analysis of subword tokenization approaches for Indian languages Neural Machine Translation of Rare Words with Subword Units

Reference 9

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:41.529877Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 1f70bd03-c8ea-47fa-9067-88fc3f01d8a4 · outbound

This paper cites K., & Patra, B.

Comparative analysis of subword tokenization approaches for Indian languages K., & Patra, B

Reference 10

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:52.194389Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 4ef9b725-680d-4f7a-a34d-beb3d2db41b6 · outbound

This paper cites B., Panda, D., Mishra, T.

Comparative analysis of subword tokenization approaches for Indian languages B., Panda, D., Mishra, T

Reference 11

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:51.927502Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 551d8edb-cb64-4d4c-b150-1f71133c0996 · outbound

This paper cites B., Biradar, A., Mishra, T.

Comparative analysis of subword tokenization approaches for Indian languages B., Biradar, A., Mishra, T

Reference 12

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:51.655445Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 1afd84be-9fb9-4b5d-9950-761ae9f6e73a · outbound

This paper cites Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation.

Comparative analysis of subword tokenization approaches for Indian languages Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Reference 13

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:41.907403Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 49cb7729-eb5b-446c-831a-5860104c1ea3 · outbound

This paper cites an unresolved cited work.

Comparative analysis of subword tokenization approaches for Indian languages Unresolved cited work

Reference 14

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:56:51.339239Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 9dc1623c-3682-40d0-a077-31310bb4604f · outbound

This paper cites an unresolved cited work.

Comparative analysis of subword tokenization approaches for Indian languages Unresolved cited work

Reference 15

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:56:51.076132Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 69713da9-d7fc-4d45-9cc8-78893e6d380e · outbound

This paper cites Statistical machine translation.

Comparative analysis of subword tokenization approaches for Indian languages Statistical machine translation

Reference 16

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:50.872502Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 3bc8e54e-839f-4145-94cc-aecdce8c0b3f · outbound

This paper cites Neural Machine Translation by Jointly Learning to Align and Translate.

Comparative analysis of subword tokenization approaches for Indian languages Neural Machine Translation by Jointly Learning to Align and Translate

Reference 17

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:42.262089Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation d9debab4-e691-4772-88d1-ae86f1796e02 · outbound

This paper cites an unresolved cited work.

Comparative analysis of subword tokenization approaches for Indian languages Unresolved cited work

Reference 18

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:56:50.602641Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 1606030e-d832-4f04-ac59-f19e4594868e · outbound

This paper cites Massively Multilingual Neural Machine Translation.

Comparative analysis of subword tokenization approaches for Indian languages Massively Multilingual Neural Machine Translation

Reference 19

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:42.479881Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation f6626084-c8ea-48df-9bdd-a08cb8a77682 · outbound

This paper cites an unresolved cited work.

Comparative analysis of subword tokenization approaches for Indian languages Unresolved cited work

Reference 20

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:56:50.349360Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation b1c26973-80a4-4afe-bd67-e6ecc1701507 · outbound

This paper cites SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing.

Comparative analysis of subword tokenization approaches for Indian languages SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Reference 21

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:42.668489Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation ba047e57-4187-44eb-8d9a-ebeefb80b38f · outbound

This paper cites Machine Translation Approaches and Survey for Indian Languages.

Comparative analysis of subword tokenization approaches for Indian languages Machine Translation Approaches and Survey for Indian Languages

Reference 22

Resolution: metadata mismatch; local_arxiv, observed 2026-08-07T14:56:46.539097Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation d6f7ee31-1969-458a-9ec1-855f6315827b · outbound

This paper cites an unresolved cited work.

Comparative analysis of subword tokenization approaches for Indian languages Unresolved cited work

Reference 23

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:56:50.100829Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 9fcba593-8f1d-4005-ac65-798d9aa798f5 · outbound

This paper cites Morphology: Indian languages and European languages.

Comparative analysis of subword tokenization approaches for Indian languages Morphology: Indian languages and European languages

Reference 24

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:49.850961Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 17b62402-c801-424e-a09b-fefe4788404e · outbound

This paper cites Fast WordPiece Tokenization.

Comparative analysis of subword tokenization approaches for Indian languages Fast WordPiece Tokenization

Reference 25

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:43.166127Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 2edf3d60-c5b2-4672-90bd-c34226b1aa9b · outbound

This paper cites NLTK documentation.

Comparative analysis of subword tokenization approaches for Indian languages NLTK documentation

Reference 26

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:49.659129Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 24fd864b-8ebd-421d-bc1b-768dec29dbe0 · outbound

This paper cites D., Tetreault, J., & Stent, A.

Comparative analysis of subword tokenization approaches for Indian languages D., Tetreault, J., & Stent, A

Reference 27

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:49.424184Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 4f013cee-8e99-4990-b05f-77b42f49ad0b · outbound

This paper cites Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP.

Comparative analysis of subword tokenization approaches for Indian languages Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP

Reference 28

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:43.616119Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 30e5eb63-3725-4b29-b9a2-43d52bfbc586 · outbound

This paper cites Character-based Neural Machine Translation.

Comparative analysis of subword tokenization approaches for Indian languages Character-based Neural Machine Translation

Reference 29

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:43.749854Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 7a1c699a-9b42-4830-9f31-bba3923be6ca · outbound

This paper cites An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks.

Comparative analysis of subword tokenization approaches for Indian languages An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks

Reference 30

Resolution: verified exact; local_arxiv, observed 2026-08-07T14:56:46.375650Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 299d4fb3-4312-4b58-bcb0-72b75f872f79 · outbound

This paper cites BPE-Dropout: Simple and Effective Subword Regularization.

Comparative analysis of subword tokenization approaches for Indian languages BPE-Dropout: Simple and Effective Subword Regularization

Reference 31

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:44.126108Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 2354ad57-c2db-4543-93a7-5f74cd55ecca · outbound

This paper cites Byte Pair Encoding is Suboptimal for Language Model Pretraining.

Comparative analysis of subword tokenization approaches for Indian languages Byte Pair Encoding is Suboptimal for Language Model Pretraining

Reference 32

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:44.277726Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 13c3f9df-408b-40e9-a3e0-c357cbb5ea39 · outbound

This paper cites Using Integrated Gradients and Constituency Parse Trees to explain Linguistic Acceptability learnt by BERT.

Comparative analysis of subword tokenization approaches for Indian languages Using Integrated Gradients and Constituency Parse Trees to explain Linguistic Acceptability learnt by BERT

Reference 33

Resolution: verified exact; local_arxiv, observed 2026-08-07T14:56:46.136522Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 8aa51c85-b120-41dd-bf41-a894350007cd · outbound

This paper cites HAN: hierarchical association network for computing semantic relatedness.

Comparative analysis of subword tokenization approaches for Indian languages HAN: hierarchical association network for computing semantic relatedness

Reference 34

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:49.182855Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation afe603d7-9c9b-432d-bd6d-c39ba4577c88 · outbound

This paper cites Meaningless yet meaningful: Morphology grounded subword- level NMT.

Comparative analysis of subword tokenization approaches for Indian languages Meaningless yet meaningful: Morphology grounded subword- level NMT

Reference 35

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:48.952378Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 73ad5378-3fce-4a90-b788-46f1acddc5e2 · outbound

This paper cites an unresolved cited work.

Comparative analysis of subword tokenization approaches for Indian languages Unresolved cited work

Reference 36

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:56:48.737339Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 6b3376c9-85a7-4743-917c-d9eeb2bbc44e · outbound

This paper cites BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.

Comparative analysis of subword tokenization approaches for Indian languages BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Reference 37

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:44.948945Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation fa1e4781-9d82-4b9a-bd61-c3bb47ad0cab · outbound

This paper cites AI as the next GPT: a Political -Economy Perspective.

Comparative analysis of subword tokenization approaches for Indian languages AI as the next GPT: a Political -Economy Perspective

Reference 38

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:48.544357Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 5da2293b-5724-472f-a723-ab5bc8ff582d · outbound

This paper cites an unresolved cited work.

Comparative analysis of subword tokenization approaches for Indian languages Unresolved cited work

Reference 39

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:56:48.346280Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation f6f2050a-1697-458d-b091-17fafffa61ad · outbound

This paper cites fairseq: A Fast, Extensible Toolkit for Sequence Modeling.

Comparative analysis of subword tokenization approaches for Indian languages fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Reference 40

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:45.244129Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation e8ee23cc-5c85-42ad-8818-fc4235c0b593 · outbound

This paper cites an unresolved cited work.

Comparative analysis of subword tokenization approaches for Indian languages Unresolved cited work

Reference 41

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:56:48.099408Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 6914086e-43ac-4199-9df7-1636f4fdc64b · outbound

This paper cites an unresolved cited work.

Comparative analysis of subword tokenization approaches for Indian languages Unresolved cited work

Reference 42

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:56:47.821428Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 717053e7-a5b4-48d7-9fce-a8e9a5f59f03 · outbound

This paper cites Moses-Statistical Machine Translation System.

Comparative analysis of subword tokenization approaches for Indian languages Moses-Statistical Machine Translation System

Reference 43

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:47.538867Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation 1bf3a788-d361-49cf-be01-6b13ca1437e3 · outbound

This paper cites Adam: A Method for Stochastic Optimization.

Comparative analysis of subword tokenization approaches for Indian languages Adam: A Method for Stochastic Optimization

Reference 44

Resolution: unresolved; no resolver link, observed 2026-08-07T14:56:45.645407Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 68133417-a204-4aac-a1cf-e6726cb1a396 · outbound

This paper cites Post, A call for clarity in reporting BLEU scores, in Proceedings of the Third Conference on Machine Translation: Research Papers.

Comparative analysis of subword tokenization approaches for Indian languages Post, A call for clarity in reporting BLEU scores, in Proceedings of the Third Conference on Machine Translation: Research Papers

Reference 45

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:47.359452Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation a80c834e-7b5d-484e-8ecf-b16e383ec87a · outbound

This paper cites an unresolved cited work.

Comparative analysis of subword tokenization approaches for Indian languages Unresolved cited work

Reference 46

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:56:47.089618Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Observation fff5327f-622b-4c4d-8999-9b0fc0d45680 · outbound

This paper cites B., Choudhury, S., Mishra, T.

Comparative analysis of subword tokenization approaches for Indian languages B., Choudhury, S., Mishra, T

Reference 47

Resolution: verified fuzzy; raw_fallback, observed 2026-08-07T14:56:46.811586Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-17T06:30:58.91139+00:00.

Pith citing papers

No inbound Pith citation observations are available.