Pith. sign in

REVIEW 7 cited by

Fast WordPiece Tokenization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2012.15524 v3 pith:YQXA5CKT submitted 2020-12-31 cs.CL

Xinying Song , Alex Salcianu , Yang Song , Dave Dopson , Denny Zhou This is my paper · ORCID
classification cs.CL
keywords tokenizationtextwordpiecealgorithmgeneralmethodproposealgorithms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Tokenization is a fundamental preprocessing step for almost all NLP tasks. In this paper, we propose efficient algorithms for the WordPiece tokenization used in BERT, from single-word tokenization to general text (e.g., sentence) tokenization. When tokenizing a single word, WordPiece uses a longest-match-first strategy, known as maximum matching. The best known algorithms so far are O(n^2) (where n is the input length) or O(nm) (where m is the maximum vocabulary token length). We propose a novel algorithm whose tokenization complexity is strictly O(n). Our method is inspired by the Aho-Corasick algorithm. We introduce additional linkages on top of the trie built from the vocabulary, allowing smart transitions when the trie matching cannot continue. For general text, we further propose an algorithm that combines pre-tokenization (splitting the text into words) and our linear-time WordPiece method into a single pass. Experimental results show that our method is 8.2x faster than HuggingFace Tokenizers and 5.1x faster than TensorFlow Text on average for general text tokenization.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multimodal Medical Code Tokenizer

    cs.CL 2025-02 conditional novelty 6.0 of 10

    MedTok encodes medical codes with text and graph information into a shared vector-quantized token space, improving downstream EHR prediction and medical QA when swapped in for standard tokenizers.

  2. Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A multidimensional extension of Byte Pair Encoding compresses visual token grids losslessly into shorter sequences, improving transformer-based generation FID on image and 3D datasets.

  3. EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery

    cs.CV 2025-01 reject novelty 5.0 of 10

    EndoChat is a grounded multimodal LLM for endoscopic surgery, trained on the new Surg-396K dataset and reported to outperform prior MLLMs, though its evaluation is confounded by training-data overlap.

  4. 1 bit is all we need: binary normalized neural networks

    cs.LG 2025-09 conditional novelty 4.0 of 10

    Binary normalized layers with all parameters in {0,1} reach near-32-bit accuracy on Food-101 and WikiText-103, with some caveats.

  5. Comparative analysis of subword tokenization approaches for Indian languages

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Across SMT and NMT models for 11 Indian languages, SentencePiece gives the highest BLEU for most language pairs, while BPE wins in the multilingual model.

  6. POINTS1.5: Building a Vision-Language Model towards Real World Applications

    cs.CV 2024-12 conditional novelty 4.0 of 10

    POINTS1.5-7B, a vision-language model with a NaViT-style encoder, bilingual data, and filtered instruction tuning, ranks first on OpenCompass among sub-10B models.

  7. Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages

    cs.CL 2024-11 reject novelty 4.0 of 10

    A single-sentence evaluation claims SUTRA's tokenizer is most token-efficient across 14 of India's 22 official languages, based on one example text per language.

Pith tools