Pith. sign in

REVIEW 10 cited by

Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2112.10508 v1 pith:XRDMPDN3 submitted 2021-12-20 cs.CL cs.LG

Sabrina J. Mielke , Zaid Alyafeai , Elizabeth Salesky , Colin Raffel , Manan Dey , Matthias Gallé , Arun Raja , Chenglei Si
show 3 more authors
Wilson Y. Lee Benoît Sagot Samson Tan
This is my paper
classification cs.CLcs.LG
keywords approachesmanywordsapplicationscharactersmodelprocessingsubword-based
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

What are the units of text that we want to model? From bytes to multi-word expressions, text can be analyzed and generated at many granularities. Until recently, most natural language processing (NLP) models operated over words, treating those as discrete and atomic tokens, but starting with byte-pair encoding (BPE), subword-based approaches have become dominant in many areas, enabling small vocabularies while still allowing for fast inference. Is the end of the road character-level model or byte-level processing? In this survey, we connect several lines of work from the pre-neural and neural era, by showing how hybrid approaches of words and characters as well as subword-based approaches based on learned segmentation have been proposed and evaluated. We conclude that there is and likely will never be a silver bullet singular solution for all applications and that thinking seriously about tokenization remains important for many applications.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 106 citations worldwide. Full citation record

  1. FlowletFormer: Network Behavioral Semantic Aware Pre-training Model for Traffic Classification

    cs.LG 2025-08 conditional novelty 6.0 of 10

    FlowletFormer is a BERT-style pre-training model for network traffic that uses flowlet segmentation, field-level tokenization, and two self-supervised objectives, reaching state-of-the-art classification on 7 of 8 pub...

  2. FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language

    cs.CL 2025-06 conditional novelty 6.0 of 10

    An adaptive, per-language data filtering and deduplication pipeline produces multilingual LLM pre-training corpora that beat prior public datasets on 11 of 14 evaluated languages, and a 20TB, 1,868 language-script dat...

  3. Bit-level BPE: Below the byte boundary

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Re-encoding UTF-8 byte fallbacks as shared 6-bit prefixes plus 9-bit tokens shortens CJK token sequences losslessly, at the cost of reduced tokenizer entropy and mixed wall-clock speedups.

  4. Speaker-Conditioned Phrase Break Prediction for Text-to-Speech with Phoneme-Level Pre-trained Language Model

    eess.AS 2025-08 conditional novelty 5.0 of 10

    Speaker-conditioned phrasing with phoneme-level PLMs (MP BERT) improves pause prediction from F0.5 0.3719 to 0.4991, and a few-shot adapter generalizes to unseen speakers.

  5. Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    Non-targeted merge-list-free BPE inference causes minimal downstream performance loss, unlike targeted merge-list corruption.

  6. AI Agents for Conversational Patient Triage: Preliminary Simulation-Based Evaluation with Real-World EHR Data

    cs.CL 2025-06 reject novelty 5.0 of 10

    A patient simulator built from EHR vignettes was rated consistent with those vignettes in 97.7% of 519 conversations by two clinicians, while the AI triage system's top three diagnoses contained the most likely diagno...

  7. Beyond Text Compression: Evaluating Tokenizers Across Scales

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Tokenizer choice matters mostly for multilingual tasks, and 350M-parameter models can predict 2.7B model ranking on translation but not on English benchmarks.

  8. Learning Mechanism Underlying NLP Pre-Training and Fine-Tuning

    cs.CL 2025-09 conditional novelty 4.0 of 10

    Masked-token prediction errors in BERT reveal clusters of interchangeable, semantically related tokens, and the average per-token accuracy increases through the transformer layers and correlates with fine-tuning accuracy.

  9. Is There a Case for Conversation Optimized Tokenizers in Large Language Models?

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Retraining LLM tokenizers on chatbot conversation data reduces token counts by 5-10% on conversational text with minimal impact on general text.

  10. Comparative analysis of subword tokenization approaches for Indian languages

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Across SMT and NMT models for 11 Indian languages, SentencePiece gives the highest BLEU for most language pairs, while BPE wins in the multilingual model.

Pith tools