Pith. sign in

REVIEW 4 cited by

Neural Machine Translation with Byte-Level Subwords

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1909.03341 v2 pith:I3N4V54I submitted 2019-09-07 cs.CL

Changhan Wang , Kyunghyun Cho , Jiatao Gu This is my paper · ORCID
classification cs.CL
keywords bbpevocabularybyte-levellanguagessubwordstranslationbytescharacter
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is. We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE. In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 9 citations worldwide. Full citation record

  1. Bit-level BPE: Below the byte boundary

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Re-encoding UTF-8 byte fallbacks as shared 6-bit prefixes plus 9-bit tokens shortens CJK token sequences losslessly, at the cost of reduced tokenizer entropy and mixed wall-clock speedups.

  2. Towards a Science of Causal Interpretability in Deep Learning for Software Engineering

    cs.SE 2025-05 conditional novelty 5.0 of 10

    The dissertation presents docode, a causal interpretability method for neural code models, and uses a case study to show that some correlations between code properties and model performance are confounded rather than causal.

  3. SnipGen: A Mining Repository Framework for Evaluating LLMs for Code

    cs.SE 2025-02 conditional novelty 5.0 of 10

    A new framework mines recent GitHub commits into about 227K method-level code snippets with prompt templates for testing LLMs on code generation while reducing training-data contamination.

  4. Is There a Case for Conversation Optimized Tokenizers in Large Language Models?

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Retraining LLM tokenizers on chatbot conversation data reduces token counts by 5-10% on conversational text with minimal impact on general text.

Pith tools