Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Bridging the Gap for Tokenizer-Free Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A tokenizer-free transformer can match word-based language models on the One Billion Word benchmark.

desk verdict A real, honestly-reported scale-up result: a 0.8B byte-level transformer ties word-level perplexity on lm1b, with a caveat about the benchmark's preprocessing that the paper itself mostly owns. read the letter →

arxiv 1908.10322 v1 pith:HXMHXSWP submitted 2019-08-27 cs.CL cs.AIcs.IRcs.LG

Dokook Choe , Rami Al-Rfou , Mandy Guo , Heeyoung Lee , Noah Constant This is my paper · ORCID
classification cs.CLcs.AIcs.IRcs.LG
keywords tokenizer-freelanguagemodelingbyte-leveltransformercharacter-levellm1bbenchmarkmodelscalingwordsimilarityopenvocabulary
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to overturn the assumption that language models need a tokenizer, whether word-level or subword, to perform well at scale. It trains a vanilla 40-layer transformer decoder to predict the next byte in raw UTF-8 input, with no word-boundary information, and reports 23.0 perplexity and 0.874 bits per byte on the lm1b test set, matching the word-level model of Baevski and Auli (2018). A sympathetic reader would take this as evidence that tokenizer-free models are not inherently inferior; the gap comes from model capacity and optimization. The paper also probes intermediate layers and shows that word-level semantic similarity emerges from next-byte prediction.

What carries the argument

The engine is a deep transformer decoder with causal attention operating on 256 byte values, with 40 layers, hidden size 1024, filter size 8192, and 16 heads, totalling about 836M parameters. Byte sequences of length 512 are sampled without sentence boundaries, and at each timestep the model predicts a distribution over the 256 possible next bytes. The argument rests on scaling: earlier byte-level transformers with 0.2B parameters reached 40.6 perplexity, while this 0.8B model reaches 23.0, showing that capacity closes the gap and that the byte-level input is not the limiting factor.

What would settle it

Run the same 40-layer byte transformer on the original uncased, untokenized text of lm1b with no UNK replacement, and compare bits per byte against a word-level model scored with an open vocabulary. A large degradation in the byte model's relative performance would show that the result depends on the corpus being closed and preprocessed rather than on tokenizer-free modeling itself.

Watch

Extended reading notes

Core claim

The central discovery is that a sufficiently large, vanilla transformer can model language at the byte level as well as word-based systems do on a standard benchmark. On the One Billion Word test set, the 40-layer byte transformer achieves 23.0 word perplexity, improving the prior byte-level result from 40.6 to within the range of word-based models, which score between 21.8 and 28.0. This is achieved without any explicit sentence-boundary or word-boundary signal, and with only 66K parameters in the byte embedding layer; the remaining model capacity does the heavy lifting. The paper takes this as evidence that tokenization is not a necessary prior for competitive language modeling.

Load-bearing premise

The benchmark corpus is not raw text: lm1b has been lowercased, tokenized, and had rare words replaced with UNK before the model ever sees it.

Editorial extensions

If this is right

  • Byte-level language models are not inherently worse than word-level language models; with enough capacity they match them on lm1b.
  • The byte-level setup removes language-dependent preprocessing and vocabulary engineering from the language-modeling pipeline.
  • Dense byte embeddings and operations can run on specialized accelerators, unlike the sparse lookup operations word-level models require.
  • Adding recurrence, along the lines of the Transformer-XL baseline cited in the paper, could extend byte-level context further because bytes form longer sequences than words.
  • Intermediate word representations extracted from the byte model capture semantic similarity, showing that word-level knowledge emerges even without word-level supervision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the result transfers beyond lm1b to truly open-vocabulary, morphologically rich text, tokenizer-free models could simplify multilingual pipelines that currently need per-language tokenizers.
  • The result suggests that tokenization is not a source of inductive bias the model absolutely needs; parameter budget and training time may be the real variables that decide whether a tokenizer helps.
  • A testable extension would compare byte-level and byte-pair-encoding transformers at equal parameter counts on the same corpus to isolate the capacity threshold at which the tokenizer's advantage disappears.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a byte-level, tokenizer-free language model: a 40-layer transformer decoder over 256 byte values, trained on the One Billion Word (lm1b) benchmark. It reports a test perplexity of 23.0 (0.874 bits per byte), matching the word-level baseline of Baevski and Auli (2018), and interprets this as evidence that tokenizer-free LMs can be competitive with word-based LMs when model capacity is sufficient. The paper also proposes extracting word representations from intermediate activations by appending a space character and evaluates these representations on word-similarity benchmarks.

Significance. If the empirical result holds, it is a meaningful data point: a standard transformer decoder consuming bytes can reach word-level perplexity on a widely used benchmark without a tokenizer, which could simplify language-modeling pipelines and open-vocabulary handling. The bits-per-byte to perplexity conversion is standard, the benchmark is standard, and the model and training details are described with enough specificity to be broadly reproducible, although no code or model weights are released and no variance is reported. The claim that capacity alone drives the improvement is not rigorously supported, because the comparison to Al-Rfou et al. (2019) is not a controlled ablation. The strongest generalization, that the model works on raw text, is conditioned on the benchmark's preprocessing and should be stated as such.

major comments (3)
  1. [Section 3.1 and Section 6] The abstract and conclusion claim that the model 'reads raw byte-level input without the use of any text preprocessing,' but the evaluation is performed on lm1b after its standard preprocessing: the corpus is lowercased, tokenized, and rare words are replaced with UNK tokens (Section 3.1). The reported 23.0 perplexity and 0.874 bits/byte are therefore likelihoods over the bytes of a preprocessed, closed-vocabulary text, not over raw text. The statement in Section 3 that these metrics provide 'a lower bound on the utility of our approach in realistic settings' is an assumption, not a demonstrated property, since replacing rare words with a highly frequent UNK token can change the difficulty of byte prediction. I request that the raw-text claim be removed or explicitly conditioned on the benchmark preprocessing, or that the model be evaluated on an unprocessed corpus.
  2. [Section 4, Table 2] The sentence 'We observe that tokenizer-free LM performance improves significantly (40.6 to 23.0) when the model capacity is increased from 0.2B to 0.8B parameters' attributes the improvement to capacity, but the two models compared differ in depth, architecture details, training procedure, and possibly context handling, not only in parameter count. The causal role of capacity is therefore not established. Please either present controlled ablations over depth/width or rephrase the conclusion to say that the larger proposed model performs better, without attributing the gain specifically to capacity.
  3. [Section 4, results reporting] The paper reports a single run without variance or the number of random seeds. Because the central claim is that the byte-level model is 'on par' with a word-level baseline (23.0 versus 23.0 in perplexity), a single set of point estimates provides no indication of whether the difference is meaningful. Please report multiple runs with standard deviations, or temper the comparative wording accordingly.
minor comments (5)
  1. [Introduction and Section 2] The paper alternates between 'character-based' and 'byte-level' terminology; since the model operates on UTF-8 bytes, and non-ASCII characters occupy multiple bytes, the term 'character' is not accurate for the general case.
  2. [Section 2.2] The description of windowed prediction is confusing: 'for each window of context size c, we score (stride = c/2) characters in parallel' should be clarified to explain that each window scores the second half of its positions and the window advances by c/2 positions.
  3. [Section 5] The word-representation extraction requires feeding single words to the model and appending a space, which means word boundaries are used at evaluation time; this should be acknowledged in the text, since the model itself is trained without word-boundary information.
  4. [Section 5] Figure 1 is referenced but not visible in the manuscript text, and Table 3 does not report a Swivel score for the ws353 dataset; please ensure the figure is included and the table is complete.
  5. [Section 5] There is a typo in the text: 'In the first phrase' should read 'In the first phase.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central result is an empirical measurement on an external benchmark, with no fitted quantity presented as a prediction and no load-bearing self-citation.

full rationale

The paper's central claim is that a large byte-level transformer can match word-level baselines on lm1b, and this is supported by direct test-set measurements reported in Table 2. The only conversion between bits/byte and perplexity is a mathematical identity derived from the amount of information in the test set being independent of segmentation, not a fitted parameter or a renamed prediction. The self-citation to Al-Rfou et al. (2019) is used only as a baseline comparator for a smaller byte-level model and does not supply any load-bearing premise for the new result. The paper's caveat that lm1b is preprocessed and has rare words replaced by UNK is an honest limitation on generalization to truly raw text, but that is an evaluation-scope concern rather than circular reasoning. No derivation step reduces to its own inputs, and no fitted input is renamed as a prediction, so the appropriate circularity score is zero.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or fitted parameters. Its central claim rests on standard probability and information-theoretic identities and on domain assumptions about the lm1b corpus and the word-similarity probing method.

assumptions (4)
  • standard math The joint probability of a token sequence factorizes left-to-right as a product of conditional probabilities.
    Used in Eq. 1 to define the language modeling objective.
  • standard math The information content of the test text is independent of segmentation, so bits/word can be converted to bits/byte using the test set's byte-to-word ratio.
    Used in Eqs. 2-3 to compare byte-level and word-level results on lm1b.
  • domain assumption Sampling random 512-byte sequences that ignore sentence boundaries is a sufficient training setup for lm1b.
    Stated in Section 2.1; lm1b sentences are short (99.78% under 512 bytes), so the truncation is mild.
  • domain assumption The activation at an appended space position provides a valid word representation for similarity probing.
    Introduced in Section 5; the activation is assumed to summarize the preceding word because the model must predict the byte after the space.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bridging the Gap for Tokenizer-Free Language Models." pith.science (2026). https://pith.science/paper/HXMHXSWP

@misc{pith2026190810322,
  author       = {Pith},
  title        = {Pith review of: Bridging the Gap for Tokenizer-Free Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HXMHXSWP}},
  note         = {Machine review of arXiv:1908.10322}
}
read the original abstract

Purely character-based language models (LMs) have been lagging in quality on large scale datasets, and current state-of-the-art LMs rely on word tokenization. It has been assumed that injecting the prior knowledge of a tokenizer into the model is essential to achieving competitive results. In this paper, we show that contrary to this conventional wisdom, tokenizer-free LMs with sufficient capacity can achieve competitive performance on a large scale dataset. We train a vanilla transformer network with 40 self-attention layers on the One Billion Word (lm1b) benchmark and achieve a new state of the art for tokenizer-free LMs, pushing these models to be on par with their word-based counterparts.

Figures

Figures reproduced from arXiv: 1908.10322 by the authors.

Figure 1
Figure 1. Performance on word similarity tasks described in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Byte Latent Transformer: Patches Scale Better Than Tokens

    cs.CL 2024-12 conditional novelty 7.0 of 10

    A byte-level transformer that dynamically groups bytes into entropy-based patches matches token-based LLM performance at 8B scale and opens a new patch-size scaling axis for fixed inference cost.

  2. Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A hierarchical byte-to-word-to-byte transformer matches subword-tokenizer LLMs at 1B-7B scale while being more robust to input corruption and faster to adapt to new languages.

Reference graph

Works this paper leans on

29 extracted references · 7 canonical work pages · cited by 2 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pa s ca, and Aitor Soroa. 2009. http://dl.acm.org/citation.cfm?id=1620754.1620758 A study on similarity and relatedness using distributional and wordnet-based approaches . In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Associ...

    work page arXiv 2009

  4. [4]

    Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2019. https://arxiv.org/abs/1808.04444 Character-level language modeling with deeper self-attention . In Thirty-Third AAAI Conference on Artificial Intelligence

    arXiv 2019

  5. [5]

    Alexei Baevski and Michael Auli. 2018. https://arxiv.org/abs/1809.10853 Adaptive input representations for neural language modeling . arXiv preprint arXiv:1809.10853

    arXiv 2018

  6. [6]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. https://doi.org/10.1162/tacl_a_00051 Enriching word vectors with subword information . Transactions of the Association for Computational Linguistics, 5:135--146

    doi:10.1162/tacl_a_00051 2017

  7. [7]

    Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. http://dl.acm.org/citation.cfm?id=2390524.2390544 Distributional semantics in technicolor . In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1, ACL '12, pages 136--145, Stroudsburg, PA, USA. Association for Computational Linguistics

    work page arXiv 2012

  8. [8]

    Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005

    arXiv 2013

Show all 29 references
  1. [9]

    Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. https://arxiv.org/abs/1901.02860 Transformer-xl: Attentive language models beyond a fixed-length context . arXiv preprint arXiv:1901.02860

    2019 arXiv

  2. [10]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. http://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . CoRR, abs/1810.04805

    2018 arXiv

  3. [11]

    Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. https://doi.org/10.1145/503104.503110 Placing search in context: The concept revisited . ACM Trans. Inf. Syst., 20(1):116--131

    2002

  4. [12]

    Philip Gage. 1994. http://dl.acm.org/citation.cfm?id=177910.177914 A new algorithm for data compression . C Users J., 12(2):23--38

    1994

  5. [13]

    Felix Hill, Roi Reichart, and Anna Korhonen. 2014. http://arxiv.org/abs/1408.3456 Simlex-999: Evaluating semantic models with (genuine) similarity estimation . CoRR, abs/1408.3456

    2014 arXiv

  6. [14]

    Jeremy Howard and Sebastian Ruder. 2018. https://arxiv.org/abs/1801.06146 Universal language model fine-tuning for text classification . arXiv preprint arXiv:1801.06146

    2018 arXiv

  7. [15]

    Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. http://dl.acm.org/citation.cfm?id=3016100.3016285 Character-aware neural language models . In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, pages 2741--2749. AAAI Press

    2016

  8. [16]

    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  9. [17]

    Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2016. http://arxiv.org/abs/1610.03017 Fully character-level neural machine translation without explicit segmentation . CoRR, abs/1610.03017

    2016 arXiv

  10. [18]

    Thang Luong, Richard Socher, and Christopher Manning. 2013. https://www.aclweb.org/anthology/W13-3512 Better word representations with recursive neural networks for morphology . In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 104-...

    2013

  11. [19]

    Rovert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. https://catalog.ldc.upenn.edu/LDC2011T07 English gigaword fifth edition . Philadelphia. Linguistic Data Consortium

    2011

  12. [20]

    Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://arxiv.org/abs/1802.05365 Deep contextualized word representations . arXiv preprint arXiv:1802.05365

    2018 arXiv

  13. [21]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf Improving language understanding by generative pre-training

    2018

  14. [22]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf Language models are unsupervised multitask learners . OpenAI Blog, 1:8

    2019

  15. [23]

    Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. 2011. https://doi.org/10.1145/1963405.1963455 A word at a time: Computing word relatedness using temporal semantic analysis . In Proceedings of the 20th International Conference on World Wide Web, WWW ...

    2011

  16. [24]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  17. [25]

    Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. 2018. https://arxiv.org/abs/1811.02084 Mesh-tensorflow: Deep learning for supercomputers . In Advances in Neural Info...

    2018 arXiv

  18. [26]

    Noam Shazeer, Ryan Doherty, Colin Evans, and Chris Waterson. 2016. http://arxiv.org/abs/1602.02215 Swivel: Improving embeddings by noticing what's missing . CoRR, abs/1602.02215

    2016 arXiv

  19. [27]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. https://arxiv.org/abs/1701.06538 Outrageously large neural networks: The sparsely-gated mixture-of-experts layer . arXiv preprint arXiv:1701.06538

    2017 arXiv

  20. [28]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://arxiv.org/abs/1706.03762 Attention is all you need . In Advances in neural information processing systems, pages 5998--6008

    2017 arXiv

  21. [29]

    Torsten Zesch, Christof M\" u ller, and Iryna Gurevych. 2008. http://dl.acm.org/citation.cfm?id=1620163.1620206 Using wiktionary for computing semantic relatedness . In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 2, AAAI'08, pages 861--866. ...

    2008

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.