Pith. sign in

REVIEW 2 major objections 5 minor 48 references

ByteSpan: Information-Driven Subword Tokenisation

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Grouping predictable bytes into subwords beats BPE on morphological alignment.

desk verdict A solid, honest intrinsic-evaluation paper for an information-driven static tokeniser; the coverage caveat on the headline morphological claim is real but the authors already began addressing it, and the missing piece is downstream LM validation. read the letter →

arxiv 2506.18639 v1 pith:B6QMVEAV submitted 2025-06-23 cs.CL

Zébulon Goriely , Suchir Salhan , Pietro Lesci , Julius Cheng , Paula Buttery This is my paper
classification cs.CL
keywords subwordtokenisationbyte-levellanguagemodelinformation-drivensegmentationmorphologicalalignmentByte-PairEncodingsurprisalRényiefficiencytokeniserevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the information signal used by dynamic byte-patching models—per-byte uncertainty from an autoregressive byte-level language model—can be turned into a fixed subword vocabulary. It presents ByteSpan, which segments training text wherever a byte's entropy or surprisal spikes, then keeps the most frequent predictable sequences as tokens. On intrinsic evaluation for English, ByteSpan reports higher morphological alignment scores than Byte-Pair Encoding across vocabulary sizes 16k, 32k, and 64k, while maintaining comparable compression and better Rényi efficiency. If the intrinsic results reflect real gains, this would give language modelling the efficiency of fixed vocabularies without the purely frequency-driven segmentation of BPE.

What carries the argument

The central mechanism is the per-byte information signal from an external byte-level LM, converted into segment boundaries by three constraints: the global constraint keeps bytes below a fixed information threshold $\theta_g$; the monotonic constraint groups bytes whose information decreases from the previous byte ($H(b_t)-H(b_{t-1})<0$); the combined constraint applies either rule. These constraints define which contiguous byte sequences are candidate subwords, and the vocabulary is fixed by ranking candidates by frequency, by raising the threshold until the desired size is reached, or by using the candidates to seed BPE. The information signal and constraints are the load-bearing part: they replace BPE's frequency-based merge ordering with an information-based segmentation, which is what produces the reported morphological alignment.

What would settle it

Pre-train identical language models on the same corpus with ByteSpan and BPE tokenisers of the same vocabulary size, then compare perplexity on held-out text and scores on a grammaticality benchmark such as BLiMP; if BPE matches or outperforms ByteSpan on these downstream measures, the claimed morphological alignment advantage does not translate into modelling gains.

Watch

Extended reading notes

Core claim

ByteSpan is a static tokeniser that learns a vocabulary in two phases: first an external byte-level language model computes per-byte entropy (or surprisal) over a training corpus; then one of three constraints—a global threshold on information, a monotonic decrease from one byte to the next, or a combination—groups contiguous bytes into candidate subwords. The candidates become a fixed vocabulary through one of three learning methods: keeping the most frequent spans, raising the threshold incrementally until the target size is reached, or seeding BPE with a fraction of ByteSpan units and letting BPE learn the rest. The paper reports that, for English, ByteSpan achieves higher morphological alignment than BPE and BPE with WordPiece-style inference for all tested vocabulary sizes, with the global constraint scoring highest on alignment but lowest on coverage; the combined constraint gives a better balance. On a 25-language corpus, ByteSpan matches BPE's compression and Rényi efficiency, and a language-balanced variant improves fertility for languages with non-Latin orthographies.

Load-bearing premise

The results stand or fall on whether intrinsic tokeniser metrics—morphological alignment, cognitive plausibility, Rényi efficiency, and fertility—are a reliable proxy for how well the tokeniser will perform when actually pre-training a language model, which the paper does not test.

Editorial extensions

If this is right

  • Information-driven static tokenisers can achieve higher morphological alignment than frequency-based BPE without sacrificing compression.
  • The seeding method, which starts with ByteSpan units and lets BPE learn the rest of the vocabulary, yields token length distributions similar to BPE while retaining higher morphological alignment.
  • The same byte-level information signal works across 25 languages, producing similar Rényi efficiency and fertility to BPE.
  • Language-balanced vocabulary selection for the frequency method improves fertility for non-Latin orthographies without harming the average across languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If intrinsic morphological alignment translates to downstream gains, tokenisers could become a lightweight way to inject linguistic structure into language models without changing their architecture.
  • The information signal need not be entropy or surprisal from a byte-level LM; any scalar that marks boundaries (for instance, the probability of a whitespace token) could seed a fixed vocabulary, opening up a family of information-driven tokenisers.
  • The language-balanced variant suggests information-driven tokenisation could mitigate the compression bias against non-Latin scripts, though this needs verification through downstream evaluation.
  • A testable extension is to scale the byte-level LM and see whether the morphological alignment advantage grows, shrinks, or plateaus; the paper currently uses only one small LM.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces ByteSpan, a static subword tokeniser that uses an external byte-level language model to estimate per-byte surprisal or entropy and groups contiguous bytes into subwords under three constraints: a global threshold, a monotonic decrease, and a combination of the two. Vocabularies are built via frequency cutoff, incremental threshold raising, or seeding BPE. On English intrinsic evaluation (Uzan et al., 2024), the paper reports higher morphological alignment and Rényi efficiency than BPE/BPE-WP at 16k/32k/64k vocabulary sizes, with lower cognitive plausibility; a 25-language multilingual experiment reports fertility and Rényi efficiency comparable to BPE. The authors themselves identify a coverage artifact in the morphological benchmark and report morphological coverage in Fig. 3, concluding that the global-constraint tokenizers are not comparable but the remaining ones still outperform BPE-WP.

Significance. If the central empirical claim survives closer scrutiny, ByteSpan is a useful contribution: it demonstrates that an information-driven static tokeniser can produce morphologically aligned fixed vocabularies without sacrificing compression, connecting subword tokenisation to computational word segmentation. The paper is honest and unusually thorough in its self-assessment: it reports the coverage-dependence of the morphological benchmark, releases code and models on GitHub/HuggingFace, and explicitly lists the lack of downstream LM evaluation as a limitation. The main risk is that the morphological alignment advantage is evaluated on vocabulary-dependent word subsets; this is a fixable concern, but it currently prevents the headline claim from being fully established.

major comments (2)
  1. [§4, Fig. 3] The claim that ByteSpan 'achieves higher morphological alignment scores than BPE' relies on the Uzan et al. (2024) benchmark, which—as the authors note—skips any word unless all gold morphemes are present in the tokenizer's vocabulary. The reported macro-F1 is therefore computed over a word subset that depends on the tokenizer. The authors correctly discard the global-constraint tokenizers because their coverage is much lower, but they do not report exact coverage values, and 'similar coverage' does not imply that the scored subsets are comparable: two tokenizers can cover the same fraction of words but different words, and the included words can differ in segmentation difficulty. Please recompute morphological alignment on the intersection of words covered by all tokenizers being compared (or report coverage-stratified scores and exact coverage numbers). Without this, the higher F1 may reflect an easier covered subset rather than better morphological alignment.
  2. [§2.2, Algorithm 1] Algorithm 1 initializes i←2 and never includes b_1 in any extracted segment, so the tokenized sequence omits the first byte of every input sequence. If the intended behavior is to start at the first byte, the initialization should be i←1; if the initial byte is deliberately skipped, the method's description and the example in Fig. 1 need to be reconciled. Since this pseudocode defines the monotonic constraint, which is one of the three core constraints and is used in several reported configurations, the correctness of the implementation must be verified and the pseudocode corrected.
minor comments (5)
  1. [§4, Fig. 3] Please add exact morphological coverage values to Fig. 3 or to a table; the text says 'similar coverage' but no numbers are given, which is important for judging the comparability claim.
  2. [§3, Table 1] Only single runs are reported; adding seeds and, for small differences such as the 64k Rényi efficiency values (0.409 vs. 0.399), confidence intervals would help interpret the comparisons.
  3. [§5] The hyperparameters p=50% for seeding and θg at the 30th percentile for the combined constraint are acknowledged as unexplored; a small sensitivity analysis (e.g., p∈{25%, 50%, 75%}) would strengthen the combined-constraint and seeding results.
  4. [§2.2, footnote 1] The statement that bytes and characters are used interchangeably is imprecise for UTF-8; byte spans and character spans diverge for non-Latin scripts, which is directly relevant to the multilingual discussion in §4.
  5. [Throughout] Typographical issues such as 'intopatches' in the abstract and missing spacing in 'ByteSpantokenisers' should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ByteSpan's construction and its evaluated claims are independent, with only a disclosed evaluation-coverage caveat that is not a circular derivation.

full rationale

The paper's derivation chain is self-contained against external benchmarks. ByteSpan constructs a fixed subword vocabulary from externally computed byte-level surprisal/entropy statistics using global, monotonic, or combined constraints, with vocabulary-size targets reached by frequency, incremental-threshold, or BPE-seeding methods. None of these construction steps is defined in terms of the evaluation metrics: morphological alignment, cognitive plausibility, Rényi efficiency, and fertility are all computed from external resources and benchmarks (Uzan et al. 2024; LADEC, MorphoLex, MorphyNet, DagoBert, etc.), and no parameter is fitted to those scores. The paper's own self-citations (Goriely et al. 2025, Goriely & Buttery 2025) are motivational background from word-segmentation modeling, not load-bearing premises for the claimed tokenizer advantage. The most substantive validity concern, that the Uzan et al. morphological-alignment benchmark skips words unless all gold segments are in the vocabulary, is explicitly disclosed by the authors in Section 4, and they respond with a morphological-coverage analysis (Fig. 3) showing similar coverage for the non-global ByteSpan variants. That is an evaluation comparability caveat, not a reduction of the claimed result to its own inputs; the morphological alignment score is not definitionally equal to any fitted quantity. The Limitations section's caveat that downstream LM evaluation would be ideal is an external-validity limitation, not circularity. Therefore no circular step meets the evidentiary bar of this analysis.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central method depends on a trained byte-level LM as an information source, but that model is a reusable training component rather than a free parameter of the tokenizer itself. The genuine free parameters are the thresholds and seeding proportion, all set once and not tuned to the evaluation metrics.

free parameters (3)
  • Global threshold theta_g for combined constraint = 30th percentile of per-byte information
    Set ad hoc in Section 3; authors state this hyperparameter was not thoroughly explored.
  • Seeding proportion p = 50%
    Set in Section 3; controls the split between ByteSpan and BPE in the seeding method.
  • Minimum frequency threshold theta_f = 20
    Used in the incremental method to filter rare subwords (Section 3).
assumptions (3)
  • domain assumption Per-byte entropy or surprisal from a byte-level LM is a reliable signal of morphological boundaries.
    Core premise of the method, motivated by the word segmentation literature in Sections 2.1 and 2.2.
  • domain assumption The intrinsic evaluation benchmarks are valid proxies for tokeniser quality.
    Required for the reported comparisons; authors flag that downstream validation is still needed in the Limitations section.
  • domain assumption BPE pre-tokenisation boundaries should be respected by the tokeniser.
    A design choice that constrains ByteSpan spans, described in Section 2.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ByteSpan: Information-Driven Subword Tokenisation." pith.science (2026). https://pith.science/paper/B6QMVEAV

@misc{pith2026250618639,
  author       = {Pith},
  title        = {Pith review of: ByteSpan: Information-Driven Subword Tokenisation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B6QMVEAV}},
  note         = {Machine review of arXiv:2506.18639}
}
read the original abstract

Recent dynamic tokenisation methods operate directly on bytes and pool their latent representations into patches. This bears similarities to computational models of word segmentation that determine lexical boundaries using spikes in an autoregressive model's prediction error. Inspired by this connection, we explore whether grouping predictable bytes - rather than pooling their representations - can yield a useful fixed subword vocabulary. We propose a new information-driven subword tokeniser, ByteSpan, that uses an external byte-level LM during training to identify contiguous predictable byte sequences and group them into subwords. Experiments show that ByteSpan yields efficient vocabularies with higher morphological alignment scores than BPE for English. Multilingual experiments show similar compression and R\'enyi efficiency for 25 languages.

Figures

Figures reproduced from arXiv: 2506.18639 by the authors.

Figure 1
Figure 1. Information-Driven Subword Creation. Per-byte surprisal of “molecules are unstable and periodically evaporate” from a byte-level LM. ByteSpan groups contiguous bytes using one of three constraints; the global constraint uses a fixed threshold, the monotonic constraint groups bytes with decreasing information and the combined constraint groups bytes that meet either constraint. Grey vertical lines indicate pre-tokeni… view at source ↗
Figure 2
Figure 2. Fertility and rényi efficiency for each language in our Common-Corpus evaluation subset, comparing multilingual BPE to our multilingual ByteSpan tokenisers using surprisal with a vocabulary size of 128k. be successfully extracted using the surprisal from a byte￾level model but that the number of splits may not correlate well with human performance in a lexical decision task. Our results for BPE and BPE-WP mirror tho… view at source ↗
Figure 4
Figure 4. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Increase in fertility for each language when balancing added tokens across languages when training our multilingual ByteSpan tokenisers with the frequency method. The tokenisers use surprisal as the byte-level measure and the vocabulary size is 128k. A decrease in fert…
Figure 6
Figure 6. Figure 6: Fertility and rényi efficiency for each language in our Common-Corpus evaluation subset, comparing multilingual BPE to our multilingual ByteSpan tokenisers using entropy with a vocabulary size of 128k. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 23 canonical work pages

  1. [1]

    Exploring BERT 's vocabulary, 2019

    \'A cs, J. Exploring BERT 's vocabulary, 2019. URL https://juditacs.github.io/2019/02/19/bert-tokenization-stats.html. Accessed: May 01, 2025

    work page 2019

  2. [2]

    M orphy N et: A large multilingual database of derivational and inflectional morphology

    Batsuren, K., Bella, G., and Giunchiglia, F. M orphy N et: A large multilingual database of derivational and inflectional morphology. In Nicolai, G., Gorman, K., and Cotterell, R. (eds.), Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pp.\ 39--48, Online, August 2021. Association for Computat...

    doi:10.18653/v1/2021.sigmorphon-1.5 2021

  3. [3]

    G., Ryskina, M., Mielke, S., Budianskaya, E., El-Khaissi, C., Pimentel, T., Gasser, M., Lane, W

    Batsuren, K., Goldman, O., Khalifa, S., Habash, N., Kiera \'s , W., Bella, G., Leonard, B., Nicolai, G., Gorman, K., Ate, Y. G., Ryskina, M., Mielke, S., Budianskaya, E., El-Khaissi, C., Pimentel, T., Gasser, M., Lane, W. A., Raj, M., Coler, M., Samame, J. R. M., Camaiteri, D. S., Rojas, E. Z., L \'o pez Francis, D., Oncevay, A., L \'o pez Bautista, J., V...

    work page 2022

  4. [4]

    and Pinter, Y

    Beinborn, L. and Pinter, Y. Analyzing cognitive plausibility of subword tokenization. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 4478--4486, Singapore, December 2023. Association for Computational Linguistics. doi:10.18653/v1/2023.emnlp-main.272. URL https://a...

    doi:10.18653/v1/2023.emnlp-main.272 2023

  5. [5]

    Brent, M. R. Efficient, probabilistically sound algorithm for segmentation and word discovery . Machine Learning, 34 0 (1): 0 71--105, 1999. ISSN 08856125. doi:10.1023/a:1007541817488. URL https://link.springer.com/article/10.1023/A:1007541817488

    work page doi:10.1023/a:1007541817488 1999

  6. [6]

    Tokenization falling short: O n subword robustness in large language models

    Chai, Y., Fang, Y., Peng, Q., and Li, X. Tokenization falling short: O n subword robustness in large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 1582--1599, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi:10.18653/v1/2...

    doi:10.18653/v1/2024.findings-emnlp.86 2024

  7. [7]

    H., Allen, J., and Seidenberg, M

    Christiansen, M. H., Allen, J., and Seidenberg, M. S. Learning to segment speech using multiple cues: A connectionist model. Language and Cognitive Processes, 13 0 (2-3): 0 221--268, 1998. URL https://www.tandfonline.com/doi/abs/10.1080/016909698386528

    work page doi:10.1080/016909698386528 1998

  8. [8]

    and Nerbonne, J

    C \"o ltekin, C . and Nerbonne, J. An explicit statistical model of learning lexical segmentation using multiple cues. In Lenci, A., Padr \'o , M., Poibeau, T., and Villavicencio, A. (eds.), Proceedings of the 5th Workshop on Cognitive Aspects of Computational Language Learning ( C og ACLL ) , pp.\ 19--28, Gothenburg, Sweden, April 2014. Association for C...

    work page doi:10.3115/v1/w14-0505 2014

Show all 48 references
  1. [9]

    Dai, Z., Lai, G., Yang, Y., and Le, Q. V. Funnel-transformer: F iltering out sequential redundancy for efficient language processing. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Nips '20, Red Hook, NY, USA, 2020. Curran Associa...

    2020

  2. [10]

    Elman, J. L. Finding structure in time. Cognitive science, 14 0 (2): 0 179--211, 1990. URL https://www.sciencedirect.com/science/article/pii/036402139090002E

    1990

  3. [11]

    L., Spalding, T

    Gagn \'e , C. L., Spalding, T. L., and Schmidtke, D. LADEC: the L arge D atabase of E nglish C ompounds. Behavior R esearch M ethods , 51: 0 2152--2179, 2019. URL https://link.springer.com/article/10.3758/s13428-019-01282-6

    2019 doi

  4. [12]

    Splintering nonconcatenative languages for better tokenization, 2025

    Gazit, B., Shmidman, S., Shmidman, A., and Pinter, Y. Splintering nonconcatenative languages for better tokenization, 2025. URL https://arxiv.org/abs/2503.14433

    2025 arXiv

  5. [13]

    and Buttery, P

    Goriely, Z. and Buttery, P. Baby LM 's first words: W ord segmentation as a phonological probing task, 2025. URL https://arxiv.org/abs/2504.03338

    2025 arXiv

  6. [14]

    Word segmentation from transcriptions of child-directed speech using lexical and sub-lexical cues

    Goriely, Z., Caines, A., and Buttery, P. Word segmentation from transcriptions of child-directed speech using lexical and sub-lexical cues. Journal of Child Language, 52 0 (1): 0 1–41, 2025. doi:10.1017/s0305000923000491

    2025 doi

  7. [15]

    Improving tokenisation by alternative treatment of spaces

    Gow-Smith, E., Tayyar Madabushi, H., Scarton, C., and Villavicencio, A. Improving tokenisation by alternative treatment of spaces. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ ...

    2022 doi

  8. [16]

    Harris, Z. S. From phoneme to morpheme. Springer, 1955

    1955

  9. [17]

    D ago BERT : G enerating derivational morphology with a pretrained language model

    Hofmann, V., Pierrehumbert, J., and Sch \"u tze, H. D ago BERT : G enerating derivational morphology with a pretrained language model. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (E...

    2020 doi

  10. [18]

    Mini CPM : U nveiling the potential of small language models with scalable training strategies

    Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et al. Mini CPM : U nveiling the potential of small language models with scalable training strategies. arXiv preprint 2404.06395, 2024. URL https://arxiv.org/abs/2404.06395

    2024 arXiv

  11. [19]

    Jacobs, C. L. and Pinter, Y. Lost in space marking. arXiv preprint 2208.01561, 2022. URL https://arxiv.org/abs/2208.01561

    2022 arXiv

  12. [20]

    The M ini P ile challenge for data-efficient language models

    Kaddour, J. The M ini P ile challenge for data-efficient language models. arXiv preprint 2304.08442, 2023. URL https://arxiv.org/abs/2304.08442

    2023 arXiv

  13. [21]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, 2015. URL http://arxiv.org/abs/1412.6980

    2015 arXiv

  14. [22]

    Subword regularization: Improving neural network translation models with multiple subword candidates

    Kudo, T. Subword regularization: Improving neural network translation models with multiple subword candidates. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 66--75, Me...

    2018 doi

  15. [23]

    The Common Corpus : A new massively multilingual dataset, September 2021

    Langlais, P.-C. The Common Corpus : A new massively multilingual dataset, September 2021. URL https://huggingface.co/blog/Pclanglais/common-corpus. Accessed: May 01, 2025

    2021

  16. [24]

    MYTE : Morphology-driven byte encoding for better and fairer multilingual language modeling

    Limisiewicz, T., Blevins, T., Gonen, H., Ahia, O., and Zettlemoyer, L. MYTE : Morphology-driven byte encoding for better and fairer multilingual language modeling. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for...

    2024 doi

  17. [25]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  18. [26]

    The acquisition of morphophonology

    MacWhinney, B. The acquisition of morphophonology. Monographs of the society for research in child development, pp.\ 1--123, 1978. URL https://psyling.talkbank.org/years/1978/monograph.pdf

    1978

  19. [27]

    C ompound P iece: E valuating and improving decompounding performance of language models

    Minixhofer, B., Pfeiffer, J., and Vuli \'c , I. C ompound P iece: E valuating and improving decompounding performance of language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\...

    2023 doi

  20. [28]

    Hierarchical transformers are more efficient language models

    Nawrot, P., Tworkowski, S., Tyrolski, M., Kaiser, L., Wu, Y., Szegedy, C., and Michalewski, H. Hierarchical transformers are more efficient language models. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V. (eds.), Findings of the Association for Computational Linguisti...

    2022 doi

  21. [29]

    Nawrot, P., Chorowski, J., Lancucki, A., and Ponti, E. M. Efficient transformers with dynamic token pooling. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),...

    2023 doi

  22. [30]

    Byte latent transformer: P atches scale better than tokens

    Pagnoni, A., Pasunuru, R., Rodriguez, P., Nguyen, J., Muller, B., Li, M., Zhou, C., Yu, L., Weston, J., Zettlemoyer, L., et al. Byte latent transformer: P atches scale better than tokens. arXiv preprint 2412.09871, 2024. URL https://arxiv.org/abs/2412.09871

    2024 arXiv

  23. [31]

    PyTorch : A n imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch : A n impe...

    2019

  24. [32]

    B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L

    Penedo, G., Kydl \' c ek, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The FineWeb datasets: D ecanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchm...

    2024

  25. [33]

    L., and Eisenstein, J

    Pinter, Y., Jacobs, C. L., and Eisenstein, J. Will it unblend? In Ettinger, A., Pavlick, E., and Prickett, B. (eds.), Proceedings of the Society for Computation in Linguistics 2021, pp.\ 474--476, Online, February 2021. Association for Computational Linguistics. URL https://ac...

    2021

  26. [34]

    How Good is Your Tokenizer ? On the Monolingual Performance of Multilingual Language Models

    Rust, P., Pfeiffer, J., Vuli\' c , I., Ruder, S., and Gurevych, I. How Good is Your Tokenizer ? On the Monolingual Performance of Multilingual Language Models . In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for ...

    2021 doi

  27. [35]

    R., Aslin, R

    Saffran, J. R., Aslin, R. N., and Newport, E. L. Statistical learning by 8-month-old infants . Science, 274 0 (5294): 0 1926--1928, December 1996. ISSN 00368075. doi:10.1126/science.274.5294.1926. URL http://science.sciencemag.org/

    1926

  28. [36]

    H., Mailhot, H., Deacon, S

    S \'a nchez-Guti \'e rrez, C. H., Mailhot, H., Deacon, S. H., and Wilson, M. A. Morpho L ex: A D erivational M orphological D atabase for 70,000 E nglish words. Behavior R esearch M ethods , 50: 0 1568--1580, 2018. URL https://link.springer.com/article/10.3758/s13428-017-0981-8

    2018 doi

  29. [37]

    and Nakajima, K

    Schuster, M. and Nakajima, K. J apanese and K orean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 5149--5152, 2012. doi:10.1109/icassp.2012.6289079

    2012

  30. [38]

    Neural machine translation of rare words with subword units

    Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Erk, K. and Smith, N. A. (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1715--1725, Berlin, Ge...

    2016 doi

  31. [39]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N....

    2023 arXiv

  32. [40]

    W., Tanner, C., and Pinter, Y

    Uzan, O., Schmidt, C. W., Tanner, C., and Pinter, Y. Greed is all you need: An evaluation of tokenizer inference methods. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Shor...

    2024 doi

  33. [41]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Informatio...

    2017

  34. [42]

    A statistical model for word discovery in transcribed speech

    Venkataraman, A. A statistical model for word discovery in transcribed speech . Computational Linguistics, 27 0 (3): 0 350--372, March 2001. ISSN 08912017. doi:10.1162/089120101317066113. URL https://www.mitpressjournals.org/doix/abs/10.1162/089120101317066113

    2001 doi

  35. [43]

    Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., and Bowman, S. R. BL i MP : The benchmark of linguistic minimal pairs for E nglish. Transactions of the Association for Computational Linguistics, 8: 0 377--392, 2020. doi:10.1162/tacl_a_00321. URL https...

    2020 doi

  36. [44]

    Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective

    Wen, K., Li, Z., Wang, J., Hall, D., Liang, P., and Ma, T. Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective. arXiv preprint 2410.05192, 2024. URL https://arxiv.org/abs/2410.05192

    2024 arXiv

  37. [45]

    Transformers: S tate-of-the-art natural language processing

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen , P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: S ta...

    2020

  38. [46]

    MEGABYTE : P redicting million-byte sequences with multiscale transformers

    Yu, L., Simig, D., Flaherty, C., Aghajanyan, A., Zettlemoyer, L., and Lewis, M. MEGABYTE : P redicting million-byte sequences with multiscale transformers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=JTmO2V9Xpz

    2023

  39. [47]

    Scaling vision transformers

    Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12104--12113, 2022. URL https://openaccess.thecvf.com/content/CVPR2022/html/Zhai\

    2022

  40. [48]

    Tokenization and the noiseless channel

    Zouhar, V., Meister, C., Gastaldi, J., Du, L., Sachan, M., and Cotterell, R. Tokenization and the noiseless channel. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    2023 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.