REVIEW 20 cited by
Byte Latent Transformer: Patches Scale Better Than Tokens
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Byte Latent Transformer: Patches Scale Better Than Tokens
read the original abstract
We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first FLOP controlled scaling study of byte-level models up to 8B parameters and 4T training bytes. Our results demonstrate the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. Overall, for fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size.
Forward citations
Cited by 20 Pith papers
-
SUNTA: Hierarchical Video Prediction with Surprise-based Chunking
SUNTA uses surprise-driven chunk boundaries and decoupled training in hierarchical state-space models to sustain accurate video predictions over 250 timesteps where baselines fail after 10.
-
User as Engram: Internalizing Per-User Memory as Local Parametric Edits
User facts are internalized as surgical local edits to a hash-keyed Engram memory table with reasoning skill held in a shared adapter, claimed to match LoRA recall, improve indirect reasoning 5.6x on average, and comp...
-
MimeLens: Position-Agnostic Content-Type Detection for Binary Fragments
MimeLens uses position-agnostic BERT encoders pretrained on random-offset binary windows to output one of 125 libmagic MIME labels, beating Magika on full files and enabling accurate classification on mid-file fragments.
-
Large Byte Model: Teaching Language Models About Compiled Code
Presents a byte-native LLM with bespoke tokenizer achieving 69-98% accuracy on malware family and architecture classification from raw bytes.
-
Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
Kronecker Embeddings replace learned embedding tables with a deterministic byte-level character-position factorization and single projection, reducing parameters over 90% with reported gains in loss and robustness on ...
-
Bin Latent Transformer (BiLT): A shift-invariant autoencoder for calibration-free spectral unmixing of turbid media
The BiLT autoencoder recovers absorption and scattering spectra from integrating sphere data with high accuracy while remaining robust to wavelength shifts up to 10 bands and generalizing to different instrument line ...
-
Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models
Scratchpad Patching decouples compute from patch size in byte-level language models by inserting entropy-triggered scratchpads to update patch context dynamically.
-
DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunking
DC-DiT learns dynamic chunking to allocate fewer tokens to smooth or noisy regions and more to detailed or late-stage areas, cutting inference FLOPs up to 36.8% while improving FID up to 37.8% on class-conditional Ima...
-
FLEXITOKENS: Flexible Tokenization for Evolving Language Models
FLEXITOKENS replaces rigid subword tokenizers and fixed-compression auxiliary losses with a simplified boundary-prediction objective in byte-level models, yielding lower over-fragmentation and up to 10-point gains on ...
-
Sampling from Your Language Model One Byte at a Time
An inference-time technique turns BPE-based LMs into byte- or character-level models, solving the prompt boundary problem while unifying vocabularies across different tokenizers.
-
Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
On chemistry SMILES with a fixed 165-token base, BPE and Unigram-LM produce near-disjoint vocabularies (Jaccard ≤0.161) and Unigram-LM emits 29–41% more tokens across 22 matched conditions.
-
SUNTA: Hierarchical Video Prediction with Surprise-based Chunking
Decoupled surprise-based hierarchical state-space models with top-down inconsistency chunking sustain accurate 250-step open-loop video prediction where prior HSSMs collapse within 10 steps.
-
Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet
IPA-based subword tokenizers trained across 24 languages improve tokenization quality and generalization to unseen languages compared to standard text tokenizers, especially for non-Latin scripts.
-
Towards Understanding Self-Pretraining for Sequence Classification
Self-pretraining improves Transformer sequence classification by enabling learning of proximity-biased attention from positional encodings that label supervision alone cannot easily acquire from random starts.
-
Protocol-Aware Tokenization and Architecture Co-Design for Wireless Packet Foundation Models
Protocol-aware tokenization is the primary performance driver for wireless packet foundation models, delivering 32-point accuracy gains while architecture changes yield only 2 points in a controlled 2x2 comparison.
-
Lost in State Space: Probing Frozen Mamba Representations
Frozen Mamba patch-boundary readouts do not outperform mean pooling for sentence representations on SST-2, CoLA, MRPC, STS-B, and IMDb due to anisotropy (cosine similarity ~0.9999) and representational collapse (MCC=0...
-
EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction
EntMTP is a training-free entropy-guided scheduler for multi-token prediction that dynamically selects from task-specific Pareto-optimal trees to accelerate LLM inference by up to 1.36x on benchmarks without quality loss.
-
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
A 355M-parameter byte-level LM on 80B multilingual tokens exhibits UTF-8 validity converging after 4.2B tokens versus 2.1B for perplexity, with higher validity on rare characters than common ones.
-
Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
This work systematically compares inter-layer and intra-layer hybridization strategies for combining self-attention and Mamba-style state space models, evaluating them on language modeling, downstream tasks, long-cont...
-
Cybersecurity is the True Frontier for Generative AI Success or Failure
Cybersecurity's scale, adversaries, labeling issues, and operational demands make it the superior test-case for general AI progress over NLP or computer vision.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.