REVIEW 11 cited by
Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Tokenization, the division of input text into input tokens, is an often overlooked aspect of the large language model (LLM) pipeline and could be the source of useful or harmful inductive biases. Historically, LLMs have relied on byte pair encoding, without care to specific input domains. With the increased use of LLMs for reasoning, various number-specific tokenization schemes have been adopted, with popular models like LLaMa and PaLM opting for single-digit tokenization while GPT-3.5 and GPT-4 have separate tokens for each 1-, 2-, and 3-digit numbers. In this work, we study the effect this choice has on numerical reasoning through the use of arithmetic tasks. We consider left-to-right and right-to-left tokenization for GPT-3.5 and -4, finding that right-to-left tokenization (enforced by comma separating numbers at inference time) leads to largely improved performance. Furthermore, we find that model errors when using standard left-to-right tokenization follow stereotyped error patterns, suggesting that model computations are systematic rather than approximate. We show that the model is able to convert between tokenizations easily, thus allowing chain-of-thought-inspired approaches to recover performance on left-to-right tokenized inputs. We also find the gap between tokenization directions decreases when models are scaled, possibly indicating that larger models are better able to override this tokenization-dependent inductive bias. In summary, our work performs the first study of how number tokenization choices lead to differences in model performance on arithmetic tasks, accompanied by a thorough analysis of error patterns. We hope this work inspires practitioners to more carefully ablate number tokenization-related choices when working towards general models of numerical reasoning.
Forward citations
Cited by 11 Pith papers
-
Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines
LongGuide automatically learns task-specific quality and length guidelines from small training sets, significantly improving LLM long-form generation.
-
Variable-length Gray codes for the Natural Numbers
Every finite digit string gets exactly one number, and consecutive numbers differ by a single edit — a complete, locally stable variable-length Gray code.
-
The Effect of Scripts and Formats on LLM Numeracy
LLM arithmetic accuracy falls sharply when numerals leave the familiar Hindu–Arabic format, and few-shot prompting with examples narrows most of that gap.
-
Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
Falcon-H1 reports competitive benchmark scores for a 0.5B to 34B family of parallel hybrid attention/Mamba-2 models, claiming 2x to 4x parameter efficiency versus dense transformers.
-
The Decrypto Benchmark for Multi-Agent Reasoning and Theory of Mind
A new interactive language-game benchmark shows LLMs lag behind simple word-embedding baselines and that newer reasoning models regress on theory-of-mind tasks.
-
Incorporating Domain Knowledge into Materials Tokenization
A domain-knowledge-weighted tokenizer, MATTER, preserves material concepts and improves materials NLP performance by 4% on generation and 2% on classification tasks.
-
BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization
A Unicode script and category based character encoding with constrained merging achieves compression competitive with byte-level BPE while removing the byte-premium penalty for non-Latin scripts.
-
multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data
multivariateGPT extends next-token prediction to jointly predict the class and continuous value of mixed categorical and numeric time series, with Gaussian uncertainty, and outperforms discrete-token baselines on clin...
-
BALM-TSF: Balanced Multimodal Alignment for LLM-Based Time Series Forecasting
BALM-TSF combines a statistical-prompt text branch with a patch-based time series branch, using scaling plus contrastive alignment to balance the two modalities, improving long-term and few-shot forecasting on five of...
-
Improving Large Language Models with Concept-Aware Fine-Tuning
Adding lightweight multi-token auxiliary heads with a weighted future-token loss improves supervised fine-tuning of Llama-3-8B-Instruct across five diverse tasks.
-
From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference
A 2.7B German-first LLM trained cheaply on public data with language-specific quality filtering matches larger 7B models on German reasoning benchmarks and runs on-device.
Discussion (0). Continue with ORCID to comment.