REVIEW 8 cited by
Investigating the Limitations of Transformers with Simple Arithmetic Tasks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The ability to perform arithmetic tasks is a remarkable trait of human intelligence and might form a critical component of more complex reasoning tasks. In this work, we investigate if the surface form of a number has any influence on how sequence-to-sequence language models learn simple arithmetic tasks such as addition and subtraction across a wide range of values. We find that how a number is represented in its surface form has a strong influence on the model's accuracy. In particular, the model fails to learn addition of five-digit numbers when using subwords (e.g., "32"), and it struggles to learn with character-level representations (e.g., "3 2"). By introducing position tokens (e.g., "3 10e1 2"), the model learns to accurately add and subtract numbers up to 60 digits. We conclude that modern pretrained language models can easily learn arithmetic from very few examples, as long as we use the proper surface representation. This result bolsters evidence that subword tokenizers and positional encodings are components in current transformer designs that might need improvement. Moreover, we show that regardless of the number of parameters and training examples, models cannot learn addition rules that are independent of the length of the numbers seen during training. Code to reproduce our experiments is available at https://github.com/castorini/transformers-arithmetic
Forward citations
Cited by 8 Pith papers
-
Length Generalization Bounds for Transformers
Length-generalization bounds are uncomputable for general two-layer transformers but computable and tight-exponential for fixed-precision transformers.
-
Artificial Effort
Twenty-three LLMs solve most of eight standard real-effort tasks accurately and cheaply, with no response to verbal incentives, so unsupervised real-effort measures may no longer capture human effort.
-
Variable-length Gray codes for the Natural Numbers
Every finite digit string gets exactly one number, and consecutive numbers differ by a single edit — a complete, locally stable variable-length Gray code.
-
Learning to Execute Graph Algorithms Exactly with Graph Neural Networks
A GNN with an ensemble of MLPs can exactly execute any LOCAL-model graph algorithm after learning a polynomial-size set of local template instructions.
-
The Effect of Scripts and Formats on LLM Numeracy
LLM arithmetic accuracy falls sharply when numerals leave the familiar Hindu–Arabic format, and few-shot prompting with examples narrows most of that gap.
-
A Group Theoretic Analysis of the Symmetries Underlying Base Addition and Their Learnability by Neural Networks
For bases 3-5, the structural complexity of a base-addition carry rule (fractal dimension, carry frequency, associativity) strongly predicts whether a tiny recurrent network can learn it and generalize from 3-digit to...
-
Long-Short Alignment for Effective Long-Context Modeling in LLMs
A long-short misalignment metric quantifies output distribution drift across context lengths, correlates with long-context performance, and a regularizer based on it improves fine-tuned LLMs.
-
ExploreGS: Explorable 3D Scene Reconstruction with Virtual Camera Samplings and Diffusion Priors
Adding information-gain-selected virtual views refined by video diffusion priors to 3D Gaussian Splatting improves arbitrary-view rendering quality.
Discussion (0). Sign in to comment.