LLM training produces models that approach the information bottleneck bound on compression, with compression optimality and retained information predicting performance on diverse benchmarks.
The 1.7B Smol model was trained on 11 Trillion tokens and performs comparably to the 1B OLMo2 model which was trained on 4 Trillion Tokens
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2026 1verdicts
UNVERDICTED 1representative citing papers
citing papers explorer
-
Learning is Forgetting: LLM Training As Lossy Compression
LLM training produces models that approach the information bottleneck bound on compression, with compression optimality and retained information predicting performance on diverse benchmarks.