REVIEW 4 cited by
Scaling Laws for Associative Memories
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Learning arguably involves the discovery and memorization of abstract rules. The aim of this paper is to study associative memory mechanisms. Our model is based on high-dimensional matrices consisting of outer products of embeddings, which relates to the inner layers of transformer language models. We derive precise scaling laws with respect to sample size and parameter size, and discuss the statistical efficiency of different estimators, including optimization-based algorithms. We provide extensive numerical experiments to validate and interpret theoretical results, including fine-grained visualizations of the stored memory associations.
Forward citations
Cited by 4 Pith papers
-
Muon in Associative Memory Learning: Training Dynamics and Scaling Laws
In a linear softmax memory model, Muon equalizes learning across frequency tiers and gives exponential (noiseless) or T^{-2} (noisy power-law) convergence, versus polynomial or T^{-(1-1/β)} for gradient descent.
-
From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis
Zipf's law, via differential Heaps and Hilberg laws, forces a power-law lower bound on the excess cross entropy of any entropy-bounded foundation model.
-
Extending LLM Context via Associative Recurrent Memory
ARMT-augmented 1B-class LLMs, trained with continued pretraining, synthetic long data, curriculum, and selective memory layers, keep in-window quality while generalizing past 32k–65k tokens at constant memory and ~30%...
-
The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representations
FACT is a first-order stationarity identity for weight matrices that matches or beats the Neural Feature Ansatz as a description of learned features at convergence.
Discussion (0). Continue with ORCID to comment.