Weight matrix dynamics during training is modeled as Dyson Brownian motion, with the learning-rate-to-batch-size ratio controlling stochasticity; verified analytically in a Gaussian RBM and empirically in a nano-GPT.
Dyson, A Brownian-Motion Model for the Eigenvalues of a Random Matrix, J
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cond-mat.dis-nn 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Dyson Brownian motion and random matrix dynamics of weight matrices during learning
Weight matrix dynamics during training is modeled as Dyson Brownian motion, with the learning-rate-to-batch-size ratio controlling stochasticity; verified analytically in a Gaussian RBM and empirically in a nano-GPT.