GLU improves optimization speed over non-GLU by reshaping the NTK spectrum toward smaller condition numbers and more compact eigenvalues, with empirical tests showing the benefit is mainly in training dynamics rather than generalization.
Consider ReLU model first: z(x) =Vϕ(Wx)
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2026 1verdicts
UNVERDICTED 1representative citing papers
citing papers explorer
-
The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?
GLU improves optimization speed over non-GLU by reshaping the NTK spectrum toward smaller condition numbers and more compact eigenvalues, with empirical tests showing the benefit is mainly in training dynamics rather than generalization.