REVIEW 1 cited by
An Isometric Stochastic Optimizer
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The Adam optimizer is the standard choice in deep learning applications. I propose a simple explanation of Adam's success: it makes each parameter's step size independent of the norms of the other parameters. Based on this principle I derive Iso, a new optimizer which makes the norm of a parameter's update invariant to the application of any linear transformation to its inputs and outputs. I develop a variant of Iso called IsoAdam that allows optimal hyperparameters to be transferred from Adam, and demonstrate that IsoAdam obtains a speedup over Adam when training a small Transformer.
Forward citations
Cited by 1 Pith paper
-
SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training
SWAN, a stateless optimizer combining gradient normalization and whitening, matches or beats Adam on LLaMA pretraining through 1.3B parameters with roughly half the memory and reported 2x token efficiency.
Discussion (0). Continue with ORCID to comment.