REVIEW 4 cited by
Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Going beyond stochastic gradient descent (SGD), what new phenomena emerge in wide neural networks trained by adaptive optimizers like Adam? Here we show: The same dichotomy between feature learning and kernel behaviors (as in SGD) holds for general optimizers as well, including Adam -- albeit with a nonlinear notion of "kernel." We derive the corresponding "neural tangent" and "maximal update" limits for any architecture. Two foundational advances underlie the above results: 1) A new Tensor Program language, NEXORT, that can express how adaptive optimizers process gradients into updates. 2) The introduction of bra-ket notation to drastically simplify expressions and calculations in Tensor Programs. This work summarizes and generalizes all previous results in the Tensor Programs series of papers.
Forward citations
Cited by 4 Pith papers
-
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
Rank survival in Transformer blocks is governed by a branch-to-skip ratio law (βα^M√L), a mean-spike coherence c_ℓ=E[σ]²/E[σ²], and a Marchenko–Pastur width threshold m/d=1/p(σ).
-
Dimension-adapted Momentum Outscales SGD
DANA, with dimension- and time-dependent momentum, provably outscales SGD on power-law random features when 2α>1, improving loss exponents and compute-optimal curves.
-
Distillation Scaling Laws
A distillation scaling law predicts student cross-entropy from teacher loss, student size, and data, and gives compute-optimal teacher-student allocations.
-
SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
SOAP and Muon, stabilized by per-step QR eigenbasis updates and KL-Shampoo covariance accumulation, beat AdamW on large-batch LLM pretraining up to 100M-token batches.
Discussion (0). Continue with ORCID to comment.