REVIEW 7 cited by
NormFormer: Improved Transformer Pretraining with Extra Normalization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
During pretraining, the Pre-LayerNorm transformer suffers from a gradient magnitude mismatch: gradients at early layers are much larger than at later layers. These issues can be alleviated by our proposed NormFormer architecture, which adds three normalization operations to each layer: a Layer Norm after self attention, head-wise scaling of self-attention outputs, and a Layer Norm after the first fully connected layer. The extra operations incur negligible compute cost (+0.4% parameter increase), but improve pretraining perplexity and downstream task performance for both causal and masked language models ranging from 125 Million to 2.7 Billion parameters. For example, adding NormFormer on top of our strongest 1.3B parameter baseline can reach equal perplexity 24% faster, or converge 0.27 perplexity better in the same compute budget. This model reaches GPT3-Large (1.3B) zero shot performance 60% faster. For masked language modeling, NormFormer improves fine-tuned GLUE performance by 1.9% on average. Code to train NormFormer models is available in fairseq https://github.com/pytorch/fairseq/tree/main/examples/normformer .
Forward citations
Cited by 7 Pith papers
-
A Controlled Study of Attention-Only Transformers
Attention-only transformers match standard transformers within 0.006 nats of loss at matched parameter count, with the residual gap localized to low-context parametric recall.
-
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
Rank survival in Transformer blocks is governed by a branch-to-skip ratio law (βα^M√L), a mean-spike coherence c_ℓ=E[σ]²/E[σ²], and a Marchenko–Pastur width threshold m/d=1/p(σ).
-
Dual Attention Residuals
Reciprocal cross-stream depth retrieval improves Transformer pretraining loss across dense and MoE models from 0.1B to 7B.
-
Enhancing next token prediction based pre-training for jet foundation models
Using continuous particle features as input and combining next-token with masked-token pre-training markedly improves classification accuracy of the OmniJet jet foundation model without visibly hurting its generative quality.
-
SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
SpanNorm—a block-level residual with PostNorm-style normalization—trains deeper transformers more stably and outperforms PreNorm and hybrid normalization on LM benchmarks.
-
Residual Matrix Transformers: Scaling the Size of the Residual Stream
The RMT uses an outer product memory as the residual stream, allowing the residual stream to be scaled up at negligible cost and improving language model efficiency.
-
Understanding Transformer from the Perspective of Associative Memory
Frames the Transformer as associative memory, derives retrieval SNR for linear, softmax, ReLU, and SoLU kernels, and proposes DeltaFormer, a softmax-plus-delta-rule variant claimed to exceed TC0 expressivity.
Discussion (0). Continue with ORCID to comment.