Pith. sign in

REVIEW 5 cited by

Transformers without Tears: Improving the Normalization of Self-Attention

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1910.05895 v2 pith:DVLKKOYY submitted 2019-10-14 cs.CL cs.LGstat.ML

Toan Q. Nguyen , Julian Salazar This is my paper · ORCID
classification cs.CLcs.LGstat.ML
keywords performancetrainingbleuchangesfixnormnormalizationprenormscalenorm
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PreNorm) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose $\ell_2$ normalization with a single scale parameter (ScaleNorm) for faster training and better performance. Finally, we reaffirm the effectiveness of normalizing word embeddings to a fixed length (FixNorm). On five low-resource translation pairs from TED Talks-based corpora, these changes always converge, giving an average +1.1 BLEU over state-of-the-art bilingual baselines and a new 32.8 BLEU on IWSLT'15 English-Vietnamese. We observe sharper performance curves, more consistent gradient norms, and a linear relationship between activation scaling and decoder depth. Surprisingly, in the high-resource setting (WMT'14 English-German), ScaleNorm and FixNorm remain competitive but PreNorm degrades performance.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling

    cs.LG 2025-06 conditional novelty 6.0 of 10

    SGD with momentum can match Adam's performance in language modeling when trained with small batches and careful tuning, a result that contradicts several popular explanations for the optimizer gap.

  2. Language as a Wave Phenomenon: Semantic Phase Locking and Interference in Neural Networks

    cs.LG 2025-12 reject novelty 5.0 of 10

    A small complex-valued spectral model (PRISM) and a hybrid Wave-Particle Transformer are claimed to show that phase-based interference is a sufficient reasoning primitive, with a 4.94 vs 5.28 perplexity win on WikiTex...

  3. Efficient and Effective Query Context-Aware Learning-to-Rank Model for Sequential Recommendation

    cs.IR 2025-07 conditional novelty 5.0 of 10

    Adding shifted query-context embeddings to the last attention layer's query position (plus the output head) improves next-item ranking in offline and online tests, though open-dataset validation uses a target-derived ...

  4. UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A redesigned memory-layer architecture with five engineering improvements reaches performance parity with 8-expert MoE at similar compute, with lower memory access and stronger long-context memorization.

  5. Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization

    cs.LG 2025-06 reject novelty 4.0 of 10

    Wavelet-based positional encodings are claimed to improve how transformers extrapolate to longer sequences, with a toy experiment supporting the claim but with weak theory.

Pith tools