REVIEW 8 cited by
Transformers without Normalization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve the same or better performance using a remarkably simple technique. We introduce Dynamic Tanh (DyT), an element-wise operation $DyT($x$) = \tanh(\alpha $x$)$, as a drop-in replacement for normalization layers in Transformers. DyT is inspired by the observation that layer normalization in Transformers often produces tanh-like, $S$-shaped input-output mappings. By incorporating DyT, Transformers without normalization can match or exceed the performance of their normalized counterparts, mostly without hyperparameter tuning. We validate the effectiveness of Transformers with DyT across diverse settings, ranging from recognition to generation, supervised to self-supervised learning, and computer vision to language models. These findings challenge the conventional understanding that normalization layers are indispensable in modern neural networks, and offer new insights into their role in deep networks.
Forward citations
Cited by 8 Pith papers
-
DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD
DNT is a transformer with InputNorm, PreNorm, MidNorm, and QKNorm that trains successfully with vanilla mSGDW, reaching performance close to AdamW on ImageNet and OpenWebText.
-
Accurate Parameter-Efficient Test-Time Adaptation for Time Series Forecasting
PETSA adapts frozen time-series forecasters at test time using gated low-rank input and output calibration modules and a combined robust, frequency, and structural loss, matching or beating full-model TTA with far few...
-
BinConv: A Neural Architecture for Ordinal Encoding in Time-Series Forecasting
A paper combines a cumulative binary encoding of target values with a lightweight convolutional architecture to improve time series forecasting accuracy and speed.
-
Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification
LayerNorm parameter shifts after fine-tuning encode domain-transition information; rescaling them via an FSR-dependent scalar lambda plus a cyclic step improves ViT classification under data scarcity and domain shift.
-
Discontinuity-aware KAN-based physics-informed neural networks
A discontinuity-aware PINN with adaptive Fourier features, DyT-plus-spline KAN activations, and learned local viscosity captures shocks with errors between 0.9% and 5% on benchmark PDEs and airfoil flows.
-
Taming LLMs by Scaling Learning Rates with Gradient Grouping
An optimizer wrapper that clusters per-layer momentum and scales learning rates by cluster-wise median deviations improves perplexity and accuracy across LLM and MLLM training, and lets LoRA pretraining approach full-...
-
Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMs
Pangu Light prunes LLMs across width, depth, and attention heads, then re-initializes remaining weights, achieving up to 4.2x throughput with modest benchmark loss.
-
EyeSim-VQA: A Free-Energy-Guided Eye Simulation Framework for Video Quality Assessment
A VQA model that combines free-energy-inspired frame restoration with a scan-and-gaze prediction head reports competitive results on five benchmarks, though the claimed state-of-the-art performance is not uniformly supported.
Discussion (0). Continue with ORCID to comment.