Pith. sign in

REVIEW 6 cited by

DeepNet: Scaling Transformers to 1,000 Layers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2203.00555 v1 pith:HKACEKVM submitted 2022-03-01 cs.CL cs.LG

Hongyu Wang , Shuming Ma , Li Dong , Shaohan Huang , Dongdong Zhang , Furu Wei This is my paper · ORCID
classification cs.CLcs.LG
keywords transformersmodeldeepdeepnormlayerlayersmethodparameters
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 54 citations worldwide. Full citation record

  1. Dynamic Parameterization Is Not Dynamic Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Dynamic parameterization alone does not establish dynamic inference or computational savings, as demonstrated by a frozen-controller audit on FeatureGate and MUDDPythia.

  2. When Does Sparsity Mitigate the Curse of Depth in LLMs

    cs.CL 2026-03 conditional novelty 6.0 of 10

    Implicit and explicit sparsity reduce residual-stream variance and improve layer effectiveness metrics, enabling a depth-scaling recipe with about 4.6 points higher downstream accuracy.

  3. Multi-Head Attention Residuals

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Splitting the depth-routing query into per-subspace heads (a parameter-free reshape) improves Transformer validation loss at 100M–1B and mid-training at 8B.

  4. AutoNorm: Understanding Adaptive Normalization in Transformers through Differentiable Gating

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Gate-freezing after warm-up stabilizes differentiable LN/DyT selection in Transformers, helping non-stationary NLP tasks while remaining competitive on vision.

  5. The Algorithm Is Not the Behavior: Learned Priors Override Look-Ahead in a Chess-Playing Neural Network

    cs.LG 2025-08 reject novelty 5.0 of 10

    The paper demonstrates non-monotonic move-policy dynamics in a chess transformer, but its abstract claims a causal safety-prior override result that never appears in the body.

  6. Taming Transformer Without Using Learning Rate Warmup

    cs.LG 2025-05 conditional novelty 5.0 of 10

    A spectral-norm ratio rule on the optimizer shrinks per-layer learning rates when weight updates are too large, letting Transformers train without warmup.

Pith tools