Pith. sign in

REVIEW 4 cited by

nGPT: Normalized Transformer with Representation Learning on the Hypersphere

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.01131 v2 pith:MZTO45HE submitted 2024-10-01 cs.LG cs.AI

Ilya Loshchilov , Cheng-Ping Hsieh , Simeng Sun , Boris Ginsburg This is my paper · ORCID
classification cs.LGcs.AI
keywords hyperspherengptnormalizedattentionlearningrepresentationsametransformer
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere. In nGPT, all vectors forming the embeddings, MLP, attention matrices and hidden states are unit norm normalized. The input stream of tokens travels on the surface of a hypersphere, with each layer contributing a displacement towards the target output predictions. These displacements are defined by the MLP and attention blocks, whose vector components also reside on the same hypersphere. Experiments show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Universal One-third Time Scaling in Learning Peaked Distributions

    cs.LG 2026-02 conditional novelty 7.0 of 10

    Softmax + cross-entropy on peaked targets yields loss ∼ t^{−1/3}, giving an architecture-driven explanation for power-law LLM training time without power-law data.

  2. Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

    cs.LG 2025-09 unverdicted novelty 7.0 of 10

    Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...

  3. Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Splitting weight matrices into a fixed-norm direction and learnable per-row/column magnitudes improves LLM training over AdamW/Muon, removes weight decay and warmup, and transfers the optimal LR across width.

  4. Peri-LN: Revisiting Normalization Layer in the Transformer Architecture

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Peri-LN, which normalizes both the input and output of each sublayer, reduces activation-variance growth and gradient spikes during LLM pretraining, outperforming Pre-LN and Post-LN at scales up to 3.2B parameters.

Pith tools