The reviewed record of science sign in
Pith

arxiv: 2311.05928 · v2 · pith:DOSBXT2R · submitted 2023-11-10 · cs.CL · cs.AI· cs.IT· cs.LG· math.GN· math.IT

The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models

Anton Razzhigaev , Matvey Mikhalchuk , Elizaveta Goncharova , Ivan Oseledets , Denis Dimitrov , Andrey Kuznetsov This is my paper

Reviewed by Pithpith:DOSBXT2Ropen to challenge →

classification cs.CL cs.AIcs.ITcs.LGmath.GNmath.IT
keywords anisotropydecodersencodersintrinsicdimensionembeddingstrainingtransformer
0
0 comments X
read the original abstract

In this study, we present an investigation into the anisotropy dynamics and intrinsic dimension of embeddings in transformer architectures, focusing on the dichotomy between encoders and decoders. Our findings reveal that the anisotropy profile in transformer decoders exhibits a distinct bell-shaped curve, with the highest anisotropy concentrations in the middle layers. This pattern diverges from the more uniformly distributed anisotropy observed in encoders. In addition, we found that the intrinsic dimension of embeddings increases in the initial phases of training, indicating an expansion into higher-dimensional space. Which is then followed by a compression phase towards the end of training with dimensionality decrease, suggesting a refinement into more compact representations. Our results provide fresh insights to the understanding of encoders and decoders embedding properties.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Transformers for Learning on Noisy and Task-Level Manifolds: Approximation and Generalization Insights

    cs.LG 2025-05 unverdicted novelty 7.0

    Transformers achieve approximation and generalization error bounds for noisy manifold regression that scale with the intrinsic dimension of the task-level manifold.

  2. Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

    cs.LG 2026-05 unverdicted novelty 6.0

    Contribution Weights combine attention, value magnitude, and directional alignment to measure token influence more faithfully than attention alone, and show attention sinks actively suppress information via a convex s...