REVIEW 1 cited by
Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through which different components of Transformer, such as the dot-product self-attention, positional encoding and feed-forward layer, affect its expressive power, and we study their combined effects through establishing explicit approximation rates. Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads. These theoretical insights are validated experimentally and offer natural suggestions for alternative architectures.
Forward citations
Cited by 1 Pith paper
-
An Analysis for Reasoning Bias of Language Models with Small Initialization
Initialization scale controls whether a transformer learns compositional reasoning or memorized mappings, because reasoning tokens acquire more differentiated embeddings early in training.
Discussion (0). Continue with ORCID to comment.