REVIEW 3 cited by
SHAPE: Shifted Absolute Position Embedding for Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Position representation is crucial for building position-aware representations in Transformers. Existing position representations suffer from a lack of generalization to test data with unseen lengths or high computational cost. We investigate shifted absolute position embedding (SHAPE) to address both issues. The basic idea of SHAPE is to achieve shift invariance, which is a key property of recent successful position representations, by randomly shifting absolute positions during training. We demonstrate that SHAPE is empirically comparable to its counterpart while being simpler and faster.
Forward citations
Cited by 3 Pith papers
-
On the Generalizability of Transformer Models to Code Completions of Different Lengths
Across two languages and three metrics, no tested positional encoding scheme generalizes to code completion lengths unseen in training; mixed-length training is the recommended safe choice.
-
Rethinking Associative Memory Mechanism in Induction Head
A two-layer transformer with relative positional encoding keeps its induction head active across the whole sequence, while absolute positional encoding loses it in the second half.
-
V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding
V2PE assigns visual tokens smaller and variable positional increments than text tokens, which allows a 2B vision-language model to effectively process multimodal sequences up to 1M tokens.
Discussion (0). Continue with ORCID to comment.