REVIEW 8 cited by
Efficient Transformers: A Survey
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of "X-former" models have been proposed - Reformer, Linformer, Performer, Longformer, to name a few - which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency. With the aim of helping the avid researcher navigate this flurry, this paper characterizes a large and thoughtful selection of recent efficiency-flavored "X-former" models, providing an organized and comprehensive overview of existing work and models across multiple domains.
Forward citations
Cited by 8 Pith papers
-
Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
Freezing query and key attention weights still lets transformers form induction heads and stay close to standard performance on language modeling, while random static attention (MixiT) fails on in-context tasks but su...
-
TransX: Scaling Transformer-based Recommendation via Behavioral and Serving Stream Crossings
A dual-stream encoder-decoder recommender with nearline-cached behavior encoding reports higher CTR/CVR than DLRM baselines at roughly one-fifth of the online compute in LinkedIn deployment.
-
ARC-Encoder: learning compressed text representations for large language models
ARC-Encoder pools queries in an encoder's last attention layer to produce compressed continuous representations that a frozen decoder consumes as token embeddings.
-
AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs
A training-free method that prunes low-magnitude dimensions of projected query/key vectors in attention, cutting dot-product cost by 25% with small benchmark degradation.
-
Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization
Central-difference random gradient estimation (CD-RGE) trains RNNs up to 1.1B parameters without backpropagation through time, matching or beating BPTT on several small tasks while using less GPU memory.
-
Gated Associative Memory: A Parallel O(N) Architecture for Efficient Sequence Modeling
GAM combines a causal convolution and a fixed memory-bank retrieval with a learnable gate to match or beat Transformer and Mamba at small scale.
-
Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization
Wavelet-based positional encodings are claimed to improve how transformers extrapolate to longer sequences, with a toy experiment supporting the claim but with weak theory.
-
Crisp Attention: Regularizing Transformers via Structured Sparsity
Top-k attention sparsity during fine-tuning gave DistilBERT a 0.97% accuracy gain on SST-2, but the result comes from single runs without error bars.
Discussion (0). Sign in to comment.