REVIEW 8 cited by
A Survey on Transformer Compression
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformer plays a vital role in the realms of natural language processing (NLP) and computer vision (CV), specially for constructing large language models (LLM) and large vision models (LVM). Model compression methods reduce the memory and computational cost of Transformer, which is a necessary step to implement large language/vision models on practical devices. Given the unique architecture of Transformer, featuring alternative attention and feedforward neural network (FFN) modules, specific compression techniques are usually required. The efficiency of these compression methods is also paramount, as retraining large models on the entire training dataset is usually impractical. This survey provides a comprehensive review of recent compression methods, with a specific focus on their application to Transformer-based models. The compression methods are primarily categorized into pruning, quantization, knowledge distillation, and efficient architecture design (Mamba, RetNet, RWKV, etc.). In each category, we discuss compression methods for both language and vision tasks, highlighting common underlying principles. Finally, we delve into the relation between various compression methods, and discuss further directions in this domain.
Forward citations
Cited by 8 Pith papers
-
LinDeps: A Fine-tuning Free Post-Pruning Method to Remove Layer-Wise Linear Dependencies with Guaranteed Performance Preservation
A post-pruning method uses pivoted QR decomposition to delete linearly dependent CNN filters and least-squares kernel adjustment to preserve output, gaining small extra compression without fine-tuning.
-
Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones
A task-conditioned projection operator converts a large transformer's weights into a smaller task-specialized transformer that outperforms same-size universal conditional models.
-
Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Construction
Distill-then-Replace builds task-specific hybrid attention LLMs by distilling each full-attention block into a linear counterpart and greedily replacing layers under a validation-performance constraint.
-
DistrAttention: An Efficient and Flexible Self-Attention Mechanism on Modern GPUs
DistrAttention approximates self-attention by LSH-based grouping of embedding-dimension columns of Q and K, reducing compute along d while keeping all tokens in context.
-
A Comparative Study of Pruning Methods in Transformer-based Time Series Forecasting
A benchmark shows most time-series Transformers tolerate about 50% unstructured pruning without clear accuracy loss, while structured pruning rarely delivers meaningful inference speedups.
-
MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformers
KL-isolation fragility plus MCKP bit allocation yields mixed-precision ViT PTQ that lags recent ImageNet PTQ but reports large COCO AP gains at MP3/MP3.
-
TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization
TinyMusician distills MusicGen and applies hand-picked mixed-precision quantization to make a 1.04 GB on-device music generator, but the headline '93% quality, 55% smaller' claims conflict with the paper's own tables.
-
A Survey on Large Language Model Acceleration based on KV Cache Management
A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.
Discussion (0). Continue with ORCID to comment.