REVIEW 7 cited by
Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformer models have been widely adopted in various domains over the last years, and especially large language models have advanced the field of AI significantly. Due to their size, the capability of these networks has increased tremendously, but this has come at the cost of a significant increase in necessary compute. Quantization is one of the most effective ways to reduce the computational time and memory consumption of neural networks. Many studies have shown, however, that modern transformer models tend to learn strong outliers in their activations, making them difficult to quantize. To retain acceptable performance, the existence of these outliers requires activations to be in higher bitwidth or the use of different numeric formats, extra fine-tuning, or other workarounds. We show that strong outliers are related to very specific behavior of attention heads that try to learn a "no-op" or just a partial update of the residual. To achieve the exact zeros needed in the attention matrix for a no-update, the input to the softmax is pushed to be larger and larger during training, causing outliers in other parts of the network. Based on these observations, we propose two simple (independent) modifications to the attention mechanism - clipped softmax and gated attention. We empirically show that models pre-trained using our methods learn significantly smaller outliers while maintaining and sometimes even improving the floating-point task performance. This enables us to quantize transformers to full INT8 quantization of the activations without any additional effort. We demonstrate the effectiveness of our methods on both language models (BERT, OPT) and vision transformers.
Forward citations
Cited by 7 Pith papers
-
FPTQuant: Function-Preserving Transforms for LLM Quantization
FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.
-
Efficient Reasoning on the Edge
LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.
-
LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
LISA replaces standard attention with linear attention plus indexer-selected sparse attention, claiming O(nM) inference cost and accuracy gains, but the complexity analysis ignores the indexer's full-context scoring.
-
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
Attention-sink weight is recast as an implicit MoE router per head, motivating a sink-aware head-balancing loss that yields small, consistent benchmark gains across three attention variants but rests on a definitional...
-
Thresholded Cross-Attention for Reliable Intensity-Chromaticity Fusion in Low-Light Image Enhancement
TCA-Net improves low-light image enhancement by replacing fixed-quota Top-K attention with an adaptive confidence threshold for intensity-chromaticity fusion in the HVI color space.
-
Rethinking the Outlier Distribution in Large Language Models: An In-depth Study
The paper claims that most massive activations in LLMs are residual-propagated 'fake' outliers that can be removed without accuracy loss.
-
On the Mathematical Impossibility of Safe Universal Approximators
This preprint argues that useful universal approximators must have dense catastrophic failures, but the proof rests on unproven measure convergence and circular use of adversarial examples.
Discussion (0). Continue with ORCID to comment.