Pith. sign in

REVIEW 5 cited by

Memory-efficient Transformers via Top-$k$ Attention

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2106.06899 v1 pith:B37CJUVA submitted 2021-06-13 cs.CL cs.LG

Ankit Gupta , Guy Dar , Shaya Goodman , David Ciprut , Jonathan Berant This is my paper · ORCID
classification cs.CLcs.LG
keywords attentionvanillalayersmemorytop-approachapproximationcompute
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Following the success of dot-product attention in Transformers, numerous approximations have been recently proposed to address its quadratic complexity with respect to the input length. While these variants are memory and compute efficient, it is not possible to directly use them with popular pre-trained language models trained using vanilla attention, without an expensive corrective pre-training stage. In this work, we propose a simple yet highly accurate approximation for vanilla attention. We process the queries in chunks, and for each query, compute the top-$k$ scores with respect to the keys. Our approach offers several advantages: (a) its memory usage is linear in the input size, similar to linear attention variants, such as Performer and RFA (b) it is a drop-in replacement for vanilla attention that does not require any corrective pre-training, and (c) it can also lead to significant memory savings in the feed-forward layers after casting them into the familiar query-key-value framework. We evaluate the quality of top-$k$ approximation for multi-head attention layers on the Long Range Arena Benchmark, and for feed-forward layers of T5 and UnifiedQA on multiple QA datasets. We show our approach leads to accuracy that is nearly-identical to vanilla attention in multiple setups including training from scratch, fine-tuning, and zero-shot inference.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Power Law Guided Dynamic Sifting for Efficient Attention

    cs.LG 2025-06 conditional novelty 6.0 of 10

    SiftAttention skips top-k sorting in sparse attention by thresholding attention weights with a threshold predicted from a power-law fit of score quantiles over generation steps.

  2. HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference

    cs.LG 2025-06 conditional novelty 6.0 of 10

    HATA learns per-head binary hash codes for queries and keys and selects top-k tokens by Hamming distance, achieving up to 7.2x decoding speedup over full attention with near-baseline accuracy on tested benchmarks.

  3. Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Static per-head thresholds calibrated on a small dataset can replace top-k selection in transformer attention, achieving 3 to 10x sparsity with negligible accuracy loss.

  4. Geometry-aware Depth-guided Representation Learning for Structure-preserving Low-light Image Enhancement

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Depth estimated from Retinex reflectance, fused via multi-scale attention in an encoder, improves structural consistency of low-light image enhancement over appearance-only baselines.

  5. Intrinsically Interpretable Attention via Sparse Post-Training

    cs.LG 2025-12 conditional novelty 5.0 of 10

    Sparsity-regularised post-training can cut LLM attention connectivity to ~0.4% while keeping loss near baseline, yielding up to 100x smaller interpretable circuits.

Pith tools