Pith. sign in

REVIEW 11 cited by

Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.16747 v1 pith:BQDSYDBB submitted 2024-06-24 cs.CL cs.LG

Chao Lou , Zixia Jia , Zilong Zheng , Kewei Tu This is my paper · ORCID
classification cs.CLcs.LG
keywords attentionsparsekmemorysparsecomplexitycomputationalconstantduring
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Accommodating long sequences efficiently in autoregressive Transformers, especially within an extended context window, poses significant challenges due to the quadratic computational complexity and substantial KV memory requirements inherent in self-attention mechanisms. In this work, we introduce SPARSEK Attention, a novel sparse attention mechanism designed to overcome these computational and memory obstacles while maintaining performance. Our approach integrates a scoring network and a differentiable top-k mask operator, SPARSEK, to select a constant number of KV pairs for each query, thereby enabling gradient-based optimization. As a result, SPARSEK Attention offers linear time complexity and constant memory footprint during generation. Experimental results reveal that SPARSEK Attention outperforms previous sparse attention methods and provides significant speed improvements during both training and inference, particularly in language modeling and downstream tasks. Furthermore, our method can be seamlessly integrated into pre-trained Large Language Models (LLMs) with minimal fine-tuning, offering a practical solution for effectively managing long-range dependencies in diverse applications.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection

    cs.LG 2026-07 reject novelty 6.0 of 10

    Per-block gzip compression ratios, computed on the full input, select 'literal' blocks for long-range attention and yield 1.71 BPB on PG-19 byte-level LM at 92M parameters—but the evaluation assumes access to future content.

  2. MatchAttention: Embedding Explicit Matching Constraints into Attention for Efficient Stereo Matching

    cs.CV 2025-10 conditional novelty 6.0 of 10

    MatchAttention replaces global cross-attention with a linearly-scaling, learnable-offset windowed attention whose relative position field is the predicted disparity/flow, yielding state-of-the-art stereo/flow accuracy...

  3. LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework

    cs.CL 2025-07 conditional novelty 6.0 of 10

    LOOM-Scope is a framework that standardizes long-context LLM evaluation across 22 benchmarks and integrates a lightweight 12-benchmark suite, LOOMBench, for fast comprehensive assessment.

  4. OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference

    cs.CL 2025-07 conditional novelty 6.0 of 10

    OrthoRank selects tokens for computation by their orthogonality to the sink token in normalized hidden states, yielding better perplexity and accuracy than layer pruning at matched sparsity without additional training.

  5. LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions

    cs.AI 2025-06 conditional novelty 6.0 of 10

    Language agents' believability and goal achievement decline over multi-episode social interactions, and curated memory summaries only partially close the gap with humans.

  6. ESPFormer: Doubly-Stochastic Attention with Expected Sliced Transport Plans

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A new Transformer attention mechanism that builds doubly-stochastic attention from sliced optimal transport plans with soft sorting, yielding modest gains over baselines.

  7. LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

    cs.AI 2026-05 reject novelty 5.0 of 10

    LISA replaces standard attention with linear attention plus indexer-selected sparse attention, claiming O(nM) inference cost and accuracy gains, but the complexity analysis ignores the indexer's full-context scoring.

  8. SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformers

    cs.LG 2025-08 conditional novelty 5.0 of 10

    SCOUT matches full-attention transformer scores on long-context language modeling and reasoning benchmarks at 400M and 1.3B scales while attending only to strided checkpoint tokens.

  9. Lag-Relative Sparse Attention In Long Context Training

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Training a language model with LagKV's static sparse attention mask improves its robustness to KV-cache compression at 2x and 4x rates, while claims of overall average gains are not consistently supported.

  10. Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents

    cs.AI 2025-02 conditional novelty 5.0 of 10

    The authors propose episodic memory, with five defining properties, as the unifying framework needed for LLM agents to learn and remember over long time horizons.

  11. Crisp Attention: Regularizing Transformers via Structured Sparsity

    cs.CL 2025-08 reject novelty 3.0 of 10

    Top-k attention sparsity during fine-tuning gave DistilBERT a 0.97% accuracy gain on SST-2, but the result comes from single runs without error bars.

Pith tools