Pith. sign in

REVIEW 16 cited by

XAttention: Block Sparse Attention with Antidiagonal Scoring

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.16428 v1 pith:BHCDPFVP submitted 2025-03-20 cs.CL cs.CV

Ruyi Xu , Guangxuan Xiao , Haofeng Huang , Junxian Guo , Song Han This is my paper · ORCID
classification cs.CLcs.CV
keywords attentionxattentionblocklong-contextsparseaccuracyantidiagonalapplications
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Long-Context Transformer Models (LCTMs) are vital for real-world applications but suffer high computational costs due to attention's quadratic complexity. Block-sparse attention mitigates this by focusing computation on critical regions, yet existing methods struggle with balancing accuracy and efficiency due to costly block importance measurements. In this paper, we introduce XAttention, a plug-and-play framework that dramatically accelerates long-context inference in Transformers models using sparse attention. XAttention's key innovation is the insight that the sum of antidiagonal values (i.e., from the lower-left to upper-right) in the attention matrix provides a powerful proxy for block importance. This allows for precise identification and pruning of non-essential blocks, resulting in high sparsity and dramatically accelerated inference. Across comprehensive evaluations on demanding long-context benchmarks-including RULER and LongBench for language, VideoMME for video understanding, and VBench for video generation. XAttention achieves accuracy comparable to full attention while delivering substantial computational gains. We demonstrate up to 13.5x acceleration in attention computation. These results underscore XAttention's ability to unlock the practical potential of block sparse attention, paving the way for scalable and efficient deployment of LCTMs in real-world applications. Code is available at https://github.com/mit-han-lab/x-attention.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models

    cs.CV 2025-09 conditional novelty 7.0 of 10

    FG-Attn accelerates video DiT inference by computing attention over only the relevant Mx1 key slices per query group, using an asynchronous gather-load kernel, and reports up to 1.65x end-to-end speedup with negligibl...

  2. CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention

    cs.CL 2026-07 conditional novelty 6.0 of 10

    CoSA couples an order-aware sparse-mask proxy with a skip-capable attention kernel, achieving higher long-context accuracy at lower compute budgets than existing sparse-attention baselines.

  3. PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Sharing one proxy full-prefix scan across nearby query groups matches dense DSA indexer accuracy while accelerating indexing up to 4× and end-to-end latency up to 1.6×.

  4. Prism: Spectral-Aware Block-Sparse Attention

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Mean-pooling under RoPE attenuates high-frequency position signals (a spectral blind spot), and Prism's dual-band RMS-calibrated block selection recovers enough signal to match full attention at up to 5.1× prefill speedup.

  5. ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs

    cs.LG 2026-02 conditional novelty 6.0 of 10

    ParisKV retrieves top-K KV pairs on GPU using analytic sign-pattern centroids and calibrated 4-bit reranking, matching full attention accuracy in most settings while cutting million-token decode latency up to 44x.

  6. Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Sparse attention with chunk-aware sparsity growth and hierarchical frame/block selection accelerates autoregressive video diffusion at ~1.3x with VBench quality on par with dense attention.

  7. Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection

    cs.CL 2026-02 unverdicted novelty 6.0 of 10

    Token Sparse Attention uses dynamic per-head token compression and decompression during attention to achieve up to 3.23x speedup at 128K context with under 1% accuracy loss.

  8. LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework

    cs.CL 2025-07 conditional novelty 6.0 of 10

    LOOM-Scope is a framework that standardizes long-context LLM evaluation across 22 benchmarks and integrates a lightweight 12-benchmark suite, LOOMBench, for fast comprehensive assessment.

  9. HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference

    cs.LG 2025-07 conditional novelty 6.0 of 10

    HGCA splits attention between GPU (dense, recent KV) and CPU (sparse, salient KV) and merges partial results with exact log-sum-exp fusion, scaling long-context decoding on commodity GPUs.

  10. Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Chipmunk speeds up diffusion transformer inference by recomputing, at each step, only the top percent of attention and MLP activation columns that change most between steps, caching the rest in column-sparse GPU kernels.

  11. SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference

    cs.CV 2026-08 conditional novelty 5.0 of 10

    SPADE combines static, semi-static, and dynamic block-sparse attention with a cheap SICS-based blocking heuristic to speed up video diffusion inference by up to 1.80x end-to-end.

  12. Stem: Rethinking Causal Information Flow in Sparse Attention

    cs.LG 2026-03 conditional novelty 5.0 of 10

    Stem prunes causal attention with a position-decaying budget and a value-magnitude-aware score, matching dense accuracy on LongBench/RULER at a 25-31% compute budget.

  13. UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention

    cs.CV 2025-10 conditional novelty 5.0 of 10

    A pretrained FLUX diffusion model is adapted with local-window attention plus low-resolution global guidance, allowing 4K text-to-image generation from 1K-only training data at about 2x lower cost.

  14. MiniCPM4: Ultra-Efficient LLMs on End Devices

    cs.CL 2025-06 conditional novelty 5.0 of 10

    MiniCPM4-8B reportedly matches Qwen3-8B on standard benchmarks while using about 22% of the training tokens, and achieves large long-context speedups on edge devices.

  15. AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularity

    cs.LG 2025-05 conditional novelty 5.0 of 10

    AnchorAttention uses the maximum attention score from initial and local tokens as an anchor to threshold-select important key-value positions at stripe granularity, achieving faster prefill with comparable accuracy.

  16. SeerAttention-R: Sparse Attention Adaptation for Long Reasoning

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A learned gate selects the important KV blocks during long decoding, preserving math reasoning accuracy while skipping up to 90% of attention work.

Pith tools