Pith. sign in

REVIEW 3 major objections 5 minor 68 references

CoSA, a training-free sparse-attention method that co-designs the importance proxy with the kernel via an ordered computation mask, achieves higher accuracy at lower compute budgets than prior proxy-based methods.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:50 UTC pith:WTUNLZUE

load-bearing objection CoSA's ordered-mask co-design is a real step forward for sparse prefilling, but the headline LongBench-v2 numbers are tuned on the test set and the two closest kernel baselines are missing; the empirical core still holds. the 3 major comments →

arxiv 2607.25291 v1 pith:WTUNLZUE submitted 2026-07-28 cs.CL

CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention

Yufei Xue , Lin Niu , Hong Liu , Siran Liu , Hanyong Shao , Wei Liu , Guanghua Yu , Jianchen Zhu
show 1 more author
Jun Zhang
This is my paper
classification cs.CL
keywords sparse attentionlong-context inferenceprefilling speeduponline softmaxcomputation-order maskproxy-kernel co-designblock sparsityKV cache paging
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that existing proxy-based sparse attention fails when the compute budget is tight because the proxy and the kernel are designed in isolation: the proxy emits a binary block mask that increasingly misses salient blocks, and the kernel applies that mask mechanically. CoSA instead couples a Kernel-Aware Proxy (KAP) with an Ordered-Skipping Kernel (OSK) through a single computation-order mask that tells the kernel which KV pages to visit and in what order. The proxy flags blocks likely to contain each query row's maximum attention (HRM blocks) and sorts them first, so the kernel's online-softmax running maximum rises early, making its exact in-kernel skip more accurate and more aggressive. On synthetic and real-world long-context benchmarks across two mainstream 8B backbones, CoSA reports the strongest accuracy among sparse methods while consuming the lowest compute budget, with a 4.93x attention speedup and 2.53x time-to-first-token speedup at a 128K context.

Core claim

The paper's central discovery is that the fidelity loss of sparse attention under tightening budgets stems from the disconnection between the proxy and the kernel, and that replacing the binary block mask with a computation-order mask restores fidelity while increasing sparsity. KAP estimates per-block importance from stride-downsampled QK scores and marks each block with an HRM flag indicating whether it contains a query row's maximum; flag-grouped sorting places HRM blocks first. OSK then consumes this ordered mask through KV-page remapping, physically jumping over masked pages, and applies a second-stage in-kernel skip on exact logits using a length-adaptive threshold. Because the running

What carries the argument

The central object is the computation-order mask MKAP, which for each query block lists the retained KV block IDs in the visiting order, with HRM-flagged blocks sorted first. It replaces a binary mask and serves as the interface between proxy and kernel. KAP constructs it via stride downsampling of Q and K, a cheap approximate softmax, key-dim MaxPool, max-normalization, per-query-block score summation, and an HRM flag; a flag-grouped descending sort yields the order. OSK physically jumps over pages according to MKAP using paged-KV remapping and applies a tile-wide in-kernel skip of Eq. (5): a page is skipped when the gap between its local row max and the running max is below ln(Delta/N). Or

Load-bearing premise

The accuracy and budget claims rest on the assumption that the stride-downsampled proxy reliably flags, per query block, the key block containing the true row-maximum under full attention; if downsampling or MaxPool misses that block, the flagged ordering front-loads the wrong pages, the running maximum stays low, and the in-kernel skip becomes both less aggressive and less accurate.

What would settle it

Construct a long-context case where the true row-max block lies between sampled key positions or at a block boundary so that the downsampled score does not flag it as HRM, then measure CoSA's attention accuracy and in-kernel skip ratio; if accuracy drops to the level of a binary-mask baseline while the skip ratio collapses, the ordering's benefit is disproved at that operating point.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At a 128K context, CoSA reports a 4.93x attention speedup and a 2.53x reduction in end-to-end time-to-first-token with negligible accuracy degradation.
  • On both evaluated 8B backbones and across synthetic and real-world long-context benchmarks, CoSA reports the highest average accuracy among compared sparse methods while consuming the lowest compute budget.
  • The ordered-mask design makes the exact in-kernel skip more aggressive: remapping raises the skip ratio at matched perplexity across stage-1 budgets from 100% down to 28%.
  • Because CoSA is training-free and consumes a paged KV cache, the same mask interface can be adopted by serving frameworks without model fine-tuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The co-design principle is likely transferable to other block-scoring proxies, not just the specific KAP; the key is that the kernel honors the proxy's ordering, so any proxy that can rank blocks by row-max likelihood should see the same safety-accuracy gain.
  • The paper explicitly limits sparsification to prefilling and leaves decoding dense; extending HRM-first ordering to single-query decoding would require re-deriving the flag from per-token queries, but the order-invariance argument carries over.
  • The reported budgets combine stage-1 selection and stage-2 skipping, so an independent measurement of proxy overhead versus kernel page-remap cost would reveal whether the 4.93x speedup scales to context lengths well beyond 128K or flattens as proxy cost per block grows.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CoSA, a training-free two-stage block-sparse attention method for long-context inference. In stage one, a Kernel-Aware Proxy (KAP) stride-downsamples queries and keys, computes approximate post-softmax scores, pools them into block scores, and emits not just a binary selection mask but an ordered computation mask that front-loads blocks flagged as containing a query row's row-maximum (HRM flag). In stage two, an Ordered-Skipping Kernel (OSK) consumes this mask via KV-page remapping, so it physically jumps over mask-skipped pages, and applies an in-kernel softmax-threshold skip to further reduce value-side work. Experiments on Qwen3-8B and Llama-3.1-8B-Instruct over RULER and LongBench-v2 report higher accuracy at lower budgets than MInference, FlexPrefill, and XAttention, with up to 4.93x attention speedup and 2.53x TTFT speedup at 128K context.

Significance. If the accuracy/budget claims hold, CoSA is a useful practical contribution: it shows that coupling proxy-side ordering with kernel-side online-softmax skipping can push sparse prefilling to lower budgets without the accuracy collapse of binary-mask proxies. The paper has real strengths: Algorithm A gives a precise, implementable workflow; Table 3 provides a cumulative component ablation; Figure B is a kernel microbenchmark against MIT-BSA; and Figure 6 / Figure C give a PPL-budget Pareto analysis across several stage-1 budgets. The motivation in Section 3 (order-invariance of online softmax, the running-max limitation, and the bucket effect) is clearly articulated and directly informs the design.

major comments (3)
  1. [Sec. 4.2, Eq. (12), Fig. 2] The central ordering benefit rests on the HRM flag of Eq. (12) correctly identifying blocks that contain the true row-maximum under full attention. The flag is computed from stride-downsampled, MaxPooled, max-normalized proxy scores; if downsampling or within-block MaxPool misses the true max key, the flag-grouped sort of Eq. (13) front-loads the wrong blocks, the running max in Eq. (2) stays low, and the in-kernel skip of Eq. (5) is both less aggressive and less accurate. This is the load-bearing axiom of the co-design, yet it is only validated indirectly. Figure 2 shows IoU for existing proxies, not for KAP or for HRM flags; Table 3 shows cumulative accuracy/budget effects but does not isolate flag correctness. Please add a direct HRM hit-rate measurement against oracle rowmax locations as a function of stride and budget, and compare the flag-grouped order with an oracle-rowmax-first o
  2. [Sec. 5.4, Fig. 6, Appendix D.1, Table 2] The skip-scale Delta and the Stage-1 Top-p settings are calibrated on 128 random LongBench-v2 samples in Sec. 5.4 / Fig. 6, and the Llama p-table is described as calibrated offline in Appendix D.1. LongBench-v2 accuracy is then reported as a headline result in Table 2. This is a form of test-set tuning, and it can inflate the apparent advantage of CoSA over baselines whose hyperparameters are not tuned on the same set. The claim of being training-free is not the issue; the issue is that the reported LongBench-v2 numbers may be optimistically biased. Please either calibrate on a separate validation distribution, use a benchmark not involved in any hyperparameter choice, or explicitly report cross-validated / sensitivity results.
  3. [Sec. 2.2, Table A, Sec. 5.1, Tables 1-2] The closest in-kernel skipping methods, BLASST and SpargeAttention, are described in the related work and compared in Table A at the backend level, but they are absent from the accuracy and speed experiments in Tables 1-2 and Figure 5. Since CoSA's stage-2 mechanism is exactly the in-kernel skip used by these methods, the central claim of 'higher accuracy at lower budgets than existing proxy-based sparse attention' is not fully established without a direct matched-budget comparison against them. At minimum, report accuracy and speed for BLASST and SpargeAttention (or a stage-2-only variant of CoSA) on the same models and benchmarks.
minor comments (5)
  1. [Abstract and Sec. 5.3] The abstract states 'negligible performance degradation,' but Table 1 shows a 1.46-point drop on Qwen3-8B at 128K and Table 2 shows about a 1-point drop on LongBench-v2 average. Please quantify what 'negligible' means and report the dense-to-sparse gap explicitly.
  2. [Table 3] The '+ KAP' row for Llama increases budget from 35% to 39%; the text says accuracy rises 'at a comparable budget,' which is inaccurate. Please rephrase or explain why the budget increase is acceptable.
  3. [Algorithm A, line 14] Algorithm A updates m_i before the in-kernel skip test, while Eq. (5) compares m_loc against the running max from previously visited blocks. The two formulations are equivalent for the threshold as used, but the discrepancy should be noted or the code should use m' to avoid confusion.
  4. [Sec. 5.3 and Figure 5] No variance or error bars are reported for speedups or TTFT, despite 'multiple runs' being mentioned. Reporting standard deviations over at least a few runs would strengthen the efficiency claims.
  5. [Figure 2] The IoU figure would be more informative for the paper's own method if KAP's mask (or the HRM-grouped mask) were included alongside the existing proxies, especially since the motivation is that proxy fidelity degrades at low budgets.

Circularity Check

1 steps flagged

No equation-level circularity: the KAP/OSK pipeline is constructive and empirically measured. Main caveat is that the skip-scale Delta is swept on LongBench-v2 perplexity (Sec. 5.4) and that tuned knob then underlies the headline LongBench-v2 accuracy/budget claim (Table 2); RULER provides independent support, and the one self-citation (Stem) is not load-bearing.

specific steps
  1. fitted input called prediction [Sec. 5.4 (Fig. 6 / 'Effect of remapping') with Sec. 5.1 Hyperparameters and Table 2; Appendix D.1]
    "Using 128 random LongBench-v2 samples, we measure perplexity (PPL) and the in-kernel skip ratio across the Stage-1 budget BStage-1, the skip scale Delta, and the two remapping modes. ... For example, we set Delta_Qwen = 2000 for CoSA."

    The operating point of the method is set by the skip-scale Delta. Sec. 5.4 sweeps Delta on 128 LongBench-v2 samples and concludes 'Delta trades quality for sparsity in a predictable way and is safe to tune'; Sec. 5.1 then fixes Delta_Qwen=2000 for the main results, and Table 2 reports LongBench-v2 accuracy at the resulting 15% budget as the headline claim ('CoSA attains the best average among all sparse methods... at the lowest budget'). The LongBench-v2 accuracy-vs-budget result is therefore in-sample: the budget knob was chosen to minimize LongBench-v2 PPL on that same benchmark. This is a tuned operating point presented as a predicted outcome, not a derivation-level equivalence - the actual attention outputs are computed and measured, and RULER (Table 1) is an unaffected benchmark - but

full rationale

I walked the claimed derivation chain: (i) KAP builds approximate scores from stride-downsampled QK (Eqs. 8-11), defines HRM flags by max-normalization (Eqs. 10, 12), and emits an ordered mask via flag-grouped sort (Eqs. 13-14); (ii) OSK consumes this mask through page remapping and applies the BLASST-style in-kernel skip (Eq. 5), justified by the mathematical order-invariance of OSM (Eq. 4). This is a constructive pipeline whose output (accuracy, budget, speedup) is empirically measured, not a theorem whose conclusion is its premise. No equation-level circularity: the HRM flag is a heuristic proxy, not defined in terms of the reported accuracy or speedup, so Eq. 12 does not force the Table 1-3 outcomes. The flagged concern about HRM fidelity (whether stride-downsampled MaxPool locates the true rowmax block) is a correctness/validation risk - the paper supplies no closed-form bound and only indirect validation (Fig. 2, Table 3) - but under the reviewing rules that is not circularity, so I do not score it here. The one genuine near-circular step is tuning: the skip scale Delta (and the 'safe to tune' operating point) is chosen from a LongBench-v2 PPL sweep (Sec. 5.4), and LongBench-v2 accuracy is then reported as a headline result (Table 2), which makes that benchmark's claim partly in-sample. RULER is independent and partially supports the central claim, and the cumulative ablation (Table 3) shows the components move accuracy/budget in the claimed direction. Self-citations: Stem (Niu et al. 2026) shares authors (Lin Niu, Guanghua Yu, Jianchen Zhu) but is cited only as related work, not as load-bearing support; the in-kernel skip is attributed to external works (Yuan et al. 2025b; Zhang et al. 2025a). No self-citation chain carries the argument. Overall: minor-to-moderate circularity from the LongBench-v2 tuning-on-eval, with the core mechanism still supported by independent content.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The paper is an empirical systems contribution; it introduces no new physical or conceptual entities beyond the computation-order mask, which is a data structure. The main ledger items are tuned hyperparameters (Delta, p-table, stride, block/page sizes) and a set of modeling assumptions about proxy fidelity and order-invariance under skipping.

free parameters (4)
  • Stage-1 Top-p threshold p = p=0.95 (Qwen); offline-calibrated p-table (Llama)
    Controls the Stage-1 budget; calibrated on benchmark data and directly affects the accuracy/budget trade-off.
  • In-kernel skip scale Delta = Delta=2000 (Qwen); Delta=200 (Llama)
    Tuned using PPL on 128 LongBench-v2 samples (Sec. 5.4, Fig. 6); directly controls skip aggressiveness and is central to the reported budget numbers.
  • Stride size s for query/key downsampling = not specified in text
    Determines proxy cost and fidelity; not reported in the main text or Appendix D.1, leaving an important replication detail unspecified.
  • Block size b and KV page size b_p = b=128; b_p configured independently (not specified)
    Kernel design choices that affect mask granularity, remapping overhead, and measured speedup.
axioms (5)
  • standard math Order-invariance of exact online softmax (Property 2, Eq. 4)
    Mathematically true when all blocks are visited; the paper relies on it to justify reordering, but reordering changes which blocks are skipped by Eq. (5), so exactness does not directly carry over to the sparse output.
  • ad hoc to paper HRM flag fidelity (Eq. 12)
    Assumes a block with max-normalized stride-sampled score equal to 1 contains the true row-maximum; if downsampling misses it, the whole HRM-first ordering loses its intended effect.
  • domain assumption Proxy fidelity at moderate budgets (Property 1)
    Figure 2 shows empirically that proxies track the oracle at moderate budgets; this is used to justify Stage-1 selection and is not backed by a formal guarantee.
  • domain assumption BLASST in-kernel skip threshold preserves accuracy (Eq. 5)
    CoSA imports the Delta/N threshold from BLASST; the paper does not re-derive or bound the accuracy loss introduced by this heuristic.
  • domain assumption Page-remap overhead is negligible on H20
    The kernel speedup claim assumes arbitrary-order page remapping costs less than the QK and value work saved; this is hardware-specific and not separately benchmarked in isolation.

pith-pipeline@v1.3.0-alltime-deepseek · 17451 in / 11287 out tokens · 104360 ms · 2026-08-01T02:50:13.691820+00:00 · methodology

0 comments
read the original abstract

The quadratic cost of self-attention makes long-context inference prohibitively expensive, and proxy-based block-sparse attention has become a practical remedy. Existing methods typically rely on a proxy to predict a binary sparse mask and a kernel to consume this mask and perform sparse attention computation. Such an approach is effective under moderate budgets. However, as the budget tightens, the estimated proxy inevitably drops some salient blocks, while the kernel can only apply the sparse mask mechanically, leading to an evident drop in model accuracy. We propose CoSA, a two-stage training-free Sparse Attention under proxy-kernel CO-design, which couples a Kernel-Aware Proxy (KAP) with an Ordered-Skipping Kernel (OSK). In the first stage, the KAP selects blocks under a moderate budget and produces an ordered mask that prescribes the order in which KV pages are visited in the kernel inner loop. In the second stage, the OSK applies this mask and skips more blocks under a tightened budget given online-softmax statistics. Across mainstream LLM backbones and long-context benchmarks, CoSA attains higher accuracy at lower budgets. Impressively, CoSA achieves a 4.93$\times$ attention speedup and reduces end-to-end Time-to-First-Token by 2.53$\times$ under a context length of 128K with negligible performance degradation.

Figures

Figures reproduced from arXiv: 2607.25291 by Guanghua Yu, Hanyong Shao, Hong Liu, Jianchen Zhu, Jun Zhang, Lin Niu, Siran Liu, Wei Liu, Yufei Xue.

Figure 1
Figure 1. Figure 1: Performance and budget of mainstream sparse at [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: IoU of proxy masks against the oracle for Qwen3- [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Position of rowmax (red dots) over the attention [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of CoSA. MKAP: the KAP mask serves as a bridge between the co-designed proxy and kernel. Left: KAP estimates a per-block score together with the HRM flag. They are merged into the computation-order mask MKAP, which places HRM blocks first. Right: OSK consumes MKAP through KV page remapping, yielding three per-block patterns: ➀ mask-skipped, ➁ in-kernel skipped, and ➂ computed. max and can veto the… view at source ↗
Figure 5
Figure 5. Figure 5: Attention (left) and time-to-first-token (TTFT) (right) speedups over the Full Attention baseline across context lengths ranging from 4K to 128K. budget. Notably, remapping tightens budget without an accu￾racy degradation. This is exactly the interplay our co-design is built for, where the proxy-side ordering and the kernel-side skip only pay off when they act together. Effect of remapping [PITH_FULL_IMAG… view at source ↗
Figure 6
Figure 6. Figure 6: Effect of KV page remapping on PPL and spar [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 33 linked inside Pith

  1. [1]

    2024 , journal=

    RULER: What's the Real Context Size of Your Long-Context Language Models? , author=. 2024 , journal=

    2024

  2. [2]

    arXiv preprint arXiv:2410.13276 , year=

    Seerattention: Learning intrinsic sparse attention in your llms , author=. arXiv preprint arXiv:2410.13276 , year=

    Pith/arXiv arXiv

  3. [3]

    arXiv preprint arXiv:2503.16428 , year=

    Xattention: Block sparse attention with antidiagonal scoring , author=. arXiv preprint arXiv:2503.16428 , year=

    Pith/arXiv arXiv

  4. [4]

    Advances in Neural Information Processing Systems , volume=

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention , author=. Advances in Neural Information Processing Systems , volume=

  5. [5]

    arXiv preprint arXiv:2502.20766 , year=

    Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference , author=. arXiv preprint arXiv:2502.20766 , year=

    Pith/arXiv arXiv

  6. [6]

    arXiv preprint arXiv:2503.17407 , year=

    A comprehensive survey on long context language modeling , author=. arXiv preprint arXiv:2503.17407 , year=

    arXiv

  7. [7]

    arXiv preprint arXiv:2503.13299 , year=

    A survey on transformer context extension: Approaches and evaluation , author=. arXiv preprint arXiv:2503.13299 , year=

    Pith/arXiv arXiv

  8. [8]

    arXiv preprint arXiv:2310.06770 , year=

    Swe-bench: Can language models resolve real-world github issues? , author=. arXiv preprint arXiv:2310.06770 , year=

    Pith/arXiv arXiv

  9. [9]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2503.09567 , year=

    Towards reasoning era: A survey of long chain-of-thought for reasoning large language models , author=. arXiv preprint arXiv:2503.09567 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  12. [12]

    IEEE transactions on knowledge and data engineering , volume=

    A general survey on attention mechanisms in deep learning , author=. IEEE transactions on knowledge and data engineering , volume=. 2021 , publisher=

    2021

  13. [13]

    A Survey of Efficient Attention Methods: Hardware-efficient, Sparse, Compact, and Linear Attention , author=

  14. [14]

    arXiv preprint arXiv:2004.05150 , year=

    Longformer: The long-document transformer , author=. arXiv preprint arXiv:2004.05150 , year=

    Pith/arXiv arXiv 2004

  15. [15]

    Advances in neural information processing systems , volume=

    Big bird: Transformers for longer sequences , author=. Advances in neural information processing systems , volume=

  16. [16]

    arXiv preprint arXiv:2412.15204 , year=

    LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks , author=. arXiv preprint arXiv:2412.15204 , year=

    Pith/arXiv arXiv

  17. [17]

    L ong B ench: A Bilingual, Multitask Benchmark for Long Context Understanding

    Bai, Yushi and Lv, Xin and Zhang, Jiajie and Lyu, Hongchang and Tang, Jiankai and Huang, Zhidian and Du, Zhengxiao and Liu, Xiao and Zeng, Aohan and Hou, Lei and Dong, Yuxiao and Tang, Jie and Li, Juanzi. L ong B ench: A Bilingual, Multitask Benchmark for Long Context Understanding. Proceedings of the 62nd Annual Meeting of the Association for Computation...

    doi:10.18653/v1/2024.acl-long.172 2024

  18. [18]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2410.10819 , year=

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads , author=. arXiv preprint arXiv:2410.10819 , year=

    Pith/arXiv arXiv

  20. [20]

    arXiv preprint arXiv:2406.14909 , year=

    Moa: Mixture of sparse attention for automatic large language model compression , author=. arXiv preprint arXiv:2406.14909 , year=

    arXiv

  21. [21]

    arXiv preprint arXiv:2305.13245 , year=

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints , author=. arXiv preprint arXiv:2305.13245 , year=

    Pith/arXiv arXiv

  22. [22]

    and Ermon, Stefano and Rudra, Atri and R

    Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R. Flash. Advances in Neural Information Processing Systems (NeurIPS) , year=

  23. [23]

    Dao, Tri , booktitle=. Flash

  24. [24]

    Advances in Neural Information Processing Systems , volume=

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision , author=. Advances in Neural Information Processing Systems , volume=

  25. [25]

    arXiv preprint arXiv:2603.05451 , year=

    Flashattention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling , author=. arXiv preprint arXiv:2603.05451 , year=

    arXiv

  26. [26]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  27. [27]

    arXiv preprint arXiv:2502.11089 , year=

    Native sparse attention: Hardware-aligned and natively trainable sparse attention , author=. arXiv preprint arXiv:2502.11089 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2506.07900 , year=

    Minicpm4: Ultra-efficient llms on end devices , author=. arXiv preprint arXiv:2506.07900 , year=

    Pith/arXiv arXiv

  29. [29]

    arXiv preprint arXiv:2309.17453 , year=

    Efficient streaming language models with attention sinks , author=. arXiv preprint arXiv:2309.17453 , year=

    Pith/arXiv arXiv

  30. [30]

    arXiv preprint arXiv:2509.24745 , year=

    ProxyAttn: Guided Sparse Attention via Representative Heads , author=. arXiv preprint arXiv:2509.24745 , year=

    arXiv

  31. [31]

    arXiv preprint arXiv:2603.12201 , year=

    IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse , author=. arXiv preprint arXiv:2603.12201 , year=

    arXiv

  32. [32]

    arXiv preprint arXiv:2502.18137 , year=

    Spargeattention: Accurate and training-free sparse attention accelerating any model inference , author=. arXiv preprint arXiv:2502.18137 , year=

    arXiv

  33. [33]

    arXiv preprint arXiv:2602.13515 , year=

    Spargeattention2: Trainable sparse attention via hybrid top-k+ top-p masking and distillation fine-tuning , author=. arXiv preprint arXiv:2602.13515 , year=

    arXiv

  34. [34]

    arXiv preprint arXiv:2603.06199 , year=

    Flashprefill: Instantaneous pattern discovery and thresholding for ultra-fast long-context prefilling , author=. arXiv preprint arXiv:2603.06199 , year=

    arXiv

  35. [35]

    arXiv preprint arXiv:2602.08426 , year=

    Prism: Spectral-aware block-sparse attention , author=. arXiv preprint arXiv:2602.08426 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2512.12087 , year=

    BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding , author=. arXiv preprint arXiv:2512.12087 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2603.26380 , year=

    Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers , author=. arXiv preprint arXiv:2603.26380 , year=

    Pith/arXiv arXiv

  38. [38]

    arXiv preprint arXiv:2601.17367 , year=

    Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers , author=. arXiv preprint arXiv:2601.17367 , year=

    arXiv

  39. [39]

    arXiv preprint arXiv:2501.01005 , year =

    FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving , author =. arXiv preprint arXiv:2501.01005 , year =

    Pith/arXiv arXiv

  40. [40]

    Proceedings of the 29th symposium on operating systems principles , pages=

    Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=

  41. [41]

    CoRR abs/2310.01889 (2023) , author=

    Ring Attention with Blockwise Transformers for Near-Infinite Context. CoRR abs/2310.01889 (2023) , author=

    Pith/arXiv arXiv 2023

  42. [42]

    International Conference on Learning Representations , volume=

    Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration , author=. International Conference on Learning Representations , volume=

  43. [43]

    arXiv preprint arXiv:2411.10958 , year=

    Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization , author=. arXiv preprint arXiv:2411.10958 , year=

    arXiv

  44. [44]

    arXiv preprint arXiv:2505.21136 , year=

    Sageattention2++: A more efficient implementation of sageattention2 , author=. arXiv preprint arXiv:2505.21136 , year=

    Pith/arXiv arXiv

  45. [45]

    2026 , publisher =

    GitHub repository , howpublished =. 2026 , publisher =

    2026

  46. [46]

    2023 , publisher =

    GitHub repository , howpublished =. 2023 , publisher =

    2023

  47. [47]

    GitHub repository , howpublished =

    Guo, Junxian and Tang, Haotian and Yang, Shang and Zhang, Zhekai and Liu, Zhijian and Han, Song , title =. GitHub repository , howpublished =. 2024 , publisher =

    2024

  48. [48]

    arXiv preprint arXiv:2606.13392 , year=

    MiniMax Sparse Attention , author=. arXiv preprint arXiv:2606.13392 , year=

    Pith/arXiv arXiv

  49. [49]

    Advances in Neural Information Processing Systems , volume=

    Moba: Mixture of block attention for long-context llms , author=. Advances in Neural Information Processing Systems , volume=

  50. [50]

    arXiv preprint arXiv:2602.03560 , year=

    HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing , author=. arXiv preprint arXiv:2602.03560 , year=

    arXiv

  51. [51]

    2: Pushing the frontier of open large language models , author=

    Deepseek-v3. 2: Pushing the frontier of open large language models , author=. arXiv preprint arXiv:2512.02556 , year=

    Pith/arXiv arXiv

  52. [52]

    arXiv , author=

    Yarn: Efficient context window extension of large language models. arXiv , author=. arXiv preprint arXiv:2309.00071 , year=

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2507.20534 , year=

    Kimi k2: Open agentic intelligence , author=. arXiv preprint arXiv:2507.20534 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2602.15763 , year=

    Glm-5: from vibe coding to agentic engineering , author=. arXiv preprint arXiv:2602.15763 , year=

    Pith/arXiv arXiv

  55. [55]

    Science , volume=

    Agentic AI and the next intelligence explosion , author=. Science , volume=. 2026 , publisher=

    2026

  56. [56]

    arXiv preprint arXiv:2503.18016 , year=

    Retrieval augmented generation and understanding in vision: A survey and new outlook , author=. arXiv preprint arXiv:2503.18016 , year=

    Pith/arXiv arXiv

  57. [57]

    arXiv preprint arXiv:2602.03442 , year=

    A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces , author=. arXiv preprint arXiv:2602.03442 , year=

    arXiv

  58. [58]

    arXiv preprint arXiv:2406.10774 , year=

    Quest: Query-aware sparsity for efficient long-context llm inference , author=. arXiv preprint arXiv:2406.10774 , year=

    Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:1805.02867 , year=

    Online normalizer calculation for softmax , author=. arXiv preprint arXiv:1805.02867 , year=

    Pith/arXiv arXiv

  60. [60]

    arXiv preprint arXiv:2510.07486 , year=

    AsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decoding , author=. arXiv preprint arXiv:2510.07486 , year=

    arXiv

  61. [61]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Tokenselect: Efficient long-context inference and length extrapolation for llms via dynamic token-level kv cache selection , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  62. [62]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  63. [63]

    arXiv preprint arXiv:2603.06274 , year=

    Stem: Rethinking Causal Information Flow in Sparse Attention , author=. arXiv preprint arXiv:2603.06274 , year=

    arXiv

  64. [64]

    arXiv preprint arXiv:2503.07518 , year=

    Tokenbutler: Token importance is predictable , author=. arXiv preprint arXiv:2503.07518 , year=

    Pith/arXiv arXiv

  65. [65]

    arXiv preprint arXiv:2605.16928 , year=

    Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps , author=. arXiv preprint arXiv:2605.16928 , year=

    Pith/arXiv arXiv

  66. [66]

    arXiv preprint arXiv:2511.20102 , year=

    SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space , author=. arXiv preprint arXiv:2511.20102 , year=

    Pith/arXiv arXiv

  67. [67]

    arXiv preprint arXiv:2512.14082 , year=

    A Unified Sparse Attention via Multi-Granularity Compression , author=. arXiv preprint arXiv:2512.14082 , year=

    arXiv

  68. [68]

    arXiv preprint arXiv:2602.05853 , year=

    RRAttention: Dynamic Block Sparse Attention via Per-Head Round-Robin Shifts for Long-Context Inference , author=. arXiv preprint arXiv:2602.05853 , year=

    arXiv