Pith. sign in

REVIEW 1 major objections 6 minor 44 references

LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

T0 review · 1 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LongCat Sparse Attention claims that the practical cost of learned sparse attention sits in its indexer—$O(L^2)$ scoring plus scattered memory access—and removes both by reserving contiguous sink and window tokens, sharing one indexer…

desk verdict Serious sparse-attention systems paper: good integration and scale, but 1M parity claims need per-length evidence. read the letter →

arxiv 2608.01662 v2 pith:SHK7BCSQ submitted 2026-08-03 cs.AI cs.CLcs.DCcs.LG

Wen Zan , Jiaqi Zhang , Jianchao Tan , Hong Liu , Cunguang Wang , Xiang Li , Duyue Ma , Guanyu Wu
show 6 more authors
Yifan Lu Fengcun Li Yerui Sun Peng Pei Yuchen Xie Xunliang Cai
This is my paper · ORCID
classification cs.AIcs.CLcs.DCcs.LG
keywords sparseattentionlong-contextlanguagemodelscross-layerindexingsinksLightningIndexerhardware-softwareco-designKVcachemulti-tokenprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the practical bottleneck in fine-grained learned sparse attention is not the attention computation itself but the indexer that picks which tokens to attend to: scoring every prefix token costs $O(L^2)$, and the scattered token sets it returns force non-coalesced, low-bandwidth memory access. LongCat Sparse Attention (LSA) attacks both problems with three composable mechanisms: Streaming-Aware Indexing reserves half the attention budget for fixed, contiguous sink and sliding-window tokens; Cross-Layer Indexing trains one indexer per pair of adjacent layers and reuses its selection for both; and Hierarchical Indexing applies a training-free coarse-to-fine page-and-token selection at long contexts. The paper reports that at 69B-A3B and 560B-A27B scale, LSA matches or slightly exceeds full attention on HELMET and standard reasoning and coding benchmarks, while cutting per-layer training latency relative to DSA by about 1.5–1.6x and prefill latency by up to 3.6x at one million tokens. If the claim holds, sparse attention can be trained natively at one-million-token contexts at roughly full-attention quality, which shifts the next constraint from compute to KV-cache memory.

What carries the argument

The load-bearing mechanism is the cross-layer distillation loss of Eq. (8), which sums the per-layer indexer distillation losses over all $N$ layers of a group so the owner layer's indexer learns to predict the joint attention pattern of the group, making index reuse safe. Two supporting objects carry the rest: the Hybrid Sparse Attention kernel, which runs the sliding-window branch and the dynamically sparse branch on separate hardware streams and merges them with online-softmax rescaling so that roughly half of all KV reads are contiguous, and the two-stage Hierarchical Indexer, which first recalls the top-$M$ pages of size $P = 128$ using mean-pooled sub-block scores and then scores tokens only inside those pages, reducing per-query selection cost from $O(L)$ to $O(L/P + M P)$. The fixed-to-dynamic budget split (16 sink tokens plus 1024 window tokens against 1024 indexer-chosen tokens) is what turns the observed streaming-mass structure into a hardware-aligned memory layout.

What would settle it

Measure an owner layer's top-K selection coverage on its reuse partner's attention at 256K–1M context on non-retrieval workloads such as long-document summarization or repository-code tasks; a material drop from the 93.2% coverage observed at 8K would break the cross-layer leg of the claim. A second, simpler test is to run the $N=2$ versus $N=4$ ablation at 1M context, since the paper's own results put success and failure only a factor of two apart.

Watch

Extended reading notes

Core claim

LSA's central claim is that retrieval-based sparse attention can be made practical by exploiting structure already present in full attention. The paper's measurements show that roughly 83% of head-averaged attention mass falls inside sink and sliding-window regions, and that adjacent transformer layers share 57.4% of their top-K token sets while reusing a neighbor layer's set still captures 93.2% of the target layer's attention mass. On that basis LSA converts about half of the sparse budget to fixed contiguous regions, shares a single indexer across each pair of layers under a group-wise distillation loss (Eq. 8) that trains the owner indexer to serve the whole group's attention patterns, and runs a two-stage page-then-token selection beyond 256K context. With a total budget of $K = 2048$, LSA is reported to match or slightly exceed full attention on long-context and general benchmarks at both model scales, while the streaming and cross-layer components cut attention-layer training latency to 1.53–1.61x over DSA and the hierarchical module speeds up the indexer up to 4.11x at 1024K.

Load-bearing premise

The cross-layer reuse premise: a single indexer trained with the group distillation loss can select tokens that preserve essentially all attention mass for every layer in its group, and this stability persists at one million tokens and across tasks rather than only in the 8K needle-style settings where the 93.2% coverage figure was measured.

Editorial extensions

If this is right

  • Indexer cost stops scaling with the full context: a group size of $N=2$ halves the number of indexing passes, the fixed window shrinks the scoring range, and beyond 256K the hierarchical selector keeps prefill indexer latency nearly flat, reaching a 4.11x indexer speedup at 1024K.
  • Native one-million-token training becomes a practical claim rather than an inference-only trick: the paper converts from dense attention at the 128K stage, trains sparsely through the 256K and 1M stages, and reports loss gaps to full attention below 0.5%.
  • Speculative decoding inherits the savings: all three MTP draft steps share a single index set with negligible acceptance-length change (3.11 versus 3.15 for dense MLA), so draft quality does not pay for cross-layer reuse.
  • The binding constraint moves from compute to memory: LSA does not reduce the KV-cache footprint, so at extreme contexts the remaining gains are capped by cache capacity and transfer, a limitation the paper states explicitly.
  • Speedups compound with scale: LSA runs slightly slower than dense MLA below 64K but beats it by up to 7.73x per attention layer at 1024K, giving practitioners a clear crossover point for converting dense checkpoints.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable ceiling follows from the $N=2$ versus $N=4$ ablation: the safe reuse depth is set by how fast layer saliency drifts, so models whose attention patterns diverge more across layers (deeper stacks, interleaved modality or tool layers) should need smaller groups, and that prediction could be checked against the coverage-versus-distance curve without retraining.
  • The 1:1 fixed-to-dynamic budget was chosen for hardware alignment, and the ablation shows anything from 0% to 75% fixed preserves quality at 128K; a natural extension is that task mixtures heavy in code or mathematics, where attention is less streaming, may want a smaller fixed share.
  • If LSA's framing is right, the next round of gains lies on the memory side: combining this indexer design with KV compression along the depth or sequence dimension would attack the constraint the paper itself names as remaining, since the compute-side savings are already largely banked.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper presents LongCat Sparse Attention (LSA), a hardware-algorithm co-designed sparse attention framework built on DeepSeek Sparse Attention (DSA). LSA combines three mechanisms: (1) Streaming-Aware Indexing (SI), which reserves roughly half of the attention budget for fixed sink and sliding-window regions to enable coalesced memory access; (2) Cross-Layer Indexing (CLI), which trains an indexer in the first layer of a group to serve all layers in the group via a cross-layer KL distillation loss, halving or further reducing indexer passes; and (3) Hierarchical Indexing (HI), a two-stage coarse-to-fine page/token selection that reduces indexing cost at long contexts. The authors validate LSA on LongCat-Flash-Lite (69B-A3B) and LongCat-Flash (560B-A27B), reporting quality parity with dense MLA on general and long-context benchmarks, training and inference speedups over DSA, and the release of LongCat-Flash-Lite-Sparse with native 1M-token context length. The central claim is that LSA preserves full-attention quality while substantially reducing indexing overhead.

Significance. If the parity claim holds, LSA is a practically relevant contribution: it directly attacks the two dominant bottlenecks of DSA-style sparse attention (non-coalesced gathers and quadratic indexer cost) with orthogonal, composable mechanisms, and it validates these at contest-scale model sizes (69B and 560B). The paper's strengths include an unusually thorough ablation program (budget splits, CLI group sizes, distillation ablations, HI pooling and recall ablations, conversion-timing robustness), a clear formal treatment of the three mechanisms in Sections 3.1-3.3, and the release of an open-source model (LongCat-Flash-Lite-Sparse). The cross-layer distillation loss (Eq. 8) is a clean, well-motivated modification of the DSA objective, and the streaming-budget analysis in Fig. 2 gives empirical grounding to SI. The main weakness is that the headline quality claim is not actually demonstrated at the longest lengths: the fixed HI candidate budget creates an increasingly aggressive recall constraint as context grows, and the experiments that would expose a length-dependent collapse (per-length scores at 512K and 1M, and a same-condition dense baseline at those lengths) are absent.

major comments (1)
  1. [Section 4.2, Fig. 4 and Tables 2-4] All latency and speedup numbers are measured on proprietary accelerators with kernels that are not released; the open-source model is released, but the custom kernels (HFA, the HI two-stage operator, KVP support) are not. This makes the efficiency claims difficult to reproduce independently. This is not a correctness error, but it should be stated more prominently in the limitations, and any publicly verifiable kernel-level details (e.g., pseudocode, memory-access patterns, or a reference implementation) would strengthen the paper.
minor comments (6)
  1. [Section 5.1] The text reads 'an-shot needle-in-a-haystack task'; this appears to be a typo for 'an n-shot needle-in-a-haystack task'.
  2. [Section 6.1] The heading 'Model Informations' should be 'Model Information'.
  3. [Table 11] The column header 'Method / Size' with values '1 4 8 16 32' is confusing; please clarify that rows are pooling methods and columns are pooling sizes.
  4. [Table 15] The table compares only Lite-Sparse (w/o HI) vs. Lite-Sparse (w/ HI) and reports length-averaged scores; please add a dense MLA column or an explicit statement that no same-condition dense baseline exists at these lengths, and provide per-length scores at least for the longest lengths.
  5. [Table 16] The Lite-Dense scores are imported from the LongCat-Flash-Lite technical report rather than measured under identical conditions; this should be stated in the caption or text, since the comparison is therefore not a same-harness evaluation.
  6. [Introduction and abstract] The abstract says LSA supports native training up to one million tokens, but the 560B model's maximum training length is 256K (Table 6) and the 1M claim is demonstrated only at the 69B scale; please qualify the claim accordingly.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: LSA parity is tested against an external dense MLA baseline; minor self-citations and benchmark-driven hyperparameter choices are not load-bearing.

full rationale

The paper's central claim—that LSA performs on par with full attention—is established by comparing LSA against an external dense MLA baseline on HELMET and standard benchmarks (Tables 7 and 8), not by an equation that reduces to the method's own training objective. The indexer is trained by KL distillation (Eqs. 4-5, 8) toward the full-attention distribution, but downstream benchmark parity is not guaranteed by that objective and is measured on held-out tasks. The three mechanism choices (fixed streaming budget 50%, CLI group size N=2, HI recall M=1024) are selected through ablations against MLA and against the no-HI baseline; this is ordinary model selection, not a fitted parameter renamed as a prediction. The only self-citation of note is the Lite-Dense baseline in Table 16, imported from the LongCat-Flash-Lite technical report [Liu et al., 2026] with overlapping authors; it is not load-bearing because the core parity result is already established in-house against measured MLA in Tables 7-8. The fixed HI recall budget (M*P=131K tokens) at 1M context is a genuine robustness/correctness risk, but it is a limitation of the scheme, not a circular step: the paper's equations do not define the evaluation outcome in terms of the method's own outputs. No step in the derivation chain reduces to its own input or to a self-citation chain.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

All load-bearing assumptions are empirical regularities measured on the authors' own models and benchmarks; none are derived from first principles. The free parameters are configuration choices tuned via ablations on the same benchmark families used for the headline claims.

free parameters (9)
  • Ksink (sink budget) = 16
    Sink size fixed without a sweep; a standard value from StreamingLLM.
  • Kswa (sliding window budget) = 1024
    Selected by ablation to give roughly 1:1 fixed-to-sparse budget ratio in Section 5.3.1.
  • Ksparse (dynamic budget) = 1024
    Derived as K minus Ksink minus Kswa with total K=2048 inherited from DSA.
  • CLI group size N = 2
    N=4 caused measurable long-context degradation; N=2 halves indexer passes without measured loss in Section 5.3.2.
  • HI page size P = 128
    Fixed page size for coarse recall, chosen in Section 3.3 without an ablation.
  • HI sub-block size B = 8
    Pooling block size selected via the NIAH 128K ablation in Table 11.
  • HI recall pages M = 1024
    Candidate page count chosen to preserve MRCR quality at 256K and 512K in Table 13.
  • HI disabled early indexers = 4
    Turning off HI for the first 4 layers improved NIAH 128K from 84 to 92 in Table 12.
  • HI enablement threshold = >=256K
    HI is applied only at sequence lengths where profiled speedup exceeds its overhead, per Section 4.1.2.
assumptions (5)
  • domain assumption The sink and sliding-window regions capture a large and stable fraction of attention mass.
    Used to justify allocating roughly 50% of the budget to fixed contiguous regions; measured on 20 InfBench-QA samples at 8K in Fig. 2 and Section 3.1.
  • domain assumption Salient token sets are stable across adjacent layers.
    Used for cross-layer index reuse; measured as 57.4% top-K overlap and 93.2% attention-mass coverage on 20 samples at 8K in Fig. 3 and Section 3.2.
  • domain assumption Mean-pooled sub-block keys provide a sufficient coarse recall signal.
    Used for Hierarchical Indexing; validated empirically on NIAH and MRCR rather than derived, in Section 3.3 and Section 5.3.5.
  • domain assumption The DSA indexer is the dominant latency bottleneck at long contexts and its scattered output limits HBM bandwidth.
    Motivates all three components; based on internal profiling on proprietary accelerators in Section 2.3.
  • domain assumption Cross-layer distillation in Eq. (8) can jointly supervise one indexer for a group of layers without changing the rest of the model.
    Core to CLI; tested only at group sizes 2 and 4 on one architecture family in Section 5.3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing." pith.science (2026). https://pith.science/paper/SHK7BCSQ

@misc{pith2026260801662,
  author       = {Pith},
  title        = {Pith review of: LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SHK7BCSQ}},
  note         = {Machine review of arXiv:2608.01662}
}
abstract

DeepSeek Sparse Attention (DSA) enables efficient long-context modeling through its Lightning Indexer. However, practical deployment remains constrained by the indexer's expensive $O(L^2)$ scoring overhead and the hardware-inefficient, discontinuous memory-access patterns induced by its outputs. To address these system-level bottlenecks, we introduce LongCat Sparse Attention (LSA), a hardware-algorithm co-designed framework comprising three complementary and orthogonal strategies: (1) Streaming-Aware Indexing, which selectively converts scattered KV entries into hardware-aligned contiguous layouts to enable coalesced HBM access; (2) Cross-Layer Indexing, which amortizes indexing overhead by reusing the results produced by a single layer across consecutive layers, supported by cross-layer distillation; and (3) Hierarchical Indexing, which adopts a coarse-to-fine scoring scheme to progressively narrow the candidate set for each query, thereby substantially reducing indexing computation. Extensive scaling experiments, ranging from 69B-A3B to 560B-A27B models, demonstrate that LSA consistently achieves performance on par with full attention across both general-purpose and long-context benchmarks. Moreover, LSA supports native training with context lengths of up to one million tokens and underpins the development of LongCat-2.0 (1.6T-A48B). To facilitate further research, we also introduce and open-source LongCat-Flash-Lite-Sparse (69B-A3B), which integrates LSA into LongCat-Flash-Lite and incorporates an updated long-context training corpus.

Figures

Figures reproduced from arXiv: 2608.01662 by the authors.

Figure 1
Figure 1. The architecture of the proposed LongCat Sparse Attention (LSA), featuring the streaming-aware hierarchical [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Attention mass distribution on a full-attention LongCat-Flash-Lite (69B-A3B) model with 14 short-cut MoE [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Cross-layer Top-K index analysis on full-attention LongCat-Flash-Lite (69B-A3B). Each layer independently selects its Top-K tokens from the full attention distribution. Only the upper triangle is shown in (a) and (b), correspond￾ing to the CLI reuse direction, where an earlier source layer provides the index set for later target layers). (a) Pairwise overlap of Top-K token sets. (b) Cumulative attention mass capture… view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Single-attention-layer training latency of LSA and DSA across context lengths. Bars report forward, backward, [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: End-to-end inference latency of LSA versus the DSA baseline: prefill time-to-first-token (TTFT, log scale) [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Streaming-Aware Indexing ablation. “X% fixed” = fraction of budget [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Cross-Layer Indexing ablation (base models). (a) Training loss gap relative to MLA. (b) Long-context [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: MTP metrics delta (DSA w/ CLI − MLA) for all 3 MTP steps. Accuracy and loss differences remain tightly centered around zero throughout training, confirming that CLI across MTP steps preserves prediction quality. We validate our design in Section 3.2 by comparing MLA wi…
Figure 9
Figure 9. Figure 9: Context-length extension ablation. (a) 128K-stage loss gap (LSA [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Single-attention-layer training latency of LSA and dense MLA across context lengths. Bars report forward, [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]
Figure 11
Figure 11. Figure 11: Abridged prompt for the case-study sample. The three spans analyzed in the visualizations below are color [PITH_FULL_IMAGE:figures/full_fig_p029_11.png]
Figure 12
Figure 12. Figure 12: Overview of attention maps for the case-study sample at four layers (12, 13, 26, 27). Columns: (1) LSA [PITH_FULL_IMAGE:figures/full_fig_p030_12.png]
Figure 13
Figure 13. Figure 13: LSA indexer selection mask for final-question query tokens at layer 26, whose indices are shared with layer [PITH_FULL_IMAGE:figures/full_fig_p031_13.png]
Figure 14
Figure 14. Figure 14: (a) Per-line selection density at layer 26: the fraction of each line’s tokens selected by the indexer, averaged [PITH_FULL_IMAGE:figures/full_fig_p031_14.png]
Figure 15
Figure 15. Figure 15: Per-token selection frequency (layer 26, shared with layer 27), averaged over the question queries, overlaid [PITH_FULL_IMAGE:figures/full_fig_p032_15.png]
Figure 16
Figure 16. Figure 16: LSA sparse attention weights for the final-question query tokens (tail rows, 88 tokens) at layer 26. Color is [PITH_FULL_IMAGE:figures/full_fig_p033_16.png]
Figure 17
Figure 17. Figure 17: Sparse selection quality over the question tokens (tail queries) for layers 12/13/26/27. Left: Selection overlap [PITH_FULL_IMAGE:figures/full_fig_p033_17.png]
Figure 18
Figure 18. Figure 18: Per-token LSA sparse attention (layer 26), averaged over the question queries (causal-aware), overlaid on the [PITH_FULL_IMAGE:figures/full_fig_p034_18.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 7 canonical work pages

  1. [5]

    Scaling embeddings outperforms scaling experts in language models.arXiv preprint arXiv:2601.21204,

    Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, Linkun Lyu, Jiaqi Sun, Xurui Yang, Bo Wang, Fengcun Li, Yulei Qian, et al. Scaling embeddings outperforms scaling experts in language models.arXiv preprint arXiv:2601.21204,

    arXiv

  2. [6]

    Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

    arXiv 1904

  3. [8]

    One of the special magic uuids for <key> is: <value>

    The indexer K cache and attention KV cache are sharded across KVP ranks at page granularity. Before processing each chunk, the complete cache blocks are all-gathered so that every rank can attend to the full context. • Decoding.Decoding uses a 4K chunk size. Short-context requests are served with data-parallel attention and expert parallelism (DP=16, EP=1...

    work page

  4. [9]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y . X. Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089,

    arXiv

  5. [10]

    Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150,

    Noam Shazeer. Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150,

    arXiv 1911

  6. [11]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024b

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024b. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhan...

    arXiv

  7. [12]

    Longcat-flash technical report.arXiv preprint arXiv:2509.01322,

    Meituan LongCat Team, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, et al. Longcat-flash technical report.arXiv preprint arXiv:2509.01322,

    arXiv

  8. [13]

    Indexcache: Accelerating sparse attention via cross-layer index reuse.arXiv preprint arXiv:2603.12201,

    24 LongCat Sparse Attention Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, and Juanzi Li. Indexcache: Accelerating sparse attention via cross-layer index reuse.arXiv preprint arXiv:2603.12201,

    arXiv

Show all 44 references
  1. [14]

    Helmet: How to evaluate long-context language models effectively and thoroughly.arXiv preprint arXiv:2410.02694,

    Howard Yen, Tianyu Gao, and Danqi Chen. Helmet: How to evaluate long-context language models effectively and thoroughly.arXiv preprint arXiv:2410.02694,

    arXiv

  2. [15]

    Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654,

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Gins- burg. Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654,

    arXiv

  3. [16]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Pro...

    2018

  4. [19]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun

    doi:10.1162/tacl_a_00023. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. ∞bench: Extending long context evaluation beyond 100k tokens.arXiv preprint arXiv:2402.13718,

    arXiv

  5. [20]

    Enabling large language models to generate text with citations

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6465–6488,

    2023

  6. [21]

    ASQA: Factoid questions meet long-form answers

    Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. ASQA: Factoid questions meet long-form answers. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8273–8288,

    2022

  7. [23]

    Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang

    Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can open-source llms truly promise on context length?, June 2023a. URL https: //lmsys.org/blog/2023-06-29-longchat. Dan Hendrycks, Collin Burns, Stev...

    2023 arXiv

  8. [24]

    Cmmlu: Measuring massive multitask language understanding in chinese.arXiv preprint arXiv:2306.09212, 2023b

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese.arXiv preprint arXiv:2306.09212, 2023b. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang,...

    arXiv

  9. [25]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022,

    arXiv

  10. [26]

    Measuring mathematical problem solving with the math dataset.Advances in Neural Information Processing Systems, 34, 2021b

    25 LongCat Sparse Attention Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.Advances in Neural Information Processing Systems, 34, 2021b. MAA. Aime 2024,

    2024

  11. [27]

    URL https://maa.org/math-competitions/ american-invitational-mathematics-examination-aime. MAA. Aime 2025,

    2025

  12. [28]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica

    doi:10.1145/3597926.3598030. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:24...

  13. [29]

    Asynctls: Efficient generative llm inference with asynchronous two-level sparse attention.arXiv preprint arXiv:2604.07815,

    Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai, and Jing Zhang. Asynctls: Efficient generative llm inference with asynchronous two-level sparse attention.arXiv preprint arXiv:2604.07815,

    arXiv

  14. [30]

    Hisa: Efficient hierarchical indexing for fine-grained sparse attention.arXiv preprint arXiv:2603.28458, 2026a

    Yufei Xu, Fanxu Meng, Fan Jiang, Yuxuan Wang, Ruijie Zhou, Zhaohui Wang, Jiexi Wu, Zhixin Pan, Xiaojuan Tang, Wenjie Pei, et al. Hisa: Efficient hierarchical indexing for fine-grained sparse attention.arXiv preprint arXiv:2603.28458, 2026a. Deli Huang, Cunguang Wang, Hongyin T...

    arXiv

  15. [31]

    Graphwalks dataset, 2025b

    OpenAI. Graphwalks dataset, 2025b. URLhttps://huggingface.co/datasets/openai/graphwalks. Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien MR Arnold, Vincent Perot, Siddharth Dalmia, et al. Can long-context language m...

    arXiv

  16. [32]

    Cheng Jiayang, Dongyu Ru, Lin Qiu, Yiyang Li, Xuezhi Cao, Yangqiu Song, and Xunliang Cai

    URL https://arxiv.org/ abs/2505.07897. Cheng Jiayang, Dongyu Ru, Lin Qiu, Yiyang Li, Xuezhi Cao, Yangqiu Song, and Xunliang Cai. Amemgym: Interactive memory benchmarking for assistants in long-horizon conversations. InThe Fourteenth International Conference on Learning Represe...

  17. [33]

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li

    URLhttps://openreview.net/forum?id=sfrVLzsmlf. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench v2: Towards deeper understanding and reasoning on realistic long-context mult...

    arXiv

  18. [34]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    URLhttps://artificialanalysis.ai/articles/announcing-aa-lcr. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770,

    arXiv

  19. [35]

    τ 2-bench: Evaluating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982,

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2-bench: Evaluating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982,

    arXiv

  20. [36]

    Vitabench: Benchmarking llm agents with versatile interactive tasks in real-world applications.arXiv preprint arXiv:2509.26490,

    Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Qi Gu, Chengcheng Han, Dengchang Zhao, Hui Su, Kefeng Zhang, Man Gao, Xi Su, Xiaodong Cai, Xunliang Cai, Yu Yang, and Yunke Zhao. Vitabench: Benchmarking llm agents with versatile interactive tasks in real-world appli...

  21. [37]

    Mcp-atlas: A large-scale benchmark for tool-use competency with real mcp servers.arXiv preprint arXiv:2602.00933,

    Chaithanya Bandi, Ben Hertzberg, Geobio Boo, Tejas Polakam, Jeff Da, Sami Hassaan, Manasi Sharma, Andrew Park, Ernesto Hernandez, Dan Rambado, et al. Mcp-atlas: A large-scale benchmark for tool-use competency with real mcp servers.arXiv preprint arXiv:2602.00933,

    arXiv

  22. [38]

    Browsecomp: A simple yet challenging benchmark for browsing agents

    26 LongCat Sparse Attention Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516,

    arXiv

  23. [39]

    Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese.arXiv preprint arXiv:2504.19314,

    Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, and Yining Hua. Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese.a...

    arXiv

  24. [40]

    Jasper Dekoninck, Nikola Jovanovi ´c, Tim Gehrunger, Kári Rögnvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev

    URLhttps://github.com/AGI-Eval-Official/RW-Search. Jasper Dekoninck, Nikola Jovanovi ´c, Tim Gehrunger, Kári Rögnvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms.arXiv preprint arXiv:2605.00674,

    arXiv

  25. [41]

    Thang Luong, Dawsen Hwang, Hoang H

    URL https://huggingface.co/datasets/ByteDance-Seed/BeyondAIME. Thang Luong, Dawsen Hwang, Hoang H. Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ah...

  26. [42]

    Deepseek-v4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026b

    Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026b. 27 LongCat Sparse Attention A Tra...

  27. [44]

    Red brackets above the panel mark the key spans of the task description, the target needle line, and the question

    Color is the per-(query, key) attention weight, linearly clipped to [0,10 −3] to reveal weak long-range dependencies (the true maximum is ∼0.26). Red brackets above the panel mark the key spans of the task description, the target needle line, and the question. The question que...

  28. [2017]

    Deepseek-v3.2-exp: Boosting long-context efficiency with deepseek sparse attention.Technical Report, 2025a

    DeepSeek-AI. Deepseek-v3.2-exp: Boosting long-context efficiency with deepseek sparse attention.Technical Report, 2025a. URLhttps://github.com/deepseek-ai/DeepSeek-V3.2-Exp. DeepSeek-AI. Deepseek-v3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:251...

    arXiv

  29. [2018]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi

    doi:10.18653/v1/D18-1259. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. InProceedings of the 61st Annual Meeting of the As...

    doi

  30. [2019]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150,

    2004 arXiv

  31. [2020]

    Retrievalattention: Accelerating long-context llm inference via vector retrieval.arXiv preprint arXiv:2409.10516, 2024a

    Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalattention: Accelerating long-context llm inference via vector retrieval.arXiv preprint arXiv:2409.10516, 2024a. Enzhe Lu, Zhejun Jiang,...

    arXiv

  32. [2022]

    QAMPARI: An open-domain question answering benchmark for questions with many answers from multiple paragraphs.arXiv preprint arXiv:2205.12665,

    Samuel Joseph Amouyal Ohad Rubin, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. QAMPARI: An open-domain question answering benchmark for questions with many answers from multiple paragraphs.arXiv preprint arXiv:2205.12665,

    arXiv

  33. [2023]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al

    doi:10.18653/v1/2023.acl-long.546. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset.arXiv preprint arXiv:1611.09268,

    2023 arXiv

  34. [2024]

    Kascade: A practical sparse attention method for long-context llm inference.arXiv preprint arXiv:2512.16391,

    Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra, and Ramachandran Ramjee. Kascade: A practical sparse attention method for long-context llm inference.arXiv preprint arXiv:2512.16391,

  35. [2025]

    Hysparse: A hybrid sparse attention architecture with oracle token selection and kv cache sharing

    Yizhao Gao, Jianyu Wei, Qihao Zhang, Yu Cheng, Shimao Chen, Zhengju Tang, Zihan Jiang, Yifan Song, Hailin Zhang, Liang Zhao, et al. Hysparse: A hybrid sparse attention architecture with oracle token selection and kv cache sharing. arXiv preprint arXiv:2602.03560,

  36. [2026]

    Efficient streaming language models with attention sinks.International Conference on Learning Representations, 2024a

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.International Conference on Learning Representations, 2024a. Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, ...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.