REVIEW 4 major objections 6 minor 2 cited by
Star Attention: Efficient LLM Inference over Long Sequences
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Star Attention claims that replacing global attention during context encoding with block-local attention plus an anchor block, and letting the query attend globally afterward, cuts long-context LLM inference cost by up to 11x while…
desk verdict A practical block-sparse inference trick that works well for localized tasks, but the 97-100% accuracy claim only holds at modest speedups. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the anchor block: every context block except the first is prefixed with the opening block c1, keeping c1's original positional indices, and each host attends only within its 2b-token augmented block before discarding the anchor's KV entries. This shifts attention sinks from the start of each block to the anchor; blocks without anchors create multiple sinks and fail. The second mechanism is distributed online softmax (log-sum-exp) aggregation: each host returns partial attention outputs and softmax denominators to a designated query host, which renormalizes them into exact global-attention outputs for query and generated tokens while never moving KV caches. Together these mechanisms replace quadratic context attention with linear block-local work while preserving global attention exactly where it matters.
What would settle it
Construct a long-context benchmark whose correct answers require combining two facts placed in blocks far apart with no local cue in either block, then run Star Attention with a small block size and compare against global attention. If accuracy on such a task degrades sharply as the number of blocks grows, the localization premise fails; if it holds flat, the method's block-local encoding is doing more than local retrieval.
Extended reading notes
Core claim
Star Attention's central claim is that exact global attention during the context-encoding phase of inference is unnecessary: the context can be split into blocks, each block can attend only to itself plus a copy of the first block, and the resulting KV cache still supports near-exact query answering when the query attends globally. The anchor block is what makes this work, because it redirects the attention sink that would otherwise appear at the start of every block, so block-local attention distributions approximate global ones. A distributed online-softmax step then aggregates per-host local attentions at a single query host, communicating only one vector and one scalar per token instead of transferring KV caches. The result is linear-time context encoding, global attention for decoding, and empirically 97-100% of baseline accuracy through 128K tokens, with a 16.9x speedup at 1M tokens at a 5.32% accuracy cost.
Load-bearing premise
Everything rests on the claim that the information a query needs is usually localized to a small part of the context; if a task requires combining facts spread across many distant blocks, Star Attention loses accuracy, so the near-parity results are only as general as that localization property.
Editorial extensions
If this is right
- Any transformer LLM trained with global attention can use Star Attention at inference time with no fine-tuning, making the speedup available for already-deployed models.
- Speedups grow with context length and model size: 1.1x-4.7x at 16K-128K tokens, and 10.8x-16.9x at 256K-1M tokens with a fixed 32K block, so the method matters most at the longest contexts.
- Retrieval and aggregation tasks retain near-full accuracy, so long-context use cases like document QA and summarization are the natural target.
- Tasks requiring repeated cross-block information propagation, such as Multi-Hop Tracing, show the largest degradation and mark the boundary of the approach.
- The anchor must carry the real first-block content; a constant-token anchor collapses accuracy to near zero, so the benefit is semantic, not merely positional.
Reading between the lines
- Beyond the paper, the accuracy guarantee is a concentration-of-relevance claim; a stress test that spreads required facts across many distant blocks should show degradation that grows with the number of blocks, independent of total length.
- Beyond the paper, because phase two is exact global attention from the query side, Star Attention should compose with KV-cache compression or eviction methods without changing its structure.
- Beyond the paper, the anchor-content result suggests that a learned or retrieved summary prefix might preserve the sink effect while improving accuracy on documents whose opening block is uninformative.
- Beyond the paper, the reported 16.9x speedup is relative to Ring Attention under fixed GPU counts, so wall-clock gains over non-distributed inference will be smaller at short contexts due to communication overhead.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Star Attention, a training-free, architecture-agnostic two-phase block-sparse attention mechanism for efficient long-context LLM inference. Phase 1 splits the context into blocks, prefixed with an anchor block (the first block), and encodes each block in parallel with block-local attention, discarding the anchor's KV cache. Phase 2 broadcasts the query to all hosts, computes local softmax statistics, and aggregates them exactly via a distributed online-softmax method (Eqs. 3-4), so the query attends globally to all cached tokens. The method is evaluated on RULER, BABILong, and InfiniteBench using Llama-3.1-8B/70B and gradient 262K/1048K variants, reporting 1.1x-16.9x speedups over Ring Attention with accuracy deltas ranging from +1.2% to -7.5%. The paper also ablates the anchor block's position, content, and size, and includes a GitHub code link.
Significance. The two-phase design is elegant and practical: the Phase 2 aggregation is mathematically exact, the method requires no fine-tuning, and it is compatible with standard Transformer-based LLMs. A training-free, blockwise-local context encoding with near-parity accuracy would be a meaningful contribution to long-context serving, especially on multi-GPU setups. The paper also provides reproducible code and evaluates across multiple benchmarks and model scales. The central risk is that the headline claim of 'preserving 97-100% of accuracy' is not supported by the paper's own extreme-speedup results, where accuracy retention falls to roughly 93-95%. The task-localization premise is acknowledged but should be more carefully scoped in the abstract and conclusion.
major comments (4)
- [Abstract; Table 5; Figure 6] The abstract's claim that Star Attention reduces inference time 'while preserving 97-100% of accuracy' is contradicted by the paper's own high-speed results. In Table 5, at 512K and 1M tokens with block size 32K, the accuracy drops relative to Ring Attention are -6.73% and -5.32%, respectively, which corresponds to about 93-95% retention, not 97-100%. Figure 6 shows the same data. The abstract and Section 6 should scoped the 97-100% claim to the regime where it actually holds (e.g., block size set to one-quarter of the sequence length, up to 256K), and present the 512K-1M results separately with their larger accuracy cost.
- [Section 3.5; Figure 7] The paper's own Multi-Hop Tracing result exposes a structural limitation: at 32K sequence length with 8K blocks, Star Attention degrades by 6.52 points relative to global attention (Figure 7). This category requires composing evidence across distant blocks, and Phase 1 (Section 2.1) has no cross-block communication for non-anchor blocks. Consequently, the 97-100% accuracy claim is conditional on the task-localization assumption stated in Section 1, and is not a general long-context result. The paper should report category-wise results for the 512K and 1M settings to show how much the degradation worsens on reasoning-heavy tasks, rather than only the aggregate RULER score.
- [All experimental tables] No repeated runs or error bars are reported for any accuracy or speedup number. Given that Figure 7 shows a +16.15% swing in Aggregation and Appendix D itself attributes the 128K Multi-Hop result to 'noise given the suboptimal baseline,' single-run results are not sufficient to support the precise quantitative claims (e.g., -6.73%, -5.32%). Please report means and standard deviations over at least 3-5 independent runs for the headline accuracy and speedup values, especially for the extreme-speedup configurations in Table 5.
- [Section 2.1; Table 4] The anchor ablation in Table 4 demonstrates that the anchor's content, not merely its structural presence, is essential: a constant-token anchor collapses accuracy to 0%, and random-token anchors drop 9-10 points. This content-specific dependency is a limitation that should be stated prominently, because it constrains the method's applicability to contexts whose opening tokens are informative and representative. The current discussion in Section 4.1 mentions this but the main text's broad claims do not reflect it.
minor comments (6)
- [Algorithm 2] In line 19, the online-softmax update writes 'exp(Ah − sglobal)·Ah'; this should read 'exp(sh − sglobal)·Ah', since Ah is a vector output and exponentiating it is dimensionally incorrect.
- [Section 2.1] The sentence 'each augmented block c′i contains 2b tokens' is inaccurate for i=1, since c′1 = c1 contains b tokens. The clarification should state that only blocks i≥2 are prefixed with the anchor and have 2b tokens.
- [Table 4] The row 'first block tokens' lists the 128K accuracy as 94.94 with a delta of -0.96%, but the Global Attention baseline at 128K is 98.49%; the correct accuracy for a -0.96% delta would be 97.53, so 94.94 appears to be a typographical error.
- [Figure 6] The y-axis label 'Star Attention Speedup (x)' appears twice in the figure; one instance should be removed or relabeled (e.g., left axis for accuracy, right axis for speedup).
- [Section 6] The conclusion repeats the 'up to 11x speedup' figure, while Section 3.4 and Table 5 report 'up to 16.9x'. Please reconcile these numbers by specifying the accuracy regime associated with each speedup.
- [Appendix B.2] Please state explicitly whether Ring Attention and Star Attention are evaluated with identical GPU counts and worker configurations for each speedup row in Table 7. The current table lists resources per sequence length but does not specify per-method settings, leaving the fairness of the speedup comparison ambiguous.
Circularity Check
No significant circularity: measured accuracy and speedup claims, plus the exact distributed-softmax derivation, are independent of the method's own design choices.
full rationale
The paper's derivation chain is self-contained and empirical rather than definitional. Phase 2's global attention aggregation (Equations 1-4) is an exact distributed-softmax identity: each host computes a local attention output and softmax denominator, and the query host reweights by s_h/s_global. This is standard algebra, not a quantity fitted to the benchmark. The headline accuracy and speedup numbers are measured outcomes on RULER, BABILong, and InfiniteBench against external baselines (Ring Attention, StreamingLLM, MInference, and full attention); no parameter is fitted to produce those accuracies. Block size and anchor size are manually chosen hyperparameters, and the paper ablates them (Figure 5, Table 4) rather than treating them as fitted predictions. The one self-citation of RULER (Hsieh et al., 2024) is a publicly available benchmark and therefore independent, externally falsifiable evidence under the rules; it is not a uniqueness theorem or an unverified premise. The paper also explicitly reports the known failure mode of inter-block composition (Figure 7, Multi-Hop Tracing -6.52%), acknowledging the task-locality limitation rather than defining it away. Consequently, no load-bearing step reduces to its own input by construction.
Assumptions & free parameters
free parameters (2)
- context block size =
one-quarter of sequence length for 16K-128K; 32K for longer
- anchor block size =
equal to context block size
assumptions (3)
- domain assumption Query-relevant information in long-context tasks is often localized within small parts of the context.
- domain assumption Pretrained LLMs exhibit attention sinks at sequence start, and prefixing blocks with the first block shifts internal attention sinks to the anchor.
- standard math Online softmax (log-sum-exp) aggregation across partitions yields the exact global softmax over concatenated keys.
Cite this review
Pith. "Pith review of Star Attention: Efficient LLM Inference over Long Sequences." pith.science (2026). https://pith.science/paper/2WVXFXZ2
@misc{pith2026241117116,
author = {Pith},
title = {Pith review of: Star Attention: Efficient LLM Inference over Long Sequences},
year = {2026},
howpublished = {\url{https://pith.science/paper/2WVXFXZ2}},
note = {Machine review of arXiv:2411.17116}
}
read the original abstract
Inference with Transformer-based Large Language Models (LLMs) on long sequences is both costly and slow due to the quadratic complexity of the self-attention mechanism. We introduce Star Attention, a two-phase block-sparse approximation that improves computational efficiency by sharding attention across multiple hosts while minimizing communication overhead. In the first phase, the context is processed using blockwise-local attention across hosts, in parallel. In the second phase, query and response tokens attend to all prior cached tokens through sequence-global attention. Star Attention integrates seamlessly with most Transformer-based LLMs trained with global attention, reducing memory requirements and inference time by up to 11x while preserving 97-100% of accuracy.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 2 Pith papers
-
SCBench: A KV Cache-Centric Analysis of Long-Context Methods
A KV-cache-centric benchmark shows that long-context methods with sub-linear memory degrade sharply in multi-turn and multi-request reuse, while sparse-encoding methods with full memory remain robust.
-
FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models
FlashSVD fuses low-rank SVD projections into attention and feed-forward GPU kernels so SVD-compressed transformers avoid materializing dense activations, cutting activation memory at a latency cost.
Reference graph
Works this paper leans on
-
[2]
Beltagy, I., Peters, M. E., and Cohan, A. Long- 9 Star Attention former: The long-document Transformer.arXiv preprint arXiv:2004.05150,
arXiv 2004
-
[4]
Gemini-Team. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context.arXiv preprint arXiv:2403.05530,
-
[5]
Han, C., Wang, Q., Peng, H., Xiong, W., Chen, Y ., Ji, H., and Wang, S
URL https://gradient.ai/ blog/ruler-vs-gradient-s-1m-context- length-llama-3-70b. Han, C., Wang, Q., Peng, H., Xiong, W., Chen, Y ., Ji, H., and Wang, S. Lm-infinite: Zero-shot extreme length generalization for large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human ...
work page 2024
-
[6]
Kuratov, Y ., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., and Burtsev, M. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a- Haystack.arXiv preprint arXiv:2406.10149,
-
[9]
URL https://ai.meta.com/blog /meta-llama-3-1. Milakov, M. and Gimelshein, N. Online normalizer cal- culation for softmax.arXiv preprint arXiv:1805.02867,
-
[11]
10 Star Attention Qin, Z., Sun, W., Li, D., Shen, X., Sun, W., and Zhong, Y
URL https://github.c om/NVIDIA/TensorRT-LLM. 10 Star Attention Qin, Z., Sun, W., Li, D., Shen, X., Sun, W., and Zhong, Y . Lightning attention-2: A free lunch for handling unlim- ited sequence lengths in large language models.arXiv preprint arXiv:2401.04658,
-
[12]
Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,
Qwen. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,
-
[14]
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LM: Training multi- billion parameter language models using model paral- lelism.arXiv preprint arXiv:1909.08053,
arXiv 1909
Show all 20 references
-
[16]
You only cache once: Decoder-decoder architectures for language models
Sun, Y ., Dong, L., Zhu, Y ., Huang, S., Wang, W., Ma, S., Zhang, Q., Wang, J., and Wei, F. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:2405.05254,
-
[17]
Transformers: State-of- the-art natural language processing
Wolf, T., Debut, L., Sanh, V ., et al. Transformers: State-of- the-art natural language processing. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,
2020
-
[18]
Scope: Optimizing key-value cache compression in long- context generation.arXiv preprint arXiv:2412.13649,
Wu, J., Wang, Z., Zhang, L., Lai, Y ., He, Y ., and Zhou, D. Scope: Optimizing key-value cache compression in long- context generation.arXiv preprint arXiv:2412.13649,
-
[19]
Infllm: Training-free long-context extrapolation for llms with an efficient context memory
Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y ., Zhang, Z., Liu, Z., and Sun, M. Infllm: Training-free long-context extrapolation for llms with an efficient context memory. arXiv preprint arXiv:2402.04617, 2024a. Xiao, G., Tian, Y ., Chen, B., Han, S., and Lewis, M. Ef- ficie...
-
[20]
Vanilla autoregressive generation encounters out-of-memory (OOM) at 128K sequence length
Time per sample (seconds) for Llama3.1-8B-Instruct model with dense, ring, and star attention, using 8 A100 GPUs. Vanilla autoregressive generation encounters out-of-memory (OOM) at 128K sequence length. It performs best in short context scenarios (i.e. sequences upto 32K toke...
2020
-
[2018]
Leave no con- text behind: Efficient infinite context Transformers with infini-attention.arXiv preprint arXiv:2404.07143,
Munkhdalai, T., Faruqui, M., and Gopal, S. Leave no con- text behind: Efficient infinite context Transformers with infini-attention.arXiv preprint arXiv:2404.07143,
-
[2019]
Tree attention: Topology-aware decoding for long-context attention on gpu clusters.arXiv preprint arXiv:2408.04093,
Shyam, V ., Pilault, J., Shepperd, E., Anthony, Q., and Mil- lidge, B. Tree attention: Topology-aware decoding for long-context attention on gpu clusters.arXiv preprint arXiv:2408.04093,
-
[2020]
Gen- erating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,
Child, R., Gray, S., Radford, A., and Sutskever, I. Gen- erating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,
1904 arXiv
-
[2021]
fb.com/2021/07/15/open-source/fsdp/
URL https://engineering. fb.com/2021/07/15/open-source/fsdp/. Meta-AI. Introducing Llama 3.1: Our most capable models to date,
2021
-
[2023]
E2llm: Encoder elongated large language models for long-context understanding and reasoning
Liao, Z., Wang, J., Yu, H., Wei, L., Li, J., Wang, J., and Zhang, W. E2llm: Encoder elongated large language models for long-context understanding and reasoning. arXiv preprint arXiv:2409.06679,
-
[2024]
Behrouz, A., Zhong, P., and Mirrokni, V
URL https://www-cdn.anthropi c.com/de8ba9b01c9ab7cbabf5c33b80b7bb c618857627/Model_Card_Claude_3.pdf. Behrouz, A., Zhong, P., and Mirrokni, V . Titans: Learning to memorize at test time.arXiv preprint arXiv:2501.00663,
-
[2025]
Writing in the margins: Better inference pattern for long context retrieval.arXiv preprint arXiv:2408.14906,
Russak, M., Jamil, U., Bryant, C., Kamble, K., Magnuson, A., Russak, M., and AlShikh, W. Writing in the margins: Better inference pattern for long context retrieval.arXiv preprint arXiv:2408.14906,
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.