Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Helix Parallelism claims that decoupling attention and FFN sharding across the same GPUs cuts token-to-token latency by up to 1.5x and supports up to 32x larger batches for multi-million-token decoding while preserving exact attention…

desk verdict A genuinely novel sharding scheme with a solid conceptual core, but the headline gains rest on an undisclosed simulator until the authors show real hardware or release the code. read the letter →

arxiv 2507.07120 v1 pith:AGLF3HR2 submitted 2025-07-07 cs.DC cs.AI

Nidhi Bhatia , Ankit More , Ritika Borkar , Tiyasa Mitra , Ramon Matas , Ritchie Zhao , Maximilian Golub , Dheevatsa Mudigere
show 2 more authors
Brian Pharris Bita Darvish Rouhani
This is my paper · ORCID
classification cs.DCcs.AI
keywords HelixParallelismLLMinferenceKVcacheshardingtensorexperttoken-to-tokenlatencylongcontextmixture-of-experts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Helix Parallelism is a proposed GPU-sharding strategy for autoregressive LLM decoding with very long KV histories. The paper's central claim is that the two biggest decoding bottlenecks—reading FFN weights and reading long KV caches—cannot both be solved by a single sharding layout, so Helix decouples them: it applies KV-parallel sharding during attention, then reuses the same GPUs for tensor parallelism (or expert parallelism in MoE models) during the FFN. A lightweight All-to-All step with log-sum-exp rescaling preserves exact softmax attention, and a batch-wise overlap scheme called HOP-B hides that communication behind compute. If right, this shifts the throughput-latency Pareto frontier: up to 1.5x lower token-to-token latency at fixed batch size and up to 32x larger batches under the same latency budget for DeepSeek-R1, with 1.13x interactivity and 4x throughput gains on Llama-405B. The authors care because interactive applications with million-token contexts need both low per-token latency and high GPU efficiency.

What carries the argument

The central mechanism is a per-layer temporal pipeline that reconfigures the same N GPUs between two layouts: attention runs as KVP × TPA (KV cache sharded over sequence, query heads split up to the number of KV heads), and the FFN runs as TPF × EP (dense TP or MoE expert-parallel TP). Exact-attention preservation comes from a single All-to-All over the query-head axis in which each GPU exchanges partial attention outputs and log-sum-exp scalars, then rescales and sums them to reconstruct the full softmax output—the same rescale step used in Flash-Decoding. Helix HOP-B is a batchwise overlap schedule that starts the All-to-All for one request's token while the next request's attention compute is still running, so communication latency is hidden behind computation and token-to-token latency stays low.

What would settle it

Run the same configuration sweep on a physical GB200 NVL72 node with FP4 weights for DeepSeek-R1 and Llama-405B at a 1M-token KV history. If the measured token-to-token latency reduction at fixed batch size is materially below 1.5x (for DeepSeek-R1) or 1.13x (for Llama-405B), or if the batch size sustained under the stated latency budget is materially below 32x or 4x, respectively, the paper's quantitative claims would be falsified.

Watch

Extended reading notes

Core claim

The core discovery is that the attention and FFN stages of a transformer layer should not be forced to share a parallelism layout. Helix partitions the GPU pool as KVP × TPA for attention, where KVP shards the KV cache along the sequence dimension and TPA splits query heads only up to the number of KV heads, avoiding duplicate KV copies. After a single All-to-All along the query-head axis and per-token log-sum-exp rescaling, the same N GPUs are reconfigured as TPF × EP for the FFN—dense TP for dense models, TP × expert parallelism for MoEs—and a TP All-Reduce (plus inter-expert All-Gather in MoEs) produces the layer output. The paper claims this yields zero-downtime pipelining, sublinear attention read cost with sequence length, and the stated 1.5x/32x (DeepSeek-R1) and 1.13x/4x (Llama-405B) gains over tensor, pipeline, expert, and vanilla KV-parallel baselines.

Load-bearing premise

The quantitative claims rest on an in-house simulator of GB200 hardware whose parameters and validation against real chips are not disclosed; if the simulator's assumptions about DRAM bandwidth, NVLink latency, FP4 throughput, or communication-computation overlap are optimistic, the 1.5x and 32x figures would shrink.

Editorial extensions

If this is right

  • With Helix, KV cache reads scale sublinearly in sequence length because the cache is sharded across KVP ranks, so million-token contexts no longer force tiny batch sizes.
  • FFN weight reads stop being a hard latency floor: all N GPUs participate in FFN sharding even when the model has very few KV heads, as in MLA-based MoEs.
  • HOP-B's value is architecture-dependent: it recovers about 12% interactivity for Llama-405B but only about 1% for DeepSeek-R1, because the All-to-All is a tiny fraction of R1's decode time.
  • Helix works with both GQA and MLA attention and with dense and MoE FFNs, giving the method broad applicability across model families.
  • For DeepSeek-R1 at 1M context, Helix claims up to 1.5x TTL reduction and up to 32x larger batch under the same latency budget; for Llama-405B it claims 1.13x interactivity and 4x throughput versus tensor parallelism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The decoupling principle should transfer to sparse attention, as the paper's future-work section suggests, but HOP-B's overlap gain would likely shrink because sparse attention lowers the compute time available to mask communication; KV capacity, not read bandwidth, would then be the binding constraint.
  • Inference: Helix's gains probably depend on the ratio of KV cache size to FFN parameter size; models with unusually small FFNs or unusually wide KV heads would likely see smaller benefits.
  • Inference: A direct hardware validation on a real GB200-class node with FP4 would be the natural next test; if the simulator's communication-computation overlap model is optimistic, the real gains could be lower, though the qualitative Pareto improvement would likely survive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. Helix Parallelism proposes a hybrid sharding strategy for low-latency autoregressive decoding of multi-million-token contexts. The idea is to decouple the parallelism used in the attention phase from that used in the FFN phase: attention uses KV parallelism (KVP) across the sequence dimension combined with tensor parallelism (TP) across heads, while the same GPUs are reconfigured for dense TP or TP x expert-parallel FFNs. Exact attention is preserved through a log-sum-exp rescaling step exchanged via a single all-to-all, and the paper introduces HOP-B, a batch-wise communication-computation overlap. The paper claims up to 1.5x lower token-to-token latency (TTL) at fixed batch size and up to 32x larger batch under the same TTL budget for DeepSeek-R1, and 1.13x TTL improvement with 4x batch capacity for Llama-405B. The evaluation is performed exclusively with an in-house GB200 simulator over 100,000 configurations, with a roofline appendix providing DRAM-read formulas for Figure 1.

Significance. If the quantitative results are correct, Helix would be a useful contribution to long-context LLM serving: the attention rescaling via log-sum-exp is an exact identity, the communication volume is independent of sequence length, and the idea of decoupling attention and FFN sharding is conceptually sound and goes beyond tying TP width to the number of KV heads. The roofline analysis in Figure 1 and Appendix A is a helpful sanity check for the memory-bound regime. However, the paper's headline numbers rest entirely on an undisclosed, unvalidated in-house simulator, and the analytical appendix deliberately excludes communication, so it does not substantiate the claimed Pareto improvements. The contribution is therefore significant in principle but currently conditional on validation that is not provided.

major comments (5)
  1. [Section 3.1, Figures 5-6] The central quantitative claims—1.5x TTL reduction and 32x larger batch for DeepSeek-R1, 1.13x/4x for Llama-405B—are based solely on an 'in-house high-fidelity simulator modeling the latest GB200 hardware' (Section 3.1). The simulator is not released, its parameters (DRAM bandwidth, NVLink latency, FP4 throughput, overlap model) are not specified, and no validation against real GB200 measurements is reported. The roofline appendix (Appendix A) only covers DRAM reads and explicitly does not include communication, so it cannot validate the simulator's communication model. Since any optimistic assumption in the simulator can materially change the reported Pareto frontier, the evaluation must be backed by a released simulator, detailed parameter disclosure, real-hardware measurements, or at minimum a thorough sensitivity analysis.
  2. [Section 2.1.3, Figure 3] HOP-B relies on overlapping one request's all-to-all communication with another request's attention compute in the batch dimension. This means the claimed TTL hiding is contingent on having a sufficiently large batch or multiple concurrent requests; for small batch sizes or a single interactive user, the communication would be exposed. The paper does not report TTL for small batch sizes or study how the benefit degrades as batch shrinks. Given that the paper targets interactive single-user TTL, this is a load-bearing point that should be analyzed explicitly rather than only through the aggregate Pareto curves.
  3. [Section 2.1.1, Section 2.2] In the attention phase, each KVP GPU independently computes the full QKV projections (Section 2.1.1), which means QKV weight reads and projection FLOPs are replicated across KVP ranks. The paper's roofline formulas in Appendix A do include QKV weight reads in the weight-read term, but they assume a per-rank shard of size Q/TPA; it is unclear how the full-replication cost for all KVP ranks is accounted for in the simulator. Since FFN weight reads are said to dominate, replication may be acceptable, but the paper should state explicitly how this replicated work is modeled and provide a sensitivity check; otherwise the claimed gains could be overstated.
  4. [Section 2.2, Figure 4] The paper repeatedly claims that Helix enables 'zero-downtime pipelining' and that the same GPUs are 'seamlessly' reconfigured from KVP x TPA to TPF x EP on a per-layer basis. No scheduler, runtime, or implementation detail is provided to support this claim, and the simulator's treatment of layout-switch overhead is not described. If reconfiguration involves even a small synchronization or memory-layout conversion cost, the TTL numbers could be affected. This should be either demonstrated in an implementation or explicitly modeled in the simulator.
  5. [Section 2.3] The distributed KV concatenation strategy staggers updates across KVP ranks in chunks of 16 tokens (Section 2.3). This chunk size is an ad-hoc parameter, and the paper provides no sensitivity analysis with respect to it. While this is unlikely to overturn the main conclusions, the paper's claim of balanced memory growth should be supported by showing how the variance in per-GPU KV size and any associated load imbalance depend on the chunk size and on the number of KVP ranks.
minor comments (6)
  1. [Figure 3 caption] The caption says each request in the bottom pipeline consumes 2 units of compute and 1.2 units of communication, while the top pipeline shows 16 units of attention per request; the relationship between these numbers is not explained and the units appear inconsistent.
  2. [Appendix A] The notation in the roofline formulas is confusing: 'Q' is used both for the number of query heads and the total query output dimension, and 'K' is used for the number of KV heads while also being the conventional symbol for the key matrix. Please define all symbols with unambiguous names, e.g., N_q and N_kv.
  3. [Section 3.1] The paper states that 'all performance numbers are normalized to that of the baseline'; this normalization can hide absolute simulator inaccuracies, so reporting at least one representative absolute latency or throughput figure would be useful.
  4. [Section 3.2] For Llama-405B, the comparison with Medha says that 'Medha systems expose all communication overheads' while both Helix and the baseline TP implementation include communication-computation overlap, but the nature of the baseline's overlap is not described. Please clarify what overlap mechanism the baseline uses.
  5. [Section 2.1.1] The exact attention reconstruction via log-sum-exp rescaling is described correctly, but the paper does not discuss numerical precision effects under FP4 storage or arithmetic; a short note on numerical stability would be appropriate.
  6. [Section 5] The statement that in the short-context regime Helix 'simplifies to data-parallel attention and tensor-parallel FFN, a pattern already widely used' sits somewhat in tension with the paper's novelty claim in Section 4; consider clarifying that Helix's contribution is the unified decoupling framework rather than the short-context instance.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's exact-attention step is a standard mathematical identity, the roofline formulas are conventional motivation, and the quantitative claims rest on an unvalidated in-house simulator, which is an evidence issue rather than a derivation loop.

full rationale

Helix's derivation chain is self-contained in the sense relevant to circularity. The exact-attention reconstruction is the standard online-softmax/log-sum-exp identity, cited to Flash-Decoding [10]; it is a mathematical identity, not a fitted or predicted quantity. The Appendix A roofline formulas are conventional DRAM-read expressions used only to motivate the design of Figure 1 and explicitly exclude communication, so they do not by themselves generate the 1.5x/32x evaluation claims. The evaluation is produced by an in-house GB200 simulator (Section 3.1) whose implementation, parameters, and validation against real hardware are undisclosed; this is a serious external-validity and reproducibility limitation, but it is not circularity because no evaluation result is defined in terms of the claim it supports, and no fitted parameter is renamed as a prediction. The only self-citation is [11] for FP4, which is independent support for a published numeric data format and is not load-bearing in a circular sense. No circular step can be exhibited from the paper's own equations or citations, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an unvalidated in-house simulator, FP4 precision assumptions, and simulated million-token contexts rather than measured hardware results. There are no fitted numeric constants in the analytic roofline, only hardware and configuration assumptions.

free parameters (1)
  • KV update stagger chunk (16 tokens)
    Chosen by hand in Section 2.3; affects KV load balance across ranks but is not used to fit the headline performance claims.
assumptions (4)
  • domain assumption GB200 NVL72 simulator fidelity
    Section 3.1: all results come from an in-house simulator with no validation against real hardware; the central quantitative claims depend on it.
  • domain assumption FP4 precision for all weights, KV states, and arithmetic
    Section 3.1 and Appendix A: results are specifically for FP4 deployment, not standard BF16, which affects both memory traffic and compute throughput.
  • domain assumption Simulated million-token KV caches for models that do not natively support them
    Section 3.1: Llama-405B and DeepSeek-R1 do not natively support million-token contexts; the authors simulate decode with such KV cache lengths.
  • standard math Softmax rescaling via log-sum-exp is exact
    Section 2.1.1: partial attention outputs combined with log-sum-exp statistics reproduce exact softmax attention; this is a standard identity and not a weakness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding." pith.science (2026). https://pith.science/paper/AGLF3HR2

@misc{pith2026250707120,
  author       = {Pith},
  title        = {Pith review of: Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AGLF3HR2}},
  note         = {Machine review of arXiv:2507.07120}
}
read the original abstract

As LLMs scale to multi-million-token KV histories, real-time autoregressive decoding under tight Token-to-Token Latency (TTL) constraints faces growing pressure. Two core bottlenecks dominate: accessing Feed-Forward Network (FFN) weights and reading long KV caches. While Tensor Parallelism (TP) helps mitigate the cost of FFN weight reads, it does not scale well for attention. When TP width exceeds the number of KV heads, it leads to inefficient KV duplication, limits parallelism, and constrains batch size. Simultaneously, DRAM reads for long KV histories scale linearly with batch size, further capping efficiency. We introduce Helix Parallelism, a hybrid execution strategy that applies KV parallelism during attention to shard KV caches across GPUs, then reuses the same GPUs for TP in dense LLMs or TPxExpert Parallel (EP) in MoEs during FFN computation. To preserve exact attention behavior, Helix includes a lightweight communication step. To minimize the exposed communication cost, we introduce Helix HOP-B. Helix HOP-B effectively minimizes communication overhead through batchwise overlap, preserving low TTL while improving GPU efficiency. Compared to conventional parallelism approaches, Helix reduces TTL by up to 1.5x at fixed batch sizes and supports up to 32x larger batches under the same latency budget for DeepSeek-R1, pushing forward the throughput-latency Pareto on Blackwell and making real-time inference with ultra-long-sequence practical.

Figures

Figures reproduced from arXiv: 2507.07120 by the authors.

Figure 1
Figure 1. Roofline analysis for KV cache and Linear weight reads, assuming a Dense LLM with batch B=8, Query heads Q=128, KV heads K=8, head size Hsz=128, and FFN dimension F=65536 running on GB200 NVL72. Both weights and KV cache, are stored and fetched in FP4. Communication overhead from TP and KVP is not included; these plots show only the change in GPU DRAM-read latency as TP width and KVP width vary. Details can be found… view at source ↗
Figure 2
Figure 2. Overview of different attention sharding strategies. Here we are using GQA as an example with Query heads Q=4 and KV heads K=2. Each block processes one token with context length S. (Left) No TP: all Q and KV heads are co-located on a single GPU; no duplication. (Middle￾Left) TP=2: Query heads are split across 2 GPUs; KV heads are still partitioned cleanly since T P ≤ K. (Middle-Right) TP=4: More shards than KV head… view at source ↗
Figure 3
Figure 3. KVP All-to-All exposed time: (Top without HOP-B) All 8 requests execute in lockstep, each consumes 16 time units of attention before initiating 9.6 units of communication, for a total span of 25.6 units with no overlap; (Bottom with HOP-B) Requests are pipelined so that while one request’s communication is ongoing, the next request begins its attention compute immediately. Here each request’s compute (2 units) and c… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Helix’s layout per-GPU workflow in attention and FFN stages. Helix reuses the same pool of N GPUs by configuring N = KV P × T P A (T P A ≤ K) −→ N = T P F × EP on a per-layer basis. (Top) During attention, each of the KV P GPUs independently projects the full batch [B,…
Figure 5
Figure 5. Figure 5: Pareto frontier of serving DeepSeek-R1 with 1-Million context length on GB200 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Pareto frontier of serving Llama-405B with 1-Million context length on GB200 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Pareto frontier of HOP-B ON vs. HOP-B OFF with 1-Million context length on GB200. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 1
Figure 1. Figure 1: B : Batch Size Q : Q heads K : KV heads Hsz : Attention Head Size S : KV Sequence length H : Hidden dimension = Q × Hsz F : Intermediate Hidden dimension T P A : T P width for Attention T P F : T P width for F F N KV P : KV P width bytesparam : Bytes per parameter MemB…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix

    cs.DC 2026-07 conditional novelty 6.0 of 10

    LAGA replaces MLA training's per-head K/V all-to-all with a latent all-gather and local up-projection, matching explicit-form memory while cutting collective communication ~1.98x.

  2. Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference

    cs.LG 2025-08 conditional novelty 6.0 of 10

    An RL agent that co-optimizes parallelism degrees and per-operator sharding dimensions finds distributed inference strategies that beat random search and simulated annealing, and slightly outperform Megatron-LM heuris...

Reference graph

Works this paper leans on

20 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, Dec 2024

    Gemini Team Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, Dec 2024

    work page 2024

  2. [2]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, Apr 2025

    Meta Llama 4 Team. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, Apr 2025

    work page 2025

  3. [3]

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, 2023

    work page 2023

  4. [4]

    Fast Transformer Decoding: One Write-Head is All You Need

    Noam Shazeer. Fast Transformer Decoding: One Write-Head is All You Need. Blog post, 2019. https://noam.github.io/2019/09/18/fast-transformer-decoding.html

    work page 2019

  5. [5]

    Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian L...

    work page 2024

  6. [6]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, 2020

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, 2020

    work page 2020

  7. [7]

    Medha: Efficiently Serving Multi-Million Context Length LLM Inference Requests Without Approximations, 2025

    Amey Agrawal, Haoran Qiu, Junda Chen, Íñigo Goiri, Ramachandran Ramjee, Chaojie Zhang, Alexey Tumanov, and Esha Choukse. Medha: Efficiently Serving Multi-Million Context Length LLM Inference Requests Without Approximations, 2025

    work page 2025

  8. [8]

    Nvidia blackwell architecture technical brief, 2024

    NVIDIA. Nvidia blackwell architecture technical brief, 2024. NVIDIA Technical Documenta- tion

    work page 2024

Show all 20 references
  1. [9]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024

    2024

  2. [10]

    Flash-Decoding for long-context inference

    Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-Decoding for long-context inference. https://crfm.stanford.edu/2023/10/12/flashdecoding.html, 2023

    2023

  3. [11]

    Microscaling data formats for deep learning, 2023

    Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Sum- mer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gaurav Kolhe, ...

    2023

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, A...

    2024

  5. [13]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2025

  6. [14]

    Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism, 2024

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism, 2024

    2024

  7. [15]

    Usp: A unified sequence parallelism approach for long context generative ai, 2024

    Jiarui Fang and Shangchun Zhao. Usp: A unified sequence parallelism approach for long context generative ai, 2024

    2024

  8. [16]

    Xing, Xuezhe Ma, Ion Stoica, Joseph E

    Dacheng Li, Rulin Shao, Anze Xie, Eric P. Xing, Xuezhe Ma, Ion Stoica, Joseph E. Gonzalez, and Hao Zhang. Distflashattn: Distributed memory-efficient attention for long-context llms training, 2024

    2024

  9. [17]

    Context parallelism for scalable million-token inference, 2025

    Amy Yang, Jingyi Yang, Aya Ibrahim, Xinfeng Xie, Bangsheng Tang, Grigory Sizov, Jeremy Reizenstein, Jongsoo Park, and Jianyu Huang. Context parallelism for scalable million-token inference, 2025

    2025

  10. [18]

    https://github.com/NVIDIA/TensorRT-LLM

    Nvidia tensorrt-llm. https://github.com/NVIDIA/TensorRT-LLM

  11. [19]

    https://lmsys.org/blog/ 2024-12-04-sglang-v0-4/#data-parallelism-attention-for-deepseek-models

    Data parallelism attention for deepseek models. https://lmsys.org/blog/ 2024-12-04-sglang-v0-4/#data-parallelism-attention-for-deepseek-models

    2024

  12. [20]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y . X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attentio...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.