REVIEW 3 cited by
Context Parallelism for Scalable Million-Token Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present context parallelism for long-context large language model inference, which achieves near-linear scaling for long-context prefill latency with up to 128 H100 GPUs across 16 nodes. Particularly, our method achieves 1M context prefill with Llama3 405B model in 77s (93% parallelization efficiency, 63% FLOPS utilization) and 128K context prefill in 3.8s. We develop two lossless exact ring attention variants: pass-KV and pass-Q to cover a wide range of use cases with the state-of-the-art performance: full prefill, persistent KV prefill and decode. Benchmarks on H100 GPU hosts inter-connected with RDMA and TCP both show similar scalability for long-context prefill, demonstrating that our method scales well using common commercial data center with medium-to-low inter-host bandwidth.
Forward citations
Cited by 3 Pith papers
-
Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models
NAtS-L learns per-chunk routing between Gated DeltaNet and softmax attention, improving long-context retrieval and length extrapolation over fixed hybrids.
-
Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap
Finer-grain, DMA-offloaded compute-communication overlap with schedule heuristics yields up to 1.6x speedups for distributed ML on full-mesh GPU systems.
-
CoDec: Prefix-Shared Decoding Kernel for LLMs
CoDec combines KV-cache reads across requests that share a prefix, yielding average 1.9x decode-attention speedup and 120.9x less global memory traffic versus FlashDecoding.
Discussion (0). Continue with ORCID to comment.