Pith. sign in

REVIEW 15 cited by

Sequence Parallelism: Long Sequence Training from System Perspective

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2105.13120 v3 pith:DZ2DMKHA submitted 2021-05-26 cs.LG cs.DC

Shenggui Li , Fuzhao Xue , Chaitanya Baranwal , Yongbin Li , Yang You This is my paper · ORCID
classification cs.LGcs.DC
keywords sequenceparallelismattentionlengthdeviceexistinglongerself-attention
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Transformer achieves promising results on various tasks. However, self-attention suffers from quadratic memory requirements with respect to the sequence length. Existing work focuses on reducing time and space complexity from an algorithm perspective. In this work, we propose sequence parallelism, a memory-efficient parallelism method to help us break input sequence length limitation and train with longer sequences on GPUs efficiently. Our approach is compatible with most existing parallelisms (e.g. data parallelism, pipeline parallelism and tensor parallelism), which means our sequence parallelism makes 4D parallelism possible. More importantly, we no longer require a single device to hold the whole sequence. That is, with sparse attention, our sequence parallelism enables us to train transformer with infinite long sequence. Specifically, we split the input sequence into multiple chunks and feed each chunk into its corresponding device (i.e. GPU). To compute the attention output, we integrated ring-style communication with self-attention calculation and proposed Ring Self-Attention (RSA). Experiments show that sequence parallelism performs well when scaling with batch size and sequence length. Compared with tensor parallelism, our approach achieved $13.7\times$ and $3.0\times$ maximum batch size and sequence length respectively when scaling up to 64 NVIDIA P100 GPUs. With sparse attention, sequence can handle sequence with over 114K tokens, which is over $27\times$ longer than existing sparse attention works holding the whole sequence on a single device.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Serving

    cs.AR 2026-07 conditional novelty 6.0 of 10

    NeuScale routes LLM inference requests to the most energy/cost-efficient configuration of heterogeneous NPU chips using roofline allocation and runtime auto-scaling.

  2. Design-CP: Context Parallelism for Design of Protein Nanoparticles

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Context-parallel inference for RFdiffusion 3 enables end-to-end all-atom design of large symmetric protein nanoparticles on multi-GPU hardware without retraining.

  3. PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint

    cs.LG 2026-07 unverdicted novelty 6.0 of 10

    PHOENIX recovers permanent node failures in LLM training via hot-swapping of spares using zero-overhead per-step in-memory optimizer-state replication, finishing recovery in under 40 s on up to 512 GPUs.

  4. Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking

    cs.LG 2026-02 conditional novelty 6.0 of 10

    UPipe chunks attention by head so QKV and all-to-all buffers scale with a small tunable chunk size rather than head count, enabling 5M-token Llama3-8B training on one 8×H100 node with throughput close to Ulysses.

  5. TetriServe: Efficiently Serving Mixed DiT Workloads

    cs.LG 2025-10 conditional novelty 6.0 of 10

    TetriServe's step-level, deadline-aware sequence parallelism improves SLO attainment for mixed-resolution diffusion transformer serving by up to 32% over fixed-SP systems.

  6. Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference

    cs.LG 2025-08 conditional novelty 6.0 of 10

    An RL agent that co-optimizes parallelism degrees and per-operator sharding dimensions finds distributed inference strategies that beat random search and simulated annealing, and slightly outperform Megatron-LM heuris...

  7. Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs

    cs.AI 2025-06 conditional novelty 6.0 of 10

    Training on more validated GitHub issue-fix trajectories steadily raises SWE-bench Verified pass@1 from 6.4% to 38.0%, with 47.0% after test-time scaling, and no plateau yet.

  8. Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences

    cs.LG 2025-06 conditional novelty 6.0 of 10

    ALST combines tiled loss and MLP compute, Ulysses-style sequence parallelism for Hugging Face GQA and MQA models, and activation-checkpoint CPU offload to train Llama-8B at up to 500K tokens on one H100 and 15M tokens...

  9. Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Enhancing a model's long-context ability before supervised fine-tuning improves math reasoning accuracy, including on short-input benchmarks.

  10. TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

    cs.LG 2025-08 conditional novelty 5.0 of 10

    TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.

  11. Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library

    cs.LG 2025-06 conditional novelty 5.0 of 10

    ROLL is a distributed RL training library for LLMs with sample-level rollout scheduling and flexible device mapping, demonstrated on RLVR and agentic tasks but without baseline or scaling measurements.

  12. ContentV: Efficient Training of Video Generation Models with Limited Compute

    cs.CV 2025-06 conditional novelty 5.0 of 10

    An 8B text-to-video model built by adapting Stable Diffusion 3.5 with a 3D video autoencoder reaches near-leading VBench scores after four weeks of NPU training.

  13. StreamBP: Memory-Efficient Exact Backpropagation for Long Sequence Training of LLMs

    cs.LG 2025-06 conditional novelty 5.0 of 10

    StreamBP partitions backpropagation over the sequence dimension to compute exact gradients with much lower activation memory, extending maximum trainable sequence length by roughly 3-5x over gradient checkpointing.

  14. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.

  15. CoDec: Prefix-Shared Decoding Kernel for LLMs

    cs.LG 2025-05 conditional novelty 5.0 of 10

    CoDec combines KV-cache reads across requests that share a prefix, yielding average 1.9x decode-attention speedup and 120.9x less global memory traffic versus FlashDecoding.

Pith tools