REVIEW 6 cited by
LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Efficiently training LLMs with long sequences is important yet challenged by the massive computation and memory requirements. Sequence parallelism has been proposed to tackle these problems, but existing methods suffer from scalability or efficiency issues. We propose LoongTrain, a novel system to efficiently train LLMs with long sequences at scale. The core of LoongTrain is the 2D-Attention mechanism, which combines both head-parallel and context-parallel techniques to break the scalability constraints while maintaining efficiency. We introduce Double-Ring-Attention and analyze the performance of device placement strategies to further speed up training. We implement LoongTrain with the hybrid ZeRO and Selective Checkpoint++ techniques. Experiment results show that LoongTrain outperforms state-of-the-art baselines, i.e., DeepSpeed-Ulysses and Megatron Context Parallelism, in both end-to-end training speed and scalability, and improves Model FLOPs Utilization (MFU) by up to 2.88x.
Forward citations
Cited by 6 Pith papers
-
Syncopate: Efficient Multi-GPU AI Kernels via Automatic Chunk-Centric Compute-Communication Overlap
Syncopate automatically overlaps compute and communication at fine chunk granularity inside a single fused Triton kernel, yielding 1.3x average and up to 4.7x end-to-end speedup on multi-GPU workloads.
-
StateFlow: Sequence Pipeline Parallelism for Long-Context Modeling with Linear Recurrence
StateFlow is a sequence pipeline parallelism system that propagates boundary states across sequence chunks to reduce activation memory and pipeline bubbles when training linear-recurrence and hybrid models.
-
Libra: Taming Attention Workload Skew in Long-Context LLM Training with Bounded Sequence Pool
Libra bounds attention load-balancing to fixed-size sequence pools and combines variance-reduced sequence placement with tiled attention pooling, achieving up to 2.54x end-to-end throughput over Ulysses on 256K- and 1...
-
DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training
DASH speeds up deterministic attention backprop by reversed and shift-based tile schedules, giving up to 1.28× throughput and ~5% end-to-end gains while keeping bitwise reproducibility.
-
Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences
ALST combines tiled loss and MLP compute, Ulysses-style sequence parallelism for Hugging Face GQA and MQA models, and activation-checkpoint CPU offload to train Llama-8B at up to 500K tokens on one H100 and 15M tokens...
-
TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication
TokenRing claims faster long-sequence attention on multiple GPUs by overlapping forward query transfers with backward partial-output transfers, supported by one 4-GPU profiling comparison.
Discussion (0). Continue with ORCID to comment.