Pith. sign in

REVIEW 6 cited by

Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.15241 v1 pith:MEO5455B submitted 2024-09-23 cs.DC cs.AIcs.LG

Guanhua Wang , Chengming Zhang , Zheyu Shen , Ang Li , Olatunji Ruwase This is my paper · ORCID
classification cs.DCcs.AIcs.LG
keywords trainingcommunicationdominogenericcomputationgpusindependentllms
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Given the popularity of generative AI, Large Language Models (LLMs) often consume hundreds or thousands of GPUs for parallelizing and accelerating the training process. Communication overhead becomes more pronounced when training LLMs at scale. To eliminate communication overhead in distributed LLM training, we propose Domino, which provides a generic scheme to hide communication behind computation. By breaking data dependency of a single batch training into smaller independent pieces, Domino pipelines these independent pieces training and provides generic strategy of fine-grained communication and computation overlapping. Extensive results show that, comparing with Megatron-LM, Domino achieves up to 1.3x speedup for LLM training on Nvidia DGX-H100 GPUs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

    cs.DC 2026-07 conditional novelty 6.0 of 10

    A tile-level signaling and scheduling co-design overlaps MoE expert GEMM with the return all-to-all, achieving up to 2.64x end-to-end speedup on 4 A100s.

  2. Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Training

    cs.LG 2026-01 conditional novelty 6.0 of 10

    Kareus jointly tunes kernel timing, GPU core allocation, and frequency to expand the time–energy tradeoff frontier for training large language models.

  3. Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap

    cs.DC 2025-12 conditional novelty 6.0 of 10

    Finer-grain, DMA-offloaded compute-communication overlap with schedule heuristics yields up to 1.6x speedups for distributed ML on full-mesh GPU systems.

  4. Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference

    cs.AI 2024-12 conditional novelty 6.0 of 10

    A quantized two-step all-reduce kernel reduces tensor-parallel communication overhead in LLM inference, achieving up to 3.18x faster all-reduce and 2.06x TTFT speedup on L40 GPUs.

  5. Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution

    cs.DC 2024-11 conditional novelty 6.0 of 10

    DHelix interleaves two micro-batches at operator granularity and folds the pipeline into a W-shape, hiding communication and improving LLM training throughput by 12-40% on tested clusters.

  6. Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications

    cs.DC 2025-07 conditional novelty 5.0 of 10

    Overlapping computation and communication in distributed GPU training slows compute kernels by up to 40% and raises power use, while remaining faster than sequential execution.

Pith tools