Pith. sign in

REVIEW 4 cited by

torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2004.09910 v1 pith:5AXI7AXP submitted 2020-04-21 cs.DC cs.LG

Chiheon Kim , Heungsub Lee , Myungryong Jeong , Woonhyuk Baek , Boogeon Yoon , Ildoo Kim , Sungbin Lim , Sungwoong Kim This is my paper · ORCID
classification cs.DCcs.LG
keywords libraryparallelismpipelinedesignenvironmentpytorchtorchgpipeamoebanet-d
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We design and implement a ready-to-use library in PyTorch for performing micro-batch pipeline parallelism with checkpointing proposed by GPipe (Huang et al., 2019). In particular, we develop a set of design components to enable pipeline-parallel gradient computation in PyTorch's define-by-run and eager execution environment. We show that each component is necessary to fully benefit from pipeline parallelism in such environment, and demonstrate the efficiency of the library by applying it to various network architectures including AmoebaNet-D and U-Net. Our library is available at https://github.com/kakaobrain/torchgpipe .

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Distributed Deep Learning using Stochastic Gradient Staleness

    cs.LG 2025-09 reject novelty 5.0 of 10

    A hybrid data- and model-parallel training scheme using stale gradients and consensus averaging, claimed to converge to critical points and speed up ResNet-20 training on CIFAR-10.

  2. FlashDP: Private Training Large Language Models with Efficient DP-SGD

    cs.LG 2025-07 conditional novelty 5.0 of 10

    FlashDP fuses per-sample gradient computation, norm calculation, clipping, and noise addition into a cache-friendly block-wise all-reduce workflow that avoids explicit per-sample gradient storage and redundant recomputation.

  3. ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling

    cs.LG 2025-05 conditional novelty 5.0 of 10

    A 10-billion-parameter climate downscaling model trained on 65,536 GPUs at up to 4.1 exaFLOPS, with R2 0.98 to 0.99 at 7 km, but with token-count and validation caveats.

  4. Modular Federated Learning: A Meta-Framework Perspective

    cs.LG 2025-05 conditional novelty 3.0 of 10

    A 63-page survey that reframes federated learning as a composition of eight modules and proposes an 'alignment operator' taxonomy, while surveying Python FL frameworks and open challenges.

Pith tools