Pith. sign in

REVIEW 2 major objections 5 minor 53 references

Adaptive Space-efficient Collectives for Dynamic and Unstructured Sparsity on GPU Platforms

T0 review · 2 major / 5 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A low-overhead bitvector format and adaptive ring algorithms cut GPU collective communication volume for unstructured sparse data, delivering multi-fold speedups over dense NCCL at high sparsity.

desk verdict Solid systems paper: new low-overhead sparse format plus adaptive ring collectives that deliver real, measured speedups on modern GPUs for unstructured medium-to-high sparsity. read the letter →

arxiv 2607.04676 v1 pith:RGZJGFOX submitted 2026-07-06 cs.DC

Lannie Dalton Hough , Emir Gencer , Hoffmann Muki , Abhinav Bhatele This is my paper · ORCID
classification cs.DC
keywords sparsecollectivesGPUcommunicationall-gatherreduce-scatterall-reducebitvectorformatPiciunstructuredsparsity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When data on GPUs is sparse—full of zeros—standard collective libraries still ship every element, wasting bandwidth. This paper shows that a new bitvector format called Pici, combined with ring algorithms that can switch between sparse and dense representations mid-collective, lets all-gather, reduce-scatter, and all-reduce send only nonzeros while keeping conversion costs low enough that the savings are real. The adaptation is driven by measured sparsity, network topology (intra-node NVLink versus inter-node fabric), and the phase of the collective, so that densification during reductions does not erase the gains. On a modern GPU cluster the resulting library, SpCCL, is up to five times faster than dense NCCL at 99 percent sparsity and still competitive down to moderate sparsity; the same primitives accelerate end-to-end gradient-pruned language-model training by 13–26 percent. A sympathetic reader cares because communication is already a dominant cost at scale, and many HPC and ML workloads produce exactly the unstructured, medium-to-high sparsity that prior sparse-collective work ignored.

What carries the argument

Pici—a flat-index, 64 imes64-tile bitvector format with only 3.15 percent space overhead relative to dense fp32 data—plus CUDA warp-cooperative (de)compression kernels and an adaptive ring algorithm that re-chooses sparse versus dense representation after every ring step according to measured sparsity and link type.

What would settle it

Re-run the same 512–2048 MiB microbenchmarks and the gradient-pruned DDP training workload on a different GPU cluster (or with structured rather than uniform sparsity) while freezing the published thresholds; if the reported speedups disappear or reverse, the claim fails.

Watch

Extended reading notes

Core claim

At 99 percent input sparsity the authors’ adaptive SpCCL implementations of all-gather, reduce-scatter and all-reduce achieve up to 5.25 imes, 2.5 imes and 2.66 imes speedups over dense NCCL on Perlmutter, with the same primitives yielding 13–26 percent faster end-to-end iterations for gradient-pruned 1.5 B–3.3 B language models; the gains come from a 3.15 percent-overhead bitvector format (Pici) and from topology- and phase-aware decisions that keep (de)compression costs from canceling the reduction in message volume.

Load-bearing premise

The densification formula and the three empirically tuned thresholds remain near-optimal for sparsity patterns and network topologies that differ from the uniform-random, Slingshot-plus-NVLink setting used to choose them.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents SpCCL, a set of sparsity-aware GPU collectives (all-gather, reduce-scatter, all-reduce) built as an extension of NCCLX. It introduces Pici, a bitvector-based sparse format with ~3.15% overhead relative to dense fp32 data and warp-cooperative CUDA (de)compression kernels optimized for moderate unstructured sparsity. Algorithms adaptively switch between sparse (Pici or COO) and dense representations during ring steps using topology-aware (inter/intra-node) and phase-aware thresholds to handle fill-in densification. Microbenchmarks on Perlmutter (up to 64 A100s) report up to 5.25 imes/2.5 imes/2.66 imes speedups over dense NCCL at 99% sparsity for AG/RS/AR, with end-to-end gains of 13–26% on gradient-pruned DDP training of 1.5B–3.3B models.

Significance. If the reported speedups hold under the evaluated conditions, the work is a solid systems contribution to sparse collective communication for modern GPU clusters and distributed DL. Strengths include a carefully engineered low-overhead format (Pici) with fused kernels, reuse of a production-quality dense baseline (NCCLX) rather than a from-scratch implementation, multi-job averaging with min/max bars, channel/threshold sweeps, and a realistic pruned-LLM case study. The adaptive densification handling and explicit comparison to both dense NCCL/NCCLX and prior sparse work (SparCML) make the empirical claims more credible than many prior sparse-collective papers. The free parameters (thresholds, channel count) are acknowledged and swept; they do not circularly define the headline numbers.

major comments (2)
  1. §IV-C and Eq. (7): The densification extrapolation δ_next = 1−(1−δ_prev)(1−δ_0) and the fixed thresholds (inter_thresh=0.5, intra_thresh=0.6, ag_thresh=0.1) are derived under the assumption of uniform independent random sparsity. While Fig. 2 shows that poorly chosen thresholds hurt, the paper never evaluates non-uniform or structured sparsity patterns that arise in real pruned models or scientific sparse matrices. This is the main load-bearing generality gap for the adaptive algorithm claim; a short additional experiment or explicit caveat with a counter-example pattern would strengthen the central result.
  2. §V-C and Fig. 4: Channel count is treated as a free parameter that is swept offline and then fixed at the best value (often 64). Dense NCCL heuristics are deliberately overridden. Because the reported speedups depend on this elevated channel count (and the associated SM occupancy trade-off), the paper should quantify how much of the gain disappears when the same channel heuristic used by stock NCCL is forced, or state clearly that SpCCL requires a different channel policy.
minor comments (5)
  1. Fig. 3 and §IV-D: COO vs Pici comparison is useful, but the text claims COO is “significantly slower for more moderate sparsities” while the all-gather curves remain competitive down to ~90%. A one-sentence clarification of when decompression cost dominates would help.
  2. §VI-A: The SparCML baseline is host-only and runs on older networks; the order-of-magnitude gap is expected but should be caveated more explicitly so readers do not over-interpret it as a pure algorithmic win.
  3. Eqs. (4)–(6): The extended Hockney model is only used for intuition; the paper never validates the predicted T_cd against measured (de)compression times. A short table or sentence would make the model more than decorative.
  4. Throughout: “NCCLX” is introduced without a citation or version pin until later; a footnote or reference to the arXiv preprint would improve reproducibility.
  5. Fig. 10–11: Error-feedback (EF) overhead is discussed but the absolute iteration times with EF are not plotted against the dense baseline; adding that curve would make the end-to-end claim clearer.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical wall-clock speedups against external dense baselines, with offline-tuned thresholds that do not define the reported results by construction.

full rationale

The paper's central claims are measured performance numbers (up to 5.25 imes/2.5 imes/2.66 imes over NCCL at 99% sparsity for AG/RS/AR, plus 13–26% end-to-end on pruned DDP) obtained from multi-job microbenchmarks and a Megatron-LM case study on Perlmutter. The extended Hockney model (Eqs. 1–6) is used only for intuition about when compression can help; no quantitative predictions are derived from it and then compared to the same data. Thresholds (inter_thresh=0.5, intra_thresh=0.6, ag_thresh=0.1) and the densification extrapolation (Eq. 7) are chosen by offline sweeps and a simple inclusion-exclusion assumption; they are hyperparameters of the adaptive algorithm, not fitted constants that are later re-labeled as predictions. Pici's space overhead (3.15%) follows directly from its bitvector+tile-index layout and is independent of the measured speedups. Related-work citations (SparCML, OmniReduce, etc.) are external and non-load-bearing for the correctness of the reported timings. No equation equates a claimed first-principles result to an input by construction, and there is no self-citation uniqueness chain. The derivation is therefore self-contained empirical engineering.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The central performance claims rest on a small set of empirically chosen thresholds, standard communication models, and the new Pici representation itself. No deep mathematical axioms are required; the work is systems engineering.

free parameters (4)
  • inter_thresh = 0.5
    Sparsity threshold below which inter-node sends switch to dense; chosen by offline sweep (0.5 for Pici).
  • intra_thresh = 0.6
    Sparsity threshold for intra-node (NVLink) sends; chosen by offline sweep (0.6 for Pici).
  • ag_thresh = 0.1
    Threshold deciding whether the all-gather phase of all-reduce stays sparse; set to 0.1 after empirical observation.
  • channel_count = 32-64 (best of sweep)
    Number of NCCL channels (CUDA blocks) raised above library defaults (often 32–64) after sweeps to saturate links under compression overhead.
assumptions (4)
  • domain assumption Ring algorithm is bandwidth-optimal and empirically superior to Tree for the large-message regime on Perlmutter.
    Stated in §IV-A; used to justify implementing only Ring variants.
  • domain assumption Densification of independent uniform random sparsity follows the inclusion-exclusion recurrence δ_next = 1-(1-δ_prev)(1-δ_0).
    Eq. 7 in §IV-C; used to extrapolate sparsity across dense intermediate steps.
  • domain assumption Hockney model (α + nβ) plus additive (de)compression terms adequately predicts when sparse communication wins.
    §II-A; guides focus on large messages and threshold design.
  • ad hoc to paper Pici’s 3.15% overhead and warp-cooperative kernels make (de)compression cheaper than the bandwidth saved at moderate-to-high sparsity.
    Core design claim of §III; validated only by the paper’s own kernels and measurements.
invented entities (2)
  • Pici sparse format
    purpose: Low-constant-overhead bitvector representation enabling fast parallel (de)compression at medium sparsities.
    Introduced in §III; no prior format matches the exact tile/index/kernel design. Independent evidence is limited to the paper’s own microbenchmarks.
  • SpCCL library
    purpose: NCCLX fork that implements the adaptive sparse collectives.
    Implementation vehicle; not a scientific entity beyond the algorithms it contains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Space-efficient Collectives for Dynamic and Unstructured Sparsity on GPU Platforms." pith.science (2026). https://pith.science/paper/RGZJGFOX

@misc{pith2026260704676,
  author       = {Pith},
  title        = {Pith review of: Adaptive Space-efficient Collectives for Dynamic and Unstructured Sparsity on GPU Platforms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RGZJGFOX}},
  note         = {Machine review of arXiv:2607.04676}
}
read the original abstract

High-performance collective communication primitives are necessary for a variety of high performance computing (HPC) and machine learning (ML) workloads. State-of-the-art collective communication libraries such as NCCL optimize exclusively for dense data. However, when sending sparse data, we can reduce communication volume by not sending zeros. Unfortunately, explicitly handling sparsity introduces challenges such as format conversion overheads and densification during collectives that involve reductions. In this paper, we introduce sparsity-exploiting algorithms for three collectives that address these challenges: all-gather, reduce-scatter, and all-reduce. Our collective implementations are backed by a new bitvector-based format, Pici, designed for low overhead and fast (de)compression at moderate sparsities. Further, our algorithms adapt to the level of sparsity in data, modifying its representation during the course of the collective. At 99% input sparsity, our collectives achieve up to 5.25x, 2.5x, and 2.66x speedups over NCCL for all-gather, reduce-scatter, and all-reduce, respectively.

Figures

Figures reproduced from arXiv: 2607.04676 by the authors.

Figure 1
Figure 1. Overview of how Pici bitvector and index components correspond to [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Threshold sweep for reduce-scatter with Pici, 512 MiB collective [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. SpCCL all-gather and all-reduce results on 16 GPUs, using Pici and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: All-gather channel sweep on 16 GPUs (512 MiB output, 99% sparsity). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Results for SpCCL all-gather (Pici) versus NCCL (Dense) and NCCLX (Dense) baselines on several GPU counts for 512 and 2048 MiB problem [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: SpCCL sp-all-gather (Pici) speedups over NCCL all-gather (Dense). Speedups are greatest for large problem sizes and at high sparsities, but substantial [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: SpCCL all-reduce (Pici) speedups over NCCL (Dense). At 99% sparsity, substantial speedups can be achieved even at 64 GPUs, while for more [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: SpCCL reduce-scatter (Pici) speedups over NCCL (Dense). At 99% sparsity, substantial speedups can be achieved even at 64 GPUs, while for more [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Results for SpCCL all-reduce (Pici) versus dense baselines and a sparse SparCML baseline on several GPU counts and problem sizes. Speedups are [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Mean training iteration time for 40 GB and 80 GB GPUs over 8, 16, [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Breakdown showing all-reduce and pruning time on 32 GPUs. All [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 11 canonical work pages

  1. [1]

    Cpu- and gpu-initiated communication strategies for conjugate gradient methods on large gpu clusters,

    J. D. Trotter, S. Ekmekc ¸ibas ¸ı, D. Sa ˘gbili, J. Langguth, X. Cai, and D. Unat, “Cpu- and gpu-initiated communication strategies for conjugate gradient methods on large gpu clusters,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’25. New York, NY , USA: Association for Computing...

    doi:10.1145/3712285.3759774 2025

  2. [2]

    Amesos2 and belos: Direct and iterative solvers for large sparse linear systems,

    E. Bavier, M. Hoemmen, S. Rajamanickam, and H. Thornquist, “Amesos2 and belos: Direct and iterative solvers for large sparse linear systems,”Scientific Programming, vol. 20, no. 3, p. 243875, 2012. [Online]. Available: https://doi.org/10.3233/SPR-2012-0352

    work page doi:10.3233/spr-2012-0352 2012

  3. [3]

    Newly released capabilities in the distributed-memory superlu sparse direct solver,

    X. S. Li, P. Lin, Y . Liu, and P. Sao, “Newly released capabilities in the distributed-memory superlu sparse direct solver,”ACM Trans. Math. Softw., vol. 49, no. 1, Mar. 2023. [Online]. Available: https://doi.org/10.1145/3577197

    doi:10.1145/3577197 2023

  4. [4]

    Qiu,Scalable and Efficient Material Point Methods on Modern Computational Platforms

    Y . Qiu,Scalable and Efficient Material Point Methods on Modern Computational Platforms. University of California, Los Angeles, 2024

    2024

  5. [5]

    Exploiting sparsity in pruned neural networks to optimize large model training,

    S. Singh and A. Bhatele, “Exploiting sparsity in pruned neural networks to optimize large model training,” in 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS). Los Alamitos, CA, USA: IEEE Computer Society, may 2023, pp. 245–255. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/IPDPS54959.2023.00033

    doi:10.1109/ipdps54959.2023.00033 2023

  6. [6]

    Deep gradient compression: Reducing the communication bandwidth for distributed training,

    Y . Lin, S. Han, H. Mao, Y . Wang, and W. J. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,”arXiv preprint arXiv:1712.01887, 2017. [Online]. Available: https://doi.org/10.48550/arXiv.1712.01887

    arXiv doi:10.48550/arxiv.1712.01887 2017

  7. [7]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks,

    J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,”arXiv preprint arXiv:1803.03635,

    arXiv

  8. [8]

    Available: https://doi.org/10.48550/arXiv.1803.03635

    [Online]. Available: https://doi.org/10.48550/arXiv.1803.03635

    arXiv doi:10.48550/arxiv.1803.03635

Show all 53 references
  1. [9]

    Plexus: Taming billion-edge graphs with 3D parallel full-graph GNN training,

    A. K. Ranjan, S. Singh, C. Wei, and A. Bhatele, “Plexus: Taming billion-edge graphs with 3D parallel full-graph GNN training,” inProceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’25. ACM, Nov. 2025. ...

    2025 doi

  2. [10]

    Coruscant: Co-designing gpu kernel and sparse tensor core to advocate unstructured sparsity in efficient llm inference,

    D. Joo, H. Hosseini, R. Hadidi, and B. Asgari, “Coruscant: Co-designing gpu kernel and sparse tensor core to advocate unstructured sparsity in efficient llm inference,” inProceedings of the 58th IEEE/ACM International Symposium on Microarchitecture®, 2025, pp. 232–245. [Online...

    2025 doi

  3. [11]

    Sparcml: High-performance sparse communication for machine learning,

    C. Renggli, S. Ashkboos, M. Aghagolzadeh, D. Alistarh, and T. Hoefler, “Sparcml: High-performance sparse communication for machine learning,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2019, pp. 1–15. [Online...

    2019 doi

  4. [12]

    Efficient sparse collective communication and its application to accelerate distributed deep learning,

    J. Fei, C.-Y . Ho, A. N. Sahu, M. Canini, and A. Sapio, “Efficient sparse collective communication and its application to accelerate distributed deep learning,” inProceedings of the 2021 ACM SIGCOMM 2021 Conference, 2021, pp. 676–691. [Online]. Available: https://doi.org/10.11...

    2021 doi

  5. [13]

    Efficient sparse matrix-vector multiplication on cuda,

    N. Bell and M. Garland, “Efficient sparse matrix-vector multiplication on cuda,” Nvidia Technical Report NVR-2008-004, Nvidia Corporation, Tech. Rep., 2008

    2008

  6. [14]

    Yale sparse matrix package. ii. the nonsymmetric codes,

    S. Eisenstat, M. Gursky, M. Schultz, and A. Sherman, “Yale sparse matrix package. ii. the nonsymmetric codes,” Tech. Rep., 1977

    1977

  7. [15]

    Collective communication for 100k+ gpus,

    M. Si, P. Balaji, Y . Chen, C.-H. Chu, A. Gangidi, S. Hasan, S. Iyengar, D. Johnson, B. Liu, J. Renet al., “Collective communication for 100k+ gpus,”arXiv preprint arXiv:2510.20171, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2510.20171

    2025 doi

  8. [16]

    Perlmutter system architecture,

    NERSC, “Perlmutter system architecture,” https://docs.nersc.gov/systems/perlmutter/architecture/

  9. [17]

    [Online]

    “Nccl,” 2020. [Online]. Avail- able: https://docs.nvidia.com/deeplearning/nccl/user- guide/docs/overview.html

    2020

  10. [18]

    Optimization of collective reduction operations,

    R. Rabenseifner, “Optimization of collective reduction operations,” in International Conference on Computational Science. Springer, 2004, pp. 1–9. [Online]. Available: https://doi.org/10.1007/978-3-540-24685- 5 1

    2004 doi

  11. [19]

    The communication challenge for mpp: Intel paragon and meiko cs-2,

    R. W. Hockney, “The communication challenge for mpp: Intel paragon and meiko cs-2,”Parallel Comput., vol. 20, no. 3, p. 389–398, Mar. 1994. [Online]. Available: https://doi.org/10.1016/S0167-8191(06)80021-9

    1994 doi

  12. [20]

    Pytorch distributed: Experiences on accelerating data parallel training,

    S. Li, Y . Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania, and S. Chintala, “Pytorch distributed: Experiences on accelerating data parallel training,”Proc. VLDB Endow., vol. 13, no. 12, p. 3005–3018, Aug. 2020. [Online]. Available...

    2020 doi

  13. [21]

    Evaluation criteria for sparse matrix storage formats,

    D. Langr and P. Tvrdik, “Evaluation criteria for sparse matrix storage formats,”IEEE Transactions on parallel and distributed systems, vol. 27, no. 2, pp. 428–440, 2015. [Online]. Available: https://doi.org/10.1109/TPDS.2015.2401575

    2015 doi

  14. [22]

    A systematic survey of general sparse matrix-matrix multiplication,

    J. Gao, W. Ji, F. Chang, S. Han, B. Wei, Z. Liu, and Y . Wang, “A systematic survey of general sparse matrix-matrix multiplication,” ACM Computing Surveys, vol. 55, no. 12, pp. 1–36, 2023. [Online]. Available: https://doi.org/10.1145/3571157

    2023 doi

  15. [23]

    Sparskit: A basic tool kit for sparse matrix computations,

    Y . Saad, “Sparskit: A basic tool kit for sparse matrix computations,” Tech. Rep., 1990

    1990

  16. [24]

    Towards a universal fpga matrix-vector multiplication architecture,

    S. Kestur, J. D. Davis, and E. S. Chung, “Towards a universal fpga matrix-vector multiplication architecture,” in2012 IEEE 20th International Symposium on Field-Programmable Custom Computing Machines. IEEE, 2012, pp. 9–16. [Online]. Available: https://doi.org/10.1109/FCCM.2012.12

    2012 doi

  17. [25]

    Space-efficient, high- performance rank and select structures on uncompressed bit sequences,

    D. Zhou, D. G. Andersen, and M. Kaminsky, “Space-efficient, high- performance rank and select structures on uncompressed bit sequences,” inInternational Symposium on Experimental Algorithms. Springer, 2013, pp. 151–163. [Online]. Available: https://doi.org/10.1007/978-3- 642-3...

    2013 doi

  18. [26]

    Engineering compact data structures for rank and select queries on bit vectors,

    F. Kurpicz, “Engineering compact data structures for rank and select queries on bit vectors,” inInternational Symposium on String Processing and Information Retrieval. Springer, 2022, pp. 257–272. [Online]. Available: https://doi.org/10.1007/978-3-031-20643-6 19

    2022 doi

  19. [27]

    Data parallel algorithms,

    W. D. Hillis and G. L. S. Jr., “Data parallel algorithms,”Communications of the ACM, vol. 29, no. 12, pp. 1170–1183, 1986. [Online]. Available: https://doi.org/10.1145/7902.7903

    1986 doi

  20. [28]

    Pat: a new algorithm for all-gather and reduce-scatter operations at scale,

    S. Jeaugey, “Pat: a new algorithm for all-gather and reduce-scatter operations at scale,” 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2506.20252

    2025 doi arXiv

  21. [29]

    Communication algorithm-architecture co-design for distributed deep learning,

    J. Huang, P. Majumder, S. Kim, A. Muzahid, K. H. Yum, and E. J. Kim, “Communication algorithm-architecture co-design for distributed deep learning,” in2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA). IEEE, 2021, pp. 181–194. [Online]. Availabl...

    2021 doi

  22. [30]

    Optimization of collective communication operations in mpich,

    R. Thakur, R. Rabenseifner, and W. Gropp, “Optimization of collective communication operations in mpich,”The International Journal of High Performance Computing Applications, vol. 19, no. 1, pp. 49–66,

  23. [31]

    Available: https://doi.org/10.1177/1094342005051521

    [Online]. Available: https://doi.org/10.1177/1094342005051521

    doi

  24. [32]

    The big send-off: Scalable and performant collectives for deep learning,

    S. Singh, K. Pradeep, M. Singh, C. Wei, and A. Bhatele, “The big send-off: Scalable and performant collectives for deep learning,” 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2504.18658

    2026 doi

  25. [33]

    Gpudirect,

    NVIDIA, “Gpudirect,” https://docs.nvidia.com/cuda/gpudirect- rdma/index.html

  26. [34]

    Nccl tests,

    ——, “Nccl tests,” https://github.com/NVIDIA/nccl-tests, 2017

    2017

  27. [35]

    The mvapich project: Transforming research into high-performance mpi library for hpc community,

    D. K. Panda, H. Subramoni, C.-H. Chu, and M. Bayatpour, “The mvapich project: Transforming research into high-performance mpi library for hpc community,”Journal of Computational Science, vol. 52, p. 101208, 2021. [Online]. Available: https://doi.org/10.1016/j.jocs.2020.101208

    2021 doi

  28. [36]

    OSU Micro-Benchmarks,

    Network-Based Computing Laboratory, The Ohio State University, “OSU Micro-Benchmarks,” accessed 2026-04-04. [Online]. Available: https://mvapich.cse.ohio-state.edu/benchmarks/

    2026

  29. [37]

    1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs,

    F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs,” inInterspeech 2014, 2014, pp. 1058–1062. [Online]. Available: https://doi.org/10.21437/Interspeech.2014-274

    2014 doi

  30. [38]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” Tech. Rep., 2019

    2019

  31. [39]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Wei, T. Liu, M. Tian, D. Kocetkov, A. Zucker, Y . Belkada, Z. Wang, Q. Liu, D. Abulkhanov, I. Paul, Z. Li, W.-D. Li, M. Risdal, J. Li, J. Zhu, T. Y . Zhuo, E. Zheltonozhskii,...

    2024 doi arXiv

  32. [40]

    Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,

    Y . Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler, “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,” inarXiv preprint arXiv:1506.06724, 2015. [Online]. Available: https://doi.org/10.485...

    2015 doi arXiv

  33. [41]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” Tech. Rep., 2020. [Online]. Available: https://doi.org/10.48550/arXiv.1909.08053

    2020 doi arXiv

  34. [42]

    Gasnet-ex rma communication performance on recent supercomputing systems,

    P. H. Hargrove and D. Bonachea, “Gasnet-ex rma communication performance on recent supercomputing systems,” 2022. [Online]. Available: https://doi.org/10.25344/S40C7D

    2022 doi

  35. [43]

    omnireduce-experiments,

    SANDS Lab, “omnireduce-experiments,” gitHub repository, commit a2bdc8082c805a1cb86499ce1eae7a9b48fbf8c4, accessed 2026-04-

    2026

  36. [44]

    Available: https://github.com/sands-lab/omnireduce- experiments/commit/a2bdc8082c805a1cb86499ce1eae7a9b48fbf8c4

    [Online]. Available: https://github.com/sands-lab/omnireduce- experiments/commit/a2bdc8082c805a1cb86499ce1eae7a9b48fbf8c4

  37. [45]

    Sparse collectives: Exploiting data sparsity to improve communication efficiency,

    D. Wijerathne, H. Javaid, G. Zhong, D. Wu, X. Y . Kom, and M. Baldi, “Sparse collectives: Exploiting data sparsity to improve communication efficiency,” inProceedings of the 2nd Workshop on Networks for AI Computing, 2025, pp. 64–66. [Online]. Available: https://doi.org/10.114...

    2025 doi

  38. [46]

    Omniccl: Zero-cost sparse allreduce with direct cache access and smartnics,

    T. Gu, J. Fei, and M. Canini, “Omniccl: Zero-cost sparse allreduce with direct cache access and smartnics,” inProceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing, 2024, pp. 75–83. [Online]. Available: https://doi.org/10.1145/3672198.3673804

    2024 doi

  39. [47]

    D-dosa: Dpu-based dataflow offloading and sparse allreduce framework for distributed training,

    Z. Yu, W. Li, S. Guo, Q. Li, F. Qi, and J. Xiu, “D-dosa: Dpu-based dataflow offloading and sparse allreduce framework for distributed training,”IEEE Transactions on Cloud Computing, 2025. [Online]. Available: https://doi.org/10.1109/TCC.2025.3634564

    2025 doi

  40. [48]

    Near-optimal sparse allreduce for distributed deep learning,

    S. Li and T. Hoefler, “Near-optimal sparse allreduce for distributed deep learning,” inProceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2022, pp. 135–149. [Online]. Available: https://doi.org/10.1145/3503221.3508399

    2022 doi

  41. [49]

    A distributed synchronous sgd algorithm with global top-k sparsification for low bandwidth networks,

    S. Shi, Q. Wang, K. Zhao, Z. Tang, Y . Wang, X. Huang, and X. Chu, “A distributed synchronous sgd algorithm with global top-k sparsification for low bandwidth networks,” in2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). IEEE, 2019, pp. 2238–22...

    2019 doi

  42. [50]

    Hecate: Unlocking efficient sparse model training via fully sharded sparse data parallelism,

    Y . Qing, G. Zhu, F. Li, L. Lei, Z. Sun, X. Guan, S. Zhao, X. Chen, D. Huang, S. Wanget al., “Hecate: Unlocking efficient sparse model training via fully sharded sparse data parallelism,”arXiv preprint arXiv:2502.02581, 2025. [Online]. Available: https://doi.org/10.48550/arXiv...

    2025 doi arXiv

  43. [51]

    Sparse collective operations for mpi,

    T. Hoefler and J. L. Traff, “Sparse collective operations for mpi,” in2009 IEEE International Symposium on Parallel & Distributed Processing. IEEE, 2009, pp. 1–8. [Online]. Available: https://doi.org/10.1109/IPDPS.2009.5160935

    2009 doi

  44. [52]

    Understanding top-k sparsi- fication in distributed deep learning,

    S. Shi, X. Chu, K. C. Cheung, and S. See, “Understanding top-k sparsi- fication in distributed deep learning,”arXiv preprint arXiv:1911.08772,

    1911 arXiv

  45. [53]

    Available: https://doi.org/10.48550/arXiv.1911.08772

    [Online]. Available: https://doi.org/10.48550/arXiv.1911.08772

    1911 doi arXiv

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.