Pith. sign in

REVIEW 2 major objections 5 minor 39 references

Optimizing ML Concurrent Computation and Communication with GPU DMA Engines

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Offloading GPU communication to DMA engines closes most of the gap between realized and ideal speedup for concurrent ML compute and communication.

desk verdict A solid C3 characterization that is currently unverifiable on its central claim because the paper never checks that the DMA transfers actually move the right bytes. read the letter →

arxiv 2412.14335 v2 pith:C5B5QJMQ submitted 2024-12-18 cs.AR cs.DC

Anirudha Agrawal , Shaizeen Aga , Suchita Pati , Mahzabeen Islam This is my paper · ORCID
classification cs.ARcs.DC
keywords concurrentcomputationandcommunicationGPUDMAenginescollectiveGEMMinterferencescheduleprioritizationresourcepartitioningMI300XMLtraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that concurrent computation and communication (C3), a common pattern in ML training, is far less beneficial on GPUs than it could be, and that most of the lost benefit is avoidable. On an MI300X 8-GPU node, overlapping a matrix-multiplication kernel (GEMM) with an all-gather or all-to-all collective reaches only 21% of the ideal speedup, defined as serial time over the longer of the two isolated kernels. The paper shows that scheduling the communication kernel first and partitioning compute units raises that to 42%, and that offloading communication to the GPU's DMA engines, through proof-of-concept libraries called ConCCL, raises it to 72% on average, up to 1.67x speedup. The reason ConCCL works is that DMA transfers use no compute units and bypass L1/L2 caches, whereas compute-unit-based collectives steal both from the concurrent GEMM. If true, the result makes a concrete case for making GPU DMA engines more capable.

What carries the argument

The central mechanism is the GPU's SDMA (system direct memory access) engine: a copy engine on the I/O die that, when given a transfer command, moves data between HBM and the intra-node network without occupying compute units and without touching L1/L2 caches. ConCCL breaks each collective operation into individual transfers scheduled on these engines from the CPU through an HSA runtime call. The supporting mechanisms are the 'ideal speedup' yardstick, defined as serial execution time divided by the maximum of isolated GEMM time and isolated collective time, and the C3 taxonomy (G-long, C-long, GC-equal; compute-bound versus memory-bound GEMMs; latency-bound versus bandwidth-bound collectives) that organizes the measured scenarios. Schedule prioritization and compute-unit partitioning are the two software-only optimizations that bring performance from 21% to 42% of ideal before the DMA offload is applied.

What would settle it

Run a compute-bound GEMM alone, then run the same GEMM concurrently with a large DMA transfer and with a compute-unit-based collective, measuring both GEMM slowdown and L2 or Infinity Cache traffic with a profiler. If the DMA-concurrent GEMM slows down by more than a negligible amount, or if DMA transfers measurably increase cache access counts, the no-interference premise fails and ConCCL's average 72%-of-ideal figure would not hold on that system.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the gap between realized and ideal concurrent computation-communication speedup on an 8-GPU MI300X node is mostly caused by compute-unit and cache interference, not by memory bandwidth, and that this gap can be substantially closed by offloading collectives to the chip's DMA engines. With ConCCL proof-of-concepts for all-gather and all-to-all, average realized speedup rises from 21% to 72% of ideal, with a maximum of 1.67x over serial execution, and this is achieved without new hardware. The paper supports this with a taxonomy of C3 scenarios, isolated kernel measurements, and two software-only optimizations that recover part of the gap before the DMA offload is added.

Load-bearing premise

The entire ConCCL benefit rests on the architectural assumption that the MI300X's DMA engines transfer data without consuming GPU compute cores or L1/L2 cache, leaving all 304 compute units and the caches fully available to the concurrent GEMM; that property is taken from the chip design rather than measured under concurrent load in this paper.

Editorial extensions

If this is right

  • If GPU DMA engines are improved, concurrent computation and communication on GPUs can approach the ideal speedup defined by the max of isolated compute and communication times.
  • Communication libraries can match compute-unit-based collectives on large transfers while freeing all compute units for concurrent GEMMs, as ConCCL does at sizes of 128MB and above.
  • A runtime can improve C3 performance simply by scheduling the lower-workgroup-count kernel first, without requiring special hardware.
  • Resource partitioning can be guided by a one-time lookup table of kernel slowdowns, which predicted the optimal compute-unit allocation for 24 of 30 studied C3 scenarios.
  • All-to-all, which needs more compute units, benefits the most from DMA offload, going from 1.05x to 1.43x speedup over serial execution.
  • The remaining 28% gap between realized and ideal speedup is attributed to HBM bandwidth contention, which the paper leaves to future work.
  • ConCCL can be applied to the intra-node phase of hierarchical multi-node collectives, extending its benefit to large-scale training without new hardware.
  • The launch and synchronization cost that makes ConCCL slower than conventional collectives below 32MB suggests that a GPU-side DMA launch path would extend the technique to latency-bound C3, not just bandwidth-bound scenarios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper, the remaining gap between 72% and 100% of ideal is likely HBM bandwidth contention, and a runtime could test this by partitioning HBM channels or traffic classes, which the paper explicitly leaves to future work.
  • Going beyond the paper, the launch and synchronization cost that makes ConCCL 4x slower than conventional collectives below 32MB suggests that on future GPUs a DMA queue readable directly by the GPU could extend the technique to latency-bound C3, not just bandwidth-bound scenarios.
  • Going beyond the paper, the same taxonomy and one-time slowdown-table method could be applied to other GPU architectures with different numbers of compute units and different cache topologies, making the runtime heuristics portable beyond MI300X.
  • Going beyond the paper, if DMA engines gained arithmetic capability, all-reduce could also be offloaded, and the measured 72%-of-ideal plateau suggests that this would be the next practical target for closing the C3 performance gap.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper characterizes concurrent computation and communication (C3) on an 8-GPU AMD MI300X node. It introduces a taxonomy based on GEMM type and collective type, measures isolated compute and memory requirements, and defines ideal speedup as serial execution time divided by the maximum of isolated GEMM and collective times. It reports that baseline C3 achieves only 21% of the ideal speedup, that schedule prioritization and compute-unit partitioning raise this to about 42%, and that offloading all-gather/all-to-all communication to the GPU's sDMA engines with ConCCL proof-of-concepts achieves on average 72% of ideal speedup (up to 1.67x). The paper also proposes runtime heuristics for schedule prioritization and resource partitioning.

Significance. The paper provides a useful, systematic characterization of an important real platform and makes a plausible case that DMA-engine-based collectives can reduce compute/cache interference in C3. The strengths include a concrete taxonomy with model-derived and synthetic scenarios, isolated compute/memory profiling, simple runtime heuristics with a held-out validation (24/30 scenarios, max 1.5% loss), and the use of an external baseline (RCCL) rather than fitting parameters to the target result. The main empirical claims are not circular. However, reproducibility and correctness are currently limited by the absence of a code artifact, the absence of error bars for the 9-run measurements, and the absence of any validation that ConCCL transfers are byte-exact.

major comments (2)
  1. [Section VI-B/VI-C/VI-F (Figs. 9-10)] No correctness check for ConCCL transfers is described anywhere in the manuscript. The paper reports that ConCCL is 'at par' with RCCL for >=128MB and uses this to justify evaluating C3 with ConCCL, but the speedup numbers in Figure 10 are only meaningful if the DMA collectives actually transfer the correct data. Because the ConCCL PoCs use simple direct transfers via hsa_amd_memory_async_copy_on_engine, a dropped or wrong-byte transfer would make the GEMM appear faster and would inflate the reported 72% of ideal speedup. Please add an explicit validation step (e.g., buffer comparison or checksum after each collective for every size and configuration) and report that all timing runs used validated transfers.
  2. [Section IV-A1 and all result figures] The paper runs 9 measured executions per configuration but reports only point averages, with no error bars, confidence intervals, or min-max values. The central quantitative claims (21%, 42%, 66%, 72% of ideal speedup; 1.67x) are averages over a small number of runs; without a measure of variance it is impossible to tell whether the differences among c3_base, c3_sp, c3_rp, ConCCL, and ConCCL_rp are larger than run-to-run noise. Please report per-configuration dispersion, such as standard deviations or min-max ranges, for Figures 8 and 10.
minor comments (5)
  1. [Section IV-B1 vs. Figure 5] The text says 'all-gather kernels need 32 CUs, while all-to-all kernels need 64 CUs', but the Figure 5 captions state 'All-gather default #CUs=64, All-to-all default #CUs=56'. Please clarify which number is the saturation threshold and which is the library default.
  2. [General] No code or artifact is provided; given that the contributions are empirical, an artifact appendix with the ConCCL PoCs and measurement scripts would substantially improve reproducibility.
  3. [Section V-B] The c3_rp result is defined as the best-performing CU allocation from a sweep over all powers-of-two allocations, which is an oracle/upper-bound result; the text should make clear that the 41% figure in Figure 8 is not the expected performance of a fixed strategy but an optimized upper bound, even though the heuristic in Section V-C recovers much of the benefit.
  4. [Section II-C] There is a typo: 'as such as be scheduled concurrently' should read 'as such can be scheduled concurrently'.
  5. [Section III] The definition of 'latency-bound' communication ('kernel latency at/before this size does not increase commensurate to size') is vague; a quantitative criterion, such as a bandwidth plateau threshold, would make the taxonomy easier to reproduce.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; central ConCCL speedup is measured against an external RCCL baseline.

full rationale

The paper's central claim—ConCCL attains on average 72% of ideal speedup versus 21% for the baseline—is an experimental measurement, not a derivation from fitted parameters or a self-citation chain. The ideal speedup is defined as serial execution time divided by the maximum of isolated GEMM and isolated collective times, and the reported 'percentage of ideal speedup' is simply the ratio of the measured concurrent speedup to that ideal; this is a normalization of measured data, not a self-definitional result. ConCCL is benchmarked against RCCL, an external communication library, in Section VI-C (Figure 9), and the paper explicitly states its simple DMA-based PoCs are at parity with RCCL for sizes of 128MB and above, which are the sizes used in the C3 evaluations. This grounds the comparison in real, externally comparable communication work. The CU-allocation heuristic in Section V-C builds a slowdown lookup table from a small subset of GEMM and collective kernels and then validates it against the C3 scenarios; although the heuristic is secondary and partly in-sample, the paper transparently reports its accuracy (24 of 30 scenarios, at most 1.5% loss versus exhaustive sweep), so it is not a fitted input masquerading as a central prediction. Self-citations [11], [19], and [33] appear as motivational context or related-work pointers; none is load-bearing for the ConCCL speedup result. The absence of a stated data-correctness validation for the DMA transfers is a legitimate correctness/benchmark-artifact risk, but it is not a circularity: no equation or fitted parameter reduces the headline result to its own inputs. Overall, the central claim is self-contained and externally benchmarked, so circularity is minimal.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

ConCCL is not an invented entity; it is a software proof-of-concept built on existing sDMA hardware. No new particles, forces, or hardware components are postulated. The free parameters listed are hand-chosen thresholds and oracle-selected CU counts that affect the reported speedups but not the core ConCCL mechanism.

free parameters (4)
  • Best CU allocation for c3_rp = varies by scenario, chosen as best of powers of two
    Section V-B sweeps CU assignments and reports the best, so the 41% result is an oracle-selected allocation.
  • Best CU removal for ConCCL_rp = 8 CUs for MI300X
    Section VI-F finds removing eight CUs from memory-bound GEMMs improves speedup.
  • Roofline efficiency factor = 0.70
    Section V-C heuristic assumes 70% efficiency to predict CU allocations.
  • Taxonomy classification thresholds = 115% and 15%
    Section III hand-chosen boundaries for G-long, C-long, GC-equal; not load-bearing for the ConCCL result.
assumptions (3)
  • domain assumption DMA engines on MI300X perform transfers without using compute units and bypass L1/L2 caches
    Stated in Sections II-A and VI-A as an architectural fact; central to the ConCCL benefit, not directly measured under concurrent load.
  • domain assumption RCCL and ConCCL collective performance is at par for collective sizes >=128MB
    Shown in Figure 9 for isolated execution; used to justify fair C3 comparison in Section VI-C.
  • standard math Ideal speedup bound equals max(isolated GEMM time, isolated collective time)
    Definition used in Section IV-B3 for ideal speedup projection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing ML Concurrent Computation and Communication with GPU DMA Engines." pith.science (2026). https://pith.science/paper/C5B5QJMQ

@misc{pith2026241214335,
  author       = {Pith},
  title        = {Pith review of: Optimizing ML Concurrent Computation and Communication with GPU DMA Engines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C5B5QJMQ}},
  note         = {Machine review of arXiv:2412.14335}
}
read the original abstract

Concurrent computation and communication (C3) is a pervasive paradigm in ML and other domains, making its performance optimization crucial. In this paper, we carefully characterize C3 in ML on GPUs, which are most widely deployed for ML training and inference. We observe that while C3 leads to performance uplifts, the uplifts are far lower than ideal speedups (serial computation and communication versus maximum of computation or communication; all times from isolated executions). That is, C3 on average achieves only 21% of ideal speedup. This is so, due to known challenges of compute and memory interference between concurrent GPU kernels (that is, sharing of GPU's compute units, caches and HBM). To attain better performance for C3, first, we evaluate dual strategies of schedule prioritization and careful resource partitioning of compute units on GPUs to push performance attained with C3 (on average 42% of ideal speedup). We also provide heuristics that can guide a runtime while employing these strategies. To further enhance C3 performance, we propose to mitigate C3 interference by offloading communication tasks to the GPU's DMA engines. To this end, we build concurrent communication collectives (ConCCL) proof-of-concepts that harness DMA engines for communication. We show how ConCCL considerably closes the gap between realized and ideal speedup for C3 (on average 72% of ideal speedup is realized, up to 1.67x speedup). Overall, our work makes a strong case for GPU DMA engine advancements to better support C3 on GPUs.

Figures

Figures reproduced from arXiv: 2412.14335 by the authors.

Figure 1
Figure 1. Baseline C3 (left) and C3 with ConCCL via DMA offloads (right). [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. State-of-art AMD Instinct™ MI300X. nication to GPU compute-units, we build concurrent communication collectives (ConCCL) proof-of-concepts, which offload communication to DMA engines on GPUs. • Using ConCCL, we demonstrate that C3 interference is lowered, considerably closing the gap between realized and ideal speedup (on average 72% of ideal speedup is realized). • Overall, we make a strong case to enhance the capa… view at source ↗
Figure 4
Figure 4. C3 taxonomy. like nano-batching [3] break down a single batch of inputs into nano-batches opening up opportunity for compute and communication kernels from different nano-batches to be co￾scheduled on a GPU. Overall, given the ample prevalence of C3 in ML, characterization and optimization of C3 on GPUs is important and hence the focus of this work. III. C3 TAXONOMY We begin with a taxonomy for C3 and subsequently a… view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: (a) GEMM kernel slowdown due to loss of compute units (CUs) in GPU. (b) All-gather, (c) All-to-all kernel slowdown with specific # CUs assigned [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Relative AMD Infinity Cache™ bandwidth utilization. number of CUs are taken away from the kernel to an execution where all CUs are available to the kernel (0 on x-axis is when the kernel has all 304 CUs available to it). As depicted in the figure, memory-bound GEMMs ar…
Figure 7
Figure 7. Figure 7: Ideal speedup possible for C3 scenarios under study. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Speedups for C3 scenarios under study with and without schedule [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 10
Figure 10. Figure 10: C3 speedup with ConCCL. B. ConCCL: DMA-collectives PoCs With the above strong motivation, we build ConCCL proof￾of-concepts (PoCs), which are DMA-based collectives wherein we offload the collectives to DMA engines on MI300X. As MI300X DMA engines do not expose any com…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 28 canonical work pages

  1. [1]

    A bridging model for parallel computation,

    L. G. Valiant, “A bridging model for parallel computation,” Commun. ACM, vol. 33, no. 8, p. 103–111, Aug. 1990. [Online]. Available: https://doi.org/10.1145/79173.79181

    arXiv 1990

  2. [2]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel,

    Y . Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y . Hao, A. Mathews, and S. Li, “Pytorch fsdp: Experiences on scaling fully sharded data parallel,”

  3. [3]

    Nanoflow: Towards optimal large language model serving throughput,

    K. Zhu, Y . Zhao, L. Zhao, G. Zuo, Y . Gu, D. Xie, Y . Gao, Q. Xu, T. Tang, Z. Ye, K. Kamahori, C.-Y . Lin, S. Wang, A. Krishnamurthy, and B. Kasikci, “Nanoflow: Towards optimal large language model serving throughput,” 2024. [Online]. Available: https://arxiv.org/abs/2408.12757

    arXiv 2024

  4. [4]

    The llama 3 herd of models,

    A. M. Llama Team, “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    arXiv 2024

  5. [5]

    {ARK}:{GPU-driven} code execution for distributed deep learning,

    C. Hwang, K. Park, R. Shu, X. Qu, P. Cheng, and Y . Xiong, “{ARK}:{GPU-driven} code execution for distributed deep learning,” in 20th USENIX Symposium on Networked Systems Design and Imple- mentation (NSDI 23) , 2023, pp. 87–101

    work page 2023

  6. [6]

    11.1 amd instincttm mi300 series modular chiplet package – hpc and ai accelerator for exa-class systems,

    A. Smith, E. Chapman, C. Patel, R. Swaminathan, J. Wuu, T. Huang, W. Jung, A. Kaganov, H. McIntyre, and R. Mangaser, “11.1 amd instincttm mi300 series modular chiplet package – hpc and ai accelerator for exa-class systems,” in 2024 IEEE International Solid-State Circuits Conference (ISSCC), vol. 67, 2024, pp. 490–492

    work page 2024

  7. [7]

    AMD Instinct™MI300X Accelerator: Packaging and Architecture Co-Optimization,

    A. Smith, G. H. Loh, J. Wuu, S. Naffziger, T. Huang, H. McIntyre, R. Mangaser, W. Jung, and R. Swaminathan, “AMD Instinct™MI300X Accelerator: Packaging and Architecture Co-Optimization,” in 2024 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits), 2024, pp. 1–8

    work page 2024

  8. [8]

    The AMD CDNA™ 3 architecture,

    AMD, “The AMD CDNA™ 3 architecture,” https://www.amd. com/content/dam/amd/en/documents/instinct-tech-docs/white-papers/ amd-cdna-3-white-paper.pdf, 2024

    work page 2024

Show all 39 references
  1. [9]

    HSA Runtime API and runtime for ROCm,

    AMD, “HSA Runtime API and runtime for ROCm,” https://rocm.docs. amd.com/projects/ROCR-Runtime/en/latest/, 2024

    2024

  2. [10]

    HIP: C++ Heterogeneous-Compute Interface for Portability,

    ——, “HIP: C++ Heterogeneous-Compute Interface for Portability,” https://github.com/ROCm/HIP, 2024

    2024

  3. [11]

    Tale of two cs: Computation vs. communication scaling for future transformers on future hardware,

    S. Pati, S. Aga, M. Islam, N. Jayasena, and M. D. Sinclair, “Tale of two cs: Computation vs. communication scaling for future transformers on future hardware,” in 2023 IEEE International Symposium on Workload Characterization (IISWC), 2023, pp. 140–153

    2023

  4. [12]

    AMD ROCm™ Software,

    AMD, “AMD ROCm™ Software,” https://www.amd.com/en/products/ software/rocm.html, 2024

    2024

  5. [13]

    ROCm/rocBLAS: Next generation BLAS implementation for ROCm platform,

    AMD, “ROCm/rocBLAS: Next generation BLAS implementation for ROCm platform,” https://github.com/ROCm/rocBLAS, 2024

    2024

  6. [14]

    ROCm Communication Collectives Library (RCCL)

    ——, “ROCm Communication Collectives Library (RCCL).” [Online]. Available: https://github.com/ROCm/rccl

  7. [15]

    ROCm: HIPStream,

    ——, “ROCm: HIPStream,” https://rocm.docs.amd.com/projects/HIP/ en/latest/reference/hip runtime api/modules/stream management.html, 2024

    2024

  8. [16]

    rocprof — ROC Profiler Documentation,

    AMD, “rocprof — ROC Profiler Documentation,” https://rocm.docs. amd.com/projects/rocprofiler/en/docs-5.5.1/rocprof.html, 2024

    2024

  9. [17]

    AMD Instinct™ MI300X Accelerator Perfor- mance Validation Guide,

    ——, “AMD Instinct™ MI300X Accelerator Perfor- mance Validation Guide,” https://www.amd.com/content/ dam/amd/en/documents/instinct-tech-docs/product-briefs/ amd-instinct-mi300x-performance-validation-guide.pdf, 2024

    2024

  10. [18]

    ROCm: ROCR-Runtime,

    AMD, “ROCm: ROCR-Runtime,” https://rocm.docs.amd.com/projects/ HIP/en/develop/doxygen/html/hsa ext amd 8h.html, 2024

    2024

  11. [19]

    T3: Transparent tracking & triggering for fine-grained overlap of compute & collectives,

    S. Pati, S. Aga, M. Islam, N. Jayasena, and M. D. Sinclair, “T3: Transparent tracking & triggering for fine-grained overlap of compute & collectives,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Syste...

    2024

  12. [20]

    Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale,

    S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y . Aminabadi, A. A. Awan, J. Rasley, and Y . He, “Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale,” 2022. [Online]. Available: https://arxiv.org/abs/2201.05596

    2022 arXiv

  13. [21]

    TimeGraph: GPU Scheduling for Real-Time Multi-Tasking Environments,

    S. Kato, K. Lakshmanan, R. Rajkumar, and Y . Ishikawa, “TimeGraph: GPU Scheduling for Real-Time Multi-Tasking Environments,” in Pro- ceedings of the 2011 USENIX Conference on USENIX Annual Technical Conference. Portland, OR: USENIX Association, Jun 2011

    2011

  14. [22]

    MSC- CLang: Microsoft Collective Communication Language,

    M. Cowan, S. Maleki, M. Musuvathi, O. Saarikivi, and Y . Xiong, “MSC- CLang: Microsoft Collective Communication Language,” inProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , 2023, pp. 502–514

    2023

  15. [23]

    Available: https://github.com/NVIDIA/nccl

    NVIDIA, “NCCL.” [Online]. Available: https://github.com/NVIDIA/nccl

  16. [24]

    TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches,

    A. Shah, V . Chidambaram, M. Cowan, S. Maleki, M. Musuvathi, T. Mytkowicz, J. Nelson, O. Saarikivi, and R. Singh, “TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches,” in 20th USENIX Symposium on Networked Systems Design and Implementation, ser. NSDI. B...

    2023

  17. [25]

    MSCCL++: A GPU-driven communication stack for scalable AI applications

    P. Cheng, R. Dathathri, C. Hwang, A. Jangda, S. Kalivardhan, B. Li, S. Liu, S. Maleki, M. Musuvathi, C. Rocha, O. Saarikivi, A. Shah, W. Tsui, and Z. Yang, “MSCCL++: A GPU-driven communication stack for scalable AI applications.” [Online]. Available: https://github.com/microso...

  18. [26]

    Deep learning recommendation model for personalization and recommenda- tion systems,

    M. Naumov, D. Mudigere, H.-J. M. Shi, J. Huang, N. Sundaraman, J. Park, X. Wang, U. Gupta, C.-J. Wu, A. G. Azzolini et al. , “Deep learning recommendation model for personalization and recommenda- tion systems,” arXiv preprint arXiv:1906.00091 , 2019

    1906 arXiv

  19. [27]

    Megatron-LM: Training Multi-Billion Parameter Language Mod- els Using Model Parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-LM: Training Multi-Billion Parameter Language Mod- els Using Model Parallelism,” 2019

    2019

  20. [28]

    Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms,

    S. Rashidi, M. Denton, S. Sridharan, S. Srinivasan, A. Suresh, J. Nie, and T. Krishna, “Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms,” in 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture , ser. ISCA, IEEE. Pis...

    2021

  21. [29]

    The Case for GPGPU Spatial Multitasking,

    J. T. Adriaens, K. Compton, N. S. Kim, and M. J. Schulte, “The Case for GPGPU Spatial Multitasking,” in IEEE International Symposium on High-Performance Comp Architecture, ser. HPCA, IEEE. Washington, DC, USA: IEEE Computer Society, 2012, pp. 1–12

    2012

  22. [30]

    Improving GPGPU energy-efficiency through concurrent kernel execution and DVFS,

    Q. Jiao, M. Lu, H. P. Huynh, and T. Mitra, “Improving GPGPU energy-efficiency through concurrent kernel execution and DVFS,” in 2015 IEEE/ACM International Symposium on Code Generation and Optimization, ser. CGO. IEEE, 2015, pp. 1–11

    2015

  23. [31]

    Improving GPGPU Concurrency with Elastic Kernels,

    S. Pai, M. J. Thazhuthaveetil, and R. Govindarajan, “Improving GPGPU Concurrency with Elastic Kernels,” in Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems , 2013, p. 407–418

    2013

  24. [32]

    Rammer: Enabling Holistic Deep Learning Compiler Optimizations with rTasks,

    L. Ma, Z. Xie, Z. Yang, J. Xue, Y . Miao, W. Cui, W. Hu, F. Yang, L. Zhang, and L. Zhou, “Rammer: Enabling Holistic Deep Learning Compiler Optimizations with rTasks,” in 14th USENIX Symposium on Operating Systems Design and Implementation , ser. OSDI. Renton, W A: USENIX Assoc...

    2020

  25. [33]

    Global optimiza- tions & lightweight dynamic logic for concurrency,

    S. Pati, S. Aga, N. Jayasena, and M. D. Sinclair, “Global optimiza- tions & lightweight dynamic logic for concurrency,” arXiv preprint arXiv:2409.02227, 2024

    2024 arXiv

  26. [34]

    An In-Network Architecture for Accelerating Shared-Memory Multiprocessor Collec- tives,

    B. Klenk, N. Jiang, G. Thorson, and L. Dennison, “An In-Network Architecture for Accelerating Shared-Memory Multiprocessor Collec- tives,” inACM/IEEE 47th Annual International Symposium on Computer Architecture, ser. ISCA, IEEE. Washington, DC, USA: IEEE Computer Society, 2020...

    2020

  27. [35]

    Introducing Async Tensor Parallelism in PyTorch,

    Horace He, Less Wright, Luca Wehrstedt, Tianyu Liu, Wanchao Liang, “Introducing Async Tensor Parallelism in PyTorch,” ”https://discuss.pytorch.org/t/distributed-w-torchtitan-introducing- async-tensor-parallelism-in-pytorch/209487”, 2024

    2024

  28. [36]

    Optimizing distributed ml communication with fused computation-collective opera- tions,

    K. Punniyamurthy, K. Hamidouche, and B. M. Beckmann, “Optimizing distributed ml communication with fused computation-collective opera- tions,” arXiv preprint arXiv:2305.06942 , 2023

    2023 arXiv

  29. [37]

    Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads,

    A. Jangda, J. Huang, G. Liu, A. H. N. Sabet, S. Maleki, Y . Miao, M. Musuvathi, T. Mytkowicz, and O. Saarikivi, “Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads,” in Proceedings of the 27th ACM International Conference o...

    2022

  30. [38]

    Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models,

    S. Wang, J. Wei, A. Sabne, A. Davis, B. Ilbeyi, B. Hechtman, D. Chen, K. S. Murthy, M. Maggioni, Q. Zhang, S. Kumar, T. Guo, Y . Xu, and Z. Zhou, “Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models,” in Proceedings of the 28th ACM ...

    2022

  31. [2023]

    Available: https://arxiv.org/abs/2304.11277

    [Online]. Available: https://arxiv.org/abs/2304.11277

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.