Pith. sign in

REVIEW 3 major objections 4 minor 76 references

NIXT: A NCCL Inspector Exporter Tool for Observability of Collective Communication in Large Model Training

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NIXT claims to make NCCL Inspector profiling data actionable at scale: a taxonomy and columnar store turn per-collective logs into spatial and temporal analyses that localize GPU stragglers and attribute slowdowns to host composition.

desk verdict NIXT is a genuinely useful NCCL Inspector observability tool with an honest limitations section; the main issues are an unsupported 2% overhead claim and a straggler attribution that is plausible but not fully controlled. read the letter →

arxiv 2608.01449 v1 pith:WYGFBGML submitted 2026-08-02 cs.DC

Ziyang Jia , Sirshak Das , Jason Sewall , Laxmi Bhuyan , Pasha Shamis , Daniel Wong This is my paper · ORCID
classification cs.DC
keywords collectivecommunicationNCCLInspectorobservabilityGPUclustermonitoringstragglerdetectionLLMpretrainingDuckDBrootcauseanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NIXT is a tool that makes the flood of raw NCCL Inspector profiling data usable in practice: it ingests per-collective logs into a compact typed database and provides taxonomy-driven SQL analysis primitives. On Nemotron-4 pretraining runs up to 2,048 H100 GPUs, it shows that a small set of "hot" communicator topologies and message sizes carry nearly all collective traffic, and that the bandwidth of each hot kernel stays stable as GPU count and model size grow. It also shows production training exhibits 2 to 5 times more bandwidth variation than isolated nccl-tests, pointing at the ML framework rather than GPU scaling as the source. The paper's central practical claim is that spatial and temporal correlation of the exported data can localize a GPU straggler to specific ranks and hosts and attribute the slowdown to job host composition, without putting analysis on the training critical path.

What carries the argument

The central mechanism is the taxonomy-plus-relational analysis layer. NIXT types each NCCL Inspector field by metric role — Identifier (rank, hostname, pid), Configuration ($n\_ranks$, $nnodes$), Workload (collective type, message size), Measurement (execution time, algorithmic and bus bandwidth), Counter (timestamps, sequence numbers) — and derives an interconnect class, $coll\_type$ (single-rank, nvlink-only, NIC-only, mixed), from communicator placement alone. This typed schema, stored columnar in DuckDB, turns observability questions into reusable SQL primitives: summary histograms plus four correlation families (temporal, spatial, resource, other). The spatial primitive carries the stra

What would settle it

Repeat the 16-GPU straggler comparison while collecting DCGM counters, NVLink CRC errors, and InfiniBand or switch link-error telemetry alongside NCCL Inspector logs. If a host flagged as a straggler shows clean link-error counts while the bandwidth collapse coincides with unrelated fabric traffic, the host-composition attribution fails. For the framework-variation claim, replay the recorded collective sequence (same topologies, message sizes, and inter-arrival times) outside the training loop: if the 2-to-5x coefficient-of-variation gap over nccl-tests does not shrink when framework schedulin

Watch

Extended reading notes

Core claim

Raw NCCL Inspector output is too high-volume and high-dimensional to yield actionable insight directly; NIXT's claim is that the data becomes interpretable once organized by a metric taxonomy (identifier, configuration, workload, measurement, counter) and a derived interconnect class (nvlink-only, NIC-only, mixed), then loaded into a columnar relational store. On Nemotron-4 pretraining traces up to 2,048 H100 GPUs, it shows a small set of hot kernels carry nearly all traffic, per-kernel bandwidth variation is stable across GPU scale but 2–5x higher in production than in nccl-tests, and per-rank/per-host disaggregation localizes a straggler to specific hosts.

Load-bearing premise

The straggler case study assumes that the only difference between healthy and degraded runs is which physical hosts the job was scheduled on; if network contention, thermal throttling, or another unmeasured condition caused the slowdown instead, the attribution to host composition would be wrong.

Editorial extensions

If this is right

  • A small set of hot (communicator topology, message size) kernels carries the bulk of collective traffic, so improving end-to-end scaling reduces to tuning a few configuration-specific kernels rather than a combinatorial configuration space.
  • Per-kernel bandwidth coefficient of variation stays roughly constant from 16 to 2,048 GPUs, so relative variance of a fixed communicator topology does not come from GPU scaling.
  • Production training shows 2 to 5 times higher bandwidth variation than isolated nccl-tests on the same cluster with similar means, implicating the ML framework's coordination and overlap as the main source of collective variation.
  • Spatial correlation by rank and host, not run-level aggregates, is what localizes a straggler: in the 16-GPU case affected communicators sustained about 10 GB/s while healthy ones ran near 170 GB/s.
  • Ingesting a 2,048-GPU trace takes under a minute with a 26x space reduction, and NCCL Inspector profiling itself costs under 2% end-to-end overhead, keeping NIXT usable in production training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The hot-kernel result gives collective-algorithm search and synthesis systems a concrete target list: enumerate the few (topology, message-size) buckets per workload and specialize tuning to those, rather than searching the full space.
  • The framework-variation claim can be settled decisively by the instrumented replay the paper plans: replay recorded collective sequences outside the training loop, with and without overlap annotation, to split the excess variation between coordination stalls and fabric contention.
  • Per-host coefficient-of-variation over time could serve as a lightweight production health signal: sustained per-host CV spikes across communicators would flag a host for evacuation before job-level throughput visibly degrades.
  • Joining NIXT's localization output with DCGM and switch link-error telemetry, which the paper explicitly defers, would convert host attribution into true root cause (degraded link versus thermal versus device).
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents NIXT, an exporter/analysis layer built on top of NCCL Inspector. NIXT ingests per-rank JSON collective-communication logs into a DuckDB/Parquet store, introduces a metadata taxonomy (Identifier/Configuration/Workload/Measurement/Counter), and provides SQL-based summary and correlation primitives (temporal, spatial, resource, other). The authors evaluate NIXT on Nemotron-4 15B and 340B pretraining runs on up to 2,048 H100 GPUs, showing communication-structure observations, scaling behavior, variation attribution, and a straggler case study. The paper also claims under-2% end-to-end profiling overhead and reports log-compression and query-latency numbers.

Significance. If the claims hold, NIXT addresses a genuine gap: NCCL Inspector emits high-volume, high-dimensional data, and the paper shows a concrete, production-oriented path from raw logs to operator-relevant summaries. The artifact is a real strength: source code is provided with an end-to-end demo, a figure-to-script mapping, and clear instructions, even though the production traces themselves are confidential and cannot be redistributed. The descriptive case studies at 2,048 GPUs are valuable in their own right. However, the paper's two most load-bearing quantitative/causal claims — the under-2% overhead statement and the host-level straggler attribution — are not currently supported by the presented evidence. Those need to be addressed before the paper can be accepted as a rigorous systems/observability contribution.

major comments (3)
  1. [§3.4] The claim that NCCL Inspector profiling costs 'under 2% end-to-end overhead' is central to the paper's production-usable positioning, but no measurement methodology or results are given. There is no description of the baseline, the workloads, the number of runs, the variance, or how 'end-to-end' is defined. Please add an overhead measurement subsection with concrete numbers (e.g., throughput or step-time comparison with and without Inspector enabled), or explicitly retract the quantitative claim and state overhead as 'not measured here.'
  2. [§7, Table 4, Figures 16–17] The straggler attribution to host composition is not established by the presented experiment. The three runs are separate job submissions with 'identical configuration,' but the assumption that 'any bandwidth change reflects the host composition' requires ruling out network contention, thermal state, and scheduling/runtime differences. The op-count mismatch in Table 4 (1,550 for Healthy and Straggler-A vs. 26,350 for Straggler-B) shows the runs are not directly comparable as collected. The paper itself concedes in the same section that NIXT does not probe the fabric and that joining with DCGM/link-error telemetry is future work. As it stands, the paper can claim localization of a sustained bandwidth-collapse signature, but not root-cause attribution to specific host resources. Please either add a controlled/interleaved experimental design with fabric and thermal telemetry, or soften the
  3. [§6.1, Table 2] The attribution of long-tail collective variation to 'the ML framework' is stronger than the evidence supports. The paper's own Limitation paragraph states that the production-vs-nccl-tests CV gap conflates coordination stalls with fabric contention and that the current data cannot separate the two. Yet §6 is titled 'Attributing Source of Collective Long-tail Variation' and the observation states that variation is 'likely caused by the ML framework.' The former is a reasonable hypothesis, but it should be framed as such, with the planned replay experiment described as the decisive test, or the title and conclusion should be adjusted to match the evidence.
minor comments (4)
  1. [Figure 6] The legend lists 'mixed r1024,n256,AllGather,1.54MB' twice. This appears to be a copy-paste error; please correct and ensure all curve labels are distinct.
  2. [§5.2.3, Figure 10a] The text explains the non-monotonic captured volume as an artifact of NCCL Inspector sampling, but no sampling rate or collection-window information is given. Since the volume axis is used to claim an increasing trend, please report how sampling was configured and whether the captured window was comparable across runs.
  3. [§3.4] The sentence '56 seconds end-to-end,24seconds for the Parquet-write stage' has a missing space. Also, please state the hardware on which the ingest time was measured (it says a 64-core commodity node, but the CPU model and memory are not given).
  4. [Table 4] The op-count discrepancy between Healthy/Straggler-A (1,550) and Straggler-B (26,350) should be explained in the text. If this is due to different run durations or sampling rates, that information is essential for interpreting the bandwidth comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NIXT's processing pipeline and case-study evaluations are grounded in external data and benchmarks, and no prediction or derived result reduces to its own inputs by construction.

full rationale

The paper's derivation chain is self-contained. NIXT ingests raw NCCL Inspector JSON, applies a descriptive taxonomy and SQL-based analysis primitives, and reports summary/correlation observations; there is no fitted parameter that is later renamed as a prediction. The Section 6 variation-attribution study compares production training against nccl-tests, an external synthetic benchmark, and explicitly limits the conclusion by acknowledging the overlap-versus-contention confound. The Section 7 straggler study uses an independent experimental condition (a host pool known to contain a straggler versus a healthy baseline) to validate NIXT's localization; the straggler status is an input to the experimental design, not an output of NIXT. The only self-citation appears in the background claim that collective communication consumes 20–40% of training time (ref [19]), and it is corroborated by other citations, so it is not load-bearing. Acknowledged limitations about fabric health and thermal/link-level attribution are causal-inference caveats, not circular reductions. No equation, derived metric, or attribution step is equivalent by construction to the paper's central claims.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

NIXT does not introduce entities like new particles or forces. The only invented construct is the coll_type interconnect classification, which is derived from communicator placement rather than direct transport information. The free parameters are analysis thresholds and visualization choices, none of which are fitted to a target result. The main axioms are domain assumptions about the accuracy of NCCL Inspector data and the experimental validity of the straggler comparison.

free parameters (4)
  • large-message threshold = 1 MiB
    Used in Figures 6, 7, and 12b to select 'large-message' collectives. Chosen by hand, not fitted to data. Affects which collectives appear in the performance ECDFs.
  • time bin width = 5 seconds
    Used in temporal correlation plots (Figures 14 and 17). Chosen for readability; affects the granularity of observed temporal patterns.
  • communicator block size = 8 or 32 ranks
    Used in Figure 13 to group ranks into blocks for visualization. Chosen by hand and does not affect the underlying analysis logic.
  • interconnect class thresholds = n_ranks=1 -> single-rank; nnodes=1 -> nvlink-only; n_ranks=nnodes -> NIC-only; else mixed
    The derived metric coll_type (Section 3.1) uses these simple rules instead of actual packet-level transport information. This is an approximation that could misclassify collectives that use multiple link types.
assumptions (4)
  • domain assumption NCCL Inspector's sampled execution time and bandwidth statistics are accurate and representative of the actual collective operations.
    NIXT performs all analysis on the records produced by the NCCL Inspector plugin. If the profiler sampling misses straggler events or computes bus bandwidth incorrectly, every derived insight inherits that error. Invoked throughout, beginning in Section 3.
  • domain assumption The bus bandwidth reported by NCCL is a meaningful measure of collective communication performance.
    The paper uses coll_busbw_gbs as the main performance metric for correlation and anomaly analysis. This assumes the NCCL bandwidth definition is consistent across collective types and message sizes. Referenced in Section 3 and used in all case studies.
  • ad hoc to paper The only difference between the healthy and straggler runs in Section 7 is the host composition of the job.
    The experimental design assumes that the workload is fixed and the known straggler host is the sole source of the observed slowdown. The paper lists reliable confirmation with DCGM/link-error telemetry as future work, so this is a load-bearing assumption.
  • domain assumption Nemotron-4 pretraining on an H100 cluster is representative of large-scale LLM training workloads.
    The case studies use only Nemotron-4 (15B and 340B) runs. The paper argues the taxonomy and primitives are workload-agnostic, but the validation evidence is limited to this one model family and cluster type. Used for the generalizability statement in Section 9.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NIXT: A NCCL Inspector Exporter Tool for Observability of Collective Communication in Large Model Training." pith.science (2026). https://pith.science/paper/WYGFBGML

@misc{pith2026260801449,
  author       = {Pith},
  title        = {Pith review of: NIXT: A NCCL Inspector Exporter Tool for Observability of Collective Communication in Large Model Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WYGFBGML}},
  note         = {Machine review of arXiv:2608.01449}
}
read the original abstract

As machine learning workloads scale, it is increasingly important to gain more observability into the performance of collective communication to easily identify performance vari- ations and accelerate root cause identification. Towards this goal, the Nvidia Collective Communication Library (NCCL) introduced NCCL Inspector, a profiler plugin that provides lightweight and continuous reporting of NCCL communication performance statistics. However, the large volume of data collected by NCCL Inspector can be difficult to assess and to extract actionable insights from. This paper presents NIXT, a NCCL Inspector Exporter Tool that improves the observability of collective communication by providing readily accessible analysis and actionable insights from NCCL Inspector profiling. To highlight the benefits of our Exporter Tool, we present a case study of Nemotron-4 LLM pretraining on an Nvidia H100 GPU cluster with up to 2,048 GPUs, demonstrate observability into how communication phases change with ML parallelism and GPU scale, and perform attribution of performance variation and root cause analysis of stragglers.

Figures

Figures reproduced from arXiv: 2608.01449 by the authors.

Figure 1
Figure 1. Virtual and physical view of NCCL collective communication. The [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The NIXT NCCL Inspector Exporter Tool workflow [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Analysis primitives implemented by the NCCL Inspector Exporter as SQL queries over the Parquet-backed log [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Nemotron-4 340B @ 2,048 GPUs: total communi [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Resource correlation analysis of communicator [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Bus-bandwidth ECDFs for dominant large-message [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Bus-bandwidth ECDFs for dominant large-message [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 10
Figure 10. Figure 10: Nemotron-4 15B scaling with GPU count. In (a), [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 12
Figure 12. Figure 12: Nemotron-4 at 2,048 GPUs: 15B versus 340B [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 13
Figure 13. Figure 13: Bus bandwidth at every (communicator, rank) cell [PITH_FULL_IMAGE:figures/full_fig_p008_13.png]
Figure 14
Figure 14. Figure 14: Bus bandwidth and exec time vs wall-clock time, [PITH_FULL_IMAGE:figures/full_fig_p009_14.png]
Figure 15
Figure 15. Figure 15: Nemotron-4 at 2,048 GPUs: top row shows the [PITH_FULL_IMAGE:figures/full_fig_p009_15.png]
Figure 16
Figure 16. Figure 16: Per-ID bandwidth for mixed AllGather 24 MiB. [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]
Figure 17
Figure 17. Figure 17: Bandwidth versus time in 5 s bins. Left: healthy. Right: straggler run. [PITH_FULL_IMAGE:figures/full_fig_p012_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 67 canonical work pages

  1. [1]

    TraceWeaver: Distributed request tracing for microser- vices without application modification,

    S. Ashok, V. Harsh, B. Godfrey, R. Mittal, S. Parthasarathy, and L. Shwartz, “TraceWeaver: Distributed request tracing for microser- vices without application modification, ” inProceedings of the ACM SIGCOMM 2024 Conference. ACM, 2024, pp. 828–842

    work page 2024

  2. [2]

    Crux: GPU-efficient communication scheduling for deep learning training,

    J. Cao, Y. Guan, K. Qian, J. Gao, W. Xiao, J. Dong, B. Fu, D. Cai, and E. Zhai, “Crux: GPU-efficient communication scheduling for deep learning training, ” inProceedings of the ACM SIGCOMM 2024 Conference, ser. ACM SIGCOMM ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 1–15. [Online]. Available: https://doi.org/10.1145/3651890.3672239

    arXiv 2024

  3. [3]

    Automatic root cause analysis via large language models for cloud incidents,

    Y. Chen, H. Xie, M. Ma, Y. Kang, X. Gao, L. Shi, Y. Cao, X. Gao, H. Fan, M. Wen, J. Zeng, S. Ghosh, X. Zhang, C. Zhang, Q. Lin, S. Rajmohan, D. Zhang, and T. Xu, “Automatic root cause analysis via large language models for cloud incidents, ” inProceedings of the Nineteenth European Conference on Computer Systems (EuroSys 2024). ACM, 2024, pp. 674–688

    work page 2024

  4. [4]

    MSCCLang: Microsoft collective communication language,

    M. Cowan, S. Maleki, M. Musuvathi, O. Saarikivi, and Y. Xiong, “MSCCLang: Microsoft collective communication language, ” inPro- ceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2023), Volume 2. ACM, 2023, pp. 502–514

    work page 2023

  5. [5]

    Enhance hardware reliability for AI acceleration at scale,

    Cudo Compute, “Enhance hardware reliability for AI acceleration at scale, ” 2024. [Online]. Available: https://www.cudocompute.com/blog /enhance-hardware-reliability-for-ai-acceleration-at-scale

    work page 2024

  6. [6]

    FLARE: Anomaly diagnostics for divergent LLM training in GPU clusters of Thousand-Plus scale,

    W. Cui, J. Zhang, H. Zhao, C. Liu, J. Sha, B. Sang, B. He, M. Guo, and Q. Chen, “FLARE: Anomaly diagnostics for divergent LLM training in GPU clusters of Thousand-Plus scale, ” in23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). Renton, WA: USENIX Association, May 2026, pp. 1021–1035. [Online]. Available: https://www.usenix.o...

    work page 2026

  7. [7]

    Minder: Faulty machine detection for large-scale distributed model training,

    Y. Deng, X. Shi, Z. Jiang, X. Zhang, L. Zhang, Z. Zhang, B. Li, Z. Song, H. Zhu, G. Liu, F. Li, S. Wang, H. Lin, J. Ye, and M. Yu, “Minder: Faulty machine detection for large-scale distributed model training, ” in22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). USENIX Association, 2025, pp. 505–521

    work page 2025

  8. [8]

    Mycroft: Tracing dependencies in collective communication towards reliable LLM training,

    Y. Deng, L. Zhang, Q. Wang, X. Zhi, X. Zhang, Z. Jiang, H. Xu, L. Wang, Z. Song, G. Liuet al., “Mycroft: Tracing dependencies in collective communication towards reliable LLM training, ” in Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, 2025, pp. 254–269

    work page 2025

Show all 76 references
  1. [9]

    DFTracer: An analysis-friendly data flow tracer for AI-driven workflows,

    H. Devarajan, L. Pottier, K. Velusamy, H. Zheng, I. Yildirim, O. Kogiou, W. Yu, A. Kougkas, X. Sun, J. Yeom, and K. M. Mohror, “DFTracer: An analysis-friendly data flow tracer for AI-driven workflows, ” in Proceedings of the International Conference for High Performance Comput...

    2024

  2. [10]

    Evolution of aegis: Fault diagnosis for AI model training service in production,

    J. Dong, K. Qian, P. Zhang, Z. Zheng, L. Chen, F. Feng, Y. Xu, Y. Zhu, G. Lu, X. Liet al., “Evolution of aegis: Fault diagnosis for AI model training service in production, ” in22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). USENIX Association, ...

    2025

  3. [11]

    eBPF - introduction, tutorials & community resources,

    eBPF Community, “eBPF - introduction, tutorials & community resources, ” https://ebpf.io/, 2024

    2024

  4. [12]

    Hardware failures won’t limit AI scaling,

    A. Erben and E. Erdil, “Hardware failures won’t limit AI scaling, ”

  5. [13]

    Low-overhead distributed application observation with DeepTrace: Achieving ac- curate tracing in production systems,

    Y. Geng, H. Zhang, Z. Wu, Y. Li, J. Wang, and X. Yin, “Low-overhead distributed application observation with DeepTrace: Achieving ac- curate tracing in production systems, ” inProceedings of the ACM SIGCOMM 2025 Conference. ACM, 2025, pp. 1056–1069

    2025

  6. [14]

    GPU reliability in AI clusters: A study of failure modes and effects,

    S. Gupta, “GPU reliability in AI clusters: A study of failure modes and effects, ”Journal Of Engineering And Computer Sciences, vol. 4, no. 6, pp. 298–306, 2025

    2025

  7. [15]

    Characterization of large language model development in the datacenter,

    Q. Hu, Z. Ye, Z. Wang, G. Wang, M. Zhang, Q. Chen, P. Sun, D. Lin, X. Wang, Y. Luo, Y. Wen, and T. Zhang, “Characterization of large language model development in the datacenter, ” in21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). USENIX Associa...

    2024

  8. [16]

    Mint: Cost-efficient tracing with all requests collection via commonality and variability analysis,

    H. Huang, C. Chen, K. Chen, P. Chen, G. Yu, Z. He, Y. Wang, H. Zhang, and Q. Zhou, “Mint: Cost-efficient tracing with all requests collection via commonality and variability analysis, ” inProceedings of the 30th ACM International Conference on Architectural Support for Program...

    2025

  9. [17]

    Neutrino: Fine-grained GPU kernel profiling via programmable probing,

    S. Huang and C. Wu, “Neutrino: Fine-grained GPU kernel profiling via programmable probing, ” in19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2025). USENIX Association, 2025, pp. 331–355. [Online]. Available: https: //www.usenix.org/conference/osdi2...

    2025

  10. [18]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism,

    Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wuet al., “Gpipe: Efficient training of giant neural networks using pipeline parallelism, ”Advances in neural information processing systems, vol. 32, 2019

    2019

  11. [19]

    Pccl: Energy-efficient LLM training with power-aware collective communication,

    Z. Jia, L. N. Bhuyan, and D. Wong, “Pccl: Energy-efficient LLM training with power-aware collective communication, ” in2024 IEEE 42nd International Conference on Computer Design (ICCD). IEEE, 2024, pp. 84–91

    2024

  12. [20]

    Training with confidence: Catching silent errors in deep learning training with automated proactive checks,

    Y. Jiang, Z. Zhou, B. Xu, B. Liu, R. Xu, and P. Huang, “Training with confidence: Catching silent errors in deep learning training with automated proactive checks, ” in19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2025). USENIX Association, 2025, p...

    2025

  13. [21]

    MegaScale: Scaling large language model training to more than 10,000 GPUs,

    Z. Jiang, H. Lin, Y. Zhong, Q. Huang, Y. Chen, Z. Zhang, Y. Peng, X. Li, C. Xie, S. Nonget al., “MegaScale: Scaling large language model training to more than 10,000 GPUs, ” in21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). USENIX Association, 2...

    2024

  14. [22]

    TraceFlow: Efficient trace analysis for large-scale parallel applications via interaction pattern-aware trace distribution,

    Y. Jin, X. Shui, M. Zhai, Z. Zong, F. Zhang, F. Wolf, and J. Zhai, “TraceFlow: Efficient trace analysis for large-scale parallel applications via interaction pattern-aware trace distribution, ” inProceedings of the International Conference for High Performance Computing, Netwo...

    2025

  15. [23]

    TCCL: Discovering better communication paths for PCIe GPU clusters,

    H. Kim, J. Ryu, and J. Lee, “TCCL: Discovering better communication paths for PCIe GPU clusters, ” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2024), Volume 3. ACM, 2024, pp. 999–1015

    2024

  16. [24]

    One weird trick for parallelizing convolutional neural networks,

    A. Krizhevsky, “One weird trick for parallelizing convolutional neural networks, ”arXiv preprint arXiv:1404.5997, 2014

    2014 arXiv

  17. [25]

    Characterizing compute- communication overlap in gpu-accelerated distributed deep learning: Performance and power implications,

    S. Lee, J. Oh, S. Go, and D. Mahajan, “Characterizing compute- communication overlap in gpu-accelerated distributed deep learning: Performance and power implications, ” in2025 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2025, pp. 353–355

    2025

  18. [26]

    Gshard: Scaling giant models with conditional computation and automatic sharding,

    D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding, ”arXiv preprint arXiv:2006.16668, 2020

    2006 arXiv

  19. [27]

    Revisiting reliability in large-scale machine learning research clusters,

    B. Li, S. Samsi, V. Gadepally, and D. Tiwari, “Revisiting reliability in large-scale machine learning research clusters, ”arXiv preprint arXiv:2410.21680, 2024

    2024 arXiv

  20. [28]

    Malleus: Straggler-resilient hybrid parallel training of large-scale mod- els via malleable data and model parallelization,

    H. Li, A. Kadav, E. Kruus, P. Mattson, D. H. Chauet al., “Malleus: Straggler-resilient hybrid parallel training of large-scale mod- els via malleable data and model parallelization, ”arXiv preprint arXiv:2410.13333, 2024

    2024 arXiv

  21. [29]

    Understanding stragglers in large model training using what-if analysis,

    J. Lin, Z. Jiang, Z. Song, S. Zhao, M. Yu, Z. Wang, C. Wang, Z. Shi, X. Shi, W. Jia, Z. Liu, S. Wang, H. Lin, X. Liu, A. Panda, and J. Li, “Understanding stragglers in large model training using what-if analysis, ” in19th USENIX Symposium on Operating Systems Design and Implem...

    2025

  22. [30]

    Ring attention with blockwise transformers for near-infinite context,

    H. Liu, M. Zaharia, and P. Abbeel, “Ring attention with blockwise transformers for near-infinite context, ”arXiv preprint arXiv:2310.01889, 2023

    2023 arXiv

  23. [31]

    R-Pingmesh: A service-aware RoCE network monitoring and diagnostic system,

    K. Liu, Z. Jiang, J. Zhang, S. Guo, X. Zhang, Y. Bai, Y. Dong, F. Luo, Z. Zhang, L. Wanget al., “R-Pingmesh: A service-aware RoCE network monitoring and diagnostic system, ” inProceedings of the ACM SIGCOMM 2024 Conference, 2024, pp. 554–567

    2024

  24. [32]

    Deriving semantic checkers from tests to detect silent failures in production distributed systems,

    C. Lou, D. S. Parikesit, Y. Huang, Z. Yang, S. Diwangkara, Y. Jing, A. I. Kistijantoro, D. Yuan, S. Nath, and P. Huang, “Deriving semantic checkers from tests to detect silent failures in production distributed systems, ” in19th USENIX Symposium on Operating Systems Design and...

    2025

  25. [33]

    One-size- fits-none: Understanding and enhancing slow-fault tolerance in modern distributed systems,

    R. Lu, Y. Lu, Y. Jiang, G. Xue, and P. Huang, “One-size- fits-none: Understanding and enhancing slow-fault tolerance in modern distributed systems, ” in22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 2025). USENIX Association, 2025, pp. 359–378. [Onl...

    2025

  26. [34]

    Dr. DNA: Combating silent data corruptions in deep learning using distribution of neuron activations,

    D. Ma, F. F. Lin, A. Desmaison, J. Coburn, D. Moore, S. Sankar, and X. Jiao, “Dr. DNA: Combating silent data corruptions in deep learning using distribution of neuron activations, ” inProceedings of the 29th ACM International Conference on Architectural Support for Programming...

    2024

  27. [35]

    Pipedream: generalized pipeline parallelism for DNN training,

    D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: generalized pipeline parallelism for DNN training, ” inProceedings of the 27th ACM Symposium on Operating Systems Principles, ser. SOSP ’19. New York, ...

    2019

  28. [36]

    How we build reliable clusters,

    Nebius, “How we build reliable clusters, ” 2024. [Online]. Available: https://nebius.com/blog/posts/how-we-build-reliable-clusters

    2024

  29. [37]

    DCGM Exporter,

    NVIDIA Corporation, “DCGM Exporter, ” 2024. [Online]. Available: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/k8s/containers/dc gm-exporter

    2024

  30. [38]

    Networking, Reliability, and Observability at Scale with NCCL 2.24,

    NVIDIA Corporation, “Networking, Reliability, and Observability at Scale with NCCL 2.24, ” 2024. [Online]. Available: https: //developer.nvidia.com/blog/networking-reliability-and-observabili ty-at-scale-with-nccl-2-24/

    2024

  31. [39]

    OpenTelemetry Best Practices for AI Workloads,

    NVIDIA Corporation, “OpenTelemetry Best Practices for AI Workloads, ” 2024. [Online]. Available: https://docs.nvidia.com/networ king-ethernet-software/knowledge-base/Configuration-and-Usage /Monitoring/OpenTelemetry-Best-Practices/

    2024

  32. [40]

    Enhancing Communication Observability of AI Workloads with NCCL Inspector,

    NVIDIA Corporation, “Enhancing Communication Observability of AI Workloads with NCCL Inspector, ” 2025. [Online]. Available: https://developer.nvidia.com/blog/enhancing-communication-obser vability-of-ai-workloads-with-nccl-inspector/

    2025

  33. [41]

    NCCL Inspector Plugin,

    NVIDIA Corporation, “NCCL Inspector Plugin, ” 2025. [Online]. Available: https://github.com/NVIDIA/nccl/tree/v2.28/ext-profiler/ins pector

    2025

  34. [42]

    NCCL inspector: Profiling and monitoring guide,

    NVIDIA Corporation, “NCCL inspector: Profiling and monitoring guide, ” NVIDIA Corporation, Tech. Rep., 2025, internal Technical Documentation

    2025

  35. [43]

    NCCL: Optimized primitives for collective multi-GPU communication,

    NVIDIA Corporation, “NCCL: Optimized primitives for collective multi-GPU communication, ” https://github.com/NVIDIA/nccl/tree/ma ster, 2025, accessed: 2025-07-20

    2025

  36. [44]

    Performance reported by NCCL tests,

    NVIDIA Corporation, “Performance reported by NCCL tests, ” https: //github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE .md, 2025, accessed: 2025-07-20

    2025

  37. [45]

    Real-Time Performance Monitoring and Faster Debugging with NCCL Inspector and Prometheus,

    NVIDIA Corporation, “Real-Time Performance Monitoring and Faster Debugging with NCCL Inspector and Prometheus, ” 2025. [Online]. Available: https://developer.nvidia.com/blog/real-time-per formance-monitoring-and-faster-debugging-with-nccl-inspector-a nd-prometheus/

    2025

  38. [46]

    Straggler-aware distributed learning: Communication-computation latency trade-off,

    E. Ozfatura, D. Gunduz, and S. Ulukus, “Straggler-aware distributed learning: Communication-computation latency trade-off, ” inIEEE Transactions on Information Theory, 2020

    2020

  39. [47]

    Com- prehensive deadlock prevention for GPU collective communication,

    L. Pan, J. Liu, Y. Fu, J. Yuan, R. Zhang, P. Li, and Z. Xiao, “Com- prehensive deadlock prevention for GPU collective communication, ” inProceedings of the Twentieth European Conference on Computer Systems (EuroSys 2025). ACM, 2025, pp. 541–557

    2025

  40. [48]

    T3: Transparent tracking & triggering for fine-grained overlap of compute & collectives,

    S. Pati, S. Aga, M. Islam, N. Jayasena, and M. D. Sinclair, “T3: Transparent tracking & triggering for fine-grained overlap of compute & collectives, ” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Syste...

    2024

  41. [49]

    What is Prometheus,

    Prometheus Authors, “What is Prometheus, ” 2025. [Online]. Available: https://prometheus.io/docs/introduction/overview/

    2025

  42. [50]

    Node exporter: Exporter for machine metrics,

    Prometheus Developers, “Node exporter: Exporter for machine metrics, ” https://github.com/prometheus/node_exporter, 2024

    2024

  43. [51]

    PyTorch NCCL Flight Recorder,

    PyTorch Team, “PyTorch NCCL Flight Recorder, ” 2024. [Online]. Available: https://discuss.pytorch.org/t/pytorch-nccl-flight-recorder/ 207410

    2024

  44. [52]

    DuckDB: an embeddable analytical database,

    M. Raasveldt and H. Mühleisen, “DuckDB: an embeddable analytical database, ” inProceedings of the 2019 International Conference on Management of Data (SIGMOD). Association for Computing Machinery, 2019, pp. 1981–1984

    2019

  45. [53]

    Horovod: fast and easy distributed deep learning in TensorFlow,

    A. Sergeev and M. Del Balso, “Horovod: fast and easy distributed deep learning in TensorFlow, ”arXiv preprint arXiv:1802.05799, 2018

    2018 arXiv

  46. [54]

    TACCL: Guiding collective algorithm synthesis using communication sketches,

    A. Shah, V. Chidambaram, M. Cowan, S. Maleki, M. Musuvathi, T. Mytkowicz, J. Nelson, and O. Saarikivi, “TACCL: Guiding collective algorithm synthesis using communication sketches, ” in20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2023). USENIX Asso...

    2023

  47. [55]

    Network-centric distributed tracing with DeepFlow: Troubleshooting your microservices in zero code,

    J. Shen, H. Zhang, Y. Xiang, X. Shi, X. Li, Y. Shen, Z. Zhang, Y. Wu, X. Yin, J. Wang, M. Xu, Y. Li, J. Yin, J. Song, Z. Li, and R. Nie, “Network-centric distributed tracing with DeepFlow: Troubleshooting your microservices in zero code, ” inProceedings of the ACM SIGCOMM 2023...

    2023

  48. [56]

    Megatron-lm: Training multi-billion parameter language mod- els using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-lm: Training multi-billion parameter language mod- els using model parallelism, ”arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  49. [57]

    Loom: Efficient capture and querying of high-frequency telemetry,

    F. Solleza, S. Li, W. Sun, R. Tang, M. Schwarzkopf, A. Crotty, D. Cohen, N. Tatbul, and S. Zdonik, “Loom: Efficient capture and querying of high-frequency telemetry, ” inProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP 2025). ACM, 2025, pp. 512–528

    2025

  50. [58]

    Optimization of collective communication operations in MPICH,

    R. Thakur, R. Rabenseifner, and W. Gropp, “Optimization of collective communication operations in MPICH, ”The International Journal of High Performance Computing Applications, vol. 19, no. 1, pp. 49–66, 2005

    2005

  51. [59]

    MPI: a standard message passing interface,

    D. W. Walker and J. J. Dongarra, “MPI: a standard message passing interface, ”Supercomputer, vol. 12, pp. 56–68, 1996

    1996

  52. [60]

    Robust LLM training infrastructure at ByteDance,

    B. Wan, G. Liu, Z. Song, J. Wang, Y. Zhang, G. Sheng, S. Wang, H. Wei, C. Wang, W. Louet al., “Robust LLM training infrastructure at ByteDance, ” inProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, 2025, pp. 186–203

    2025

  53. [61]

    Characterizing and optimizing end-to-end systems for private inference,

    J. Wang, J. Huang, S. Lee, J. Park, and N. S. Kim, “Characterizing and optimizing end-to-end systems for private inference, ” inProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, vol. 2, 2023, pp. 781–794

    2023

  54. [62]

    Understanding silent data corruptions in a large production CPU population,

    S. Wang, G. Zhang, J. Wei, Y. Wang, J. Wu, and Q. Luo, “Understanding silent data corruptions in a large production CPU population, ” in Proceedings of the 29th Symposium on Operating Systems Principles (SOSP 2023). ACM, 2023, pp. 216–230

    2023

  55. [63]

    Hawkeye: Diagnosing RDMA network performance anomalies with PFC provenance,

    S. Wang, M. Zhang, X. Li, Q. Peng, H. Yu, Z. Wang, M. Xu, X. Hu, J. Yang, and X. Shi, “Hawkeye: Diagnosing RDMA network performance anomalies with PFC provenance, ” inProceedings of the ACM SIGCOMM 2025 Conference, 2025, pp. 481–495

    2025

  56. [64]

    EXIST: Enabling extremely efficient intra-service tracing observability in datacenters,

    X. Wang, X. Hou, C. Li, Y. Li, D. Liu, G. Xu, G. Yang, L. Zhang, Y. Wu, X. Yuan, Q. Chen, and M. Guo, “EXIST: Enabling extremely efficient intra-service tracing observability in datacenters, ” inProceedings of the 30th ACM International Conference on Architectural Support for ...

    2025

  57. [65]

    OptiReduce: Resilient and tail-optimal AllReduce for distributed deep learning in the cloud,

    E. Warraich, O. Shabtai, K. Manaa, S. Vargaftik, Y. Piasetzky, M. Kadosh, L. Suresh, and M. Shahbaz, “OptiReduce: Resilient and tail-optimal AllReduce for distributed deep learning in the cloud, ” in22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 202...

    2025

  58. [66]

    Attack of the bubbles: Straggler-Resilient pipeline parallelism for large model training,

    T. Wu, L. Cao, H. Lu, X. Jiang, Y. Yu, S. Yang, G. Yang, J. Wang, L. Qu, L. Zhang, and W. Wang, “Attack of the bubbles: Straggler-Resilient pipeline parallelism for large model training, ” in23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). Renton...

    2026

  59. [67]

    MCCS: A service-based approach to collective communication for multi-tenant cloud,

    Y. Wu, Y. Xu, J. Chen, Z. Wang, Y. Zhang, M. Lentz, and D. Zhuo, “MCCS: A service-based approach to collective communication for multi-tenant cloud, ” inProceedings of the ACM SIGCOMM 2024 Conference. ACM, 2024, pp. 679–690

    2024

  60. [68]

    SuperBench: Improving cloud AI infrastructure reliability with proactive validation,

    Y. Xiong, Y. Jiang, Z. Yang, L. Qu, G. Zhao, S. Liu, D. Zhong, B. Pinzur, J. Zhang, Y. Wanget al., “SuperBench: Improving cloud AI infrastructure reliability with proactive validation, ” in2024 USENIX Annual Technical Conference (USENIX ATC 24). USENIX Association, 2024, pp. 835–850

    2024

  61. [69]

    AutoCCL: Automated collective communication tuning for accelerating distributed and parallel DNN training,

    G. Xu, Z. Le, Y. Chen, Z. Lin, Z. Jin, Y. Miao, and C. Li, “AutoCCL: Automated collective communication tuning for accelerating distributed and parallel DNN training, ” in22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 2025). USENIX Association, 2025...

    2025

  62. [70]

    eACGM: Non-instrumented performance tracing and anomaly detection towards machine learning systems,

    R. Xu, Z. Xie, and P. Chen, “eACGM: Non-instrumented performance tracing and anomaly detection towards machine learning systems, ” in2025 IEEE/ACM 33rd International Symposium on Quality of Service (IWQoS). IEEE, 2025, pp. 1–6

    2025

  63. [71]

    Holmes: Localizing irregularities in LLM training with mega-scale GPU clusters,

    Z. Yao, P. Hu, C. Miao, X. Jia, Z. Liang, Y. Xu, C. He, H. Lu, M. Chen, X. Li, Z. He, Y. Wang, X. Zou, and J. Jiang, “Holmes: Localizing irregularities in LLM training with mega-scale GPU clusters, ” in22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI ...

    2025

  64. [72]

    FBDetect: Catching tiny performance regressions at hyperscale through in-production monitoring,

    D. Y. Yoon, Y. Wang, M. Yu, E. Huang, J. I. Jones, A. Kukkadapu, O. Kocas, J. Wiepert, K. Goenka, S. Chenet al., “FBDetect: Catching tiny performance regressions at hyperscale through in-production monitoring, ” inProceedings of the ACM SIGOPS 30th Symposium on Operating Syste...

    2024

  65. [73]

    The benefit of Hindsight: Tracing edge-cases in distributed systems,

    L. Zhang, Z. Xie, V. Anand, Y. Vigfusson, and J. Mace, “The benefit of Hindsight: Tracing edge-cases in distributed systems, ” in20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2023). USENIX Association, 2023, pp. 321–339. [Online]. Available: https:...

    2023

  66. [74]

    DeepContext: A context-aware, cross-platform, and cross-framework tool for performance profiling and analysis of deep learning workloads,

    Q. Zhao, H. Wu, Y. Hao, Z. Ye, J. Li, X. Liu, and K. Zhou, “DeepContext: A context-aware, cross-platform, and cross-framework tool for performance profiling and analysis of deep learning workloads, ” in Proceedings of the 30th ACM International Conference on Architectural Supp...

    2025

  67. [75]

    Accelerating model training on Ascend chips: An industrial system for profiling, analysis and optimization,

    Y. Zhou, Z. Wang, Z. Wang, R. Zhang, C. Tian, X. Wang, W. Dou, G. Chen, B. Wang, Y. Tian, Y. Zhang, H. Wang, F. Wei, B. Sun, J. Zhang, B. She, T. Su, Y. Yao, C. Li, Z. Zhang, Y. Wang, B. Zhou, and G. Liu, “Accelerating model training on Ascend chips: An industrial system for p...

    2025

  68. [2024]

    Available: https://epoch.ai/blog/hardware-failures-w ont-limit-ai-scaling

    [Online]. Available: https://epoch.ai/blog/hardware-failures-w ont-limit-ai-scaling

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.