Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

COSMIC: Enabling Full-Stack Co-Design and Optimization of Distributed Machine Learning Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Full-stack search of distributed ML systems beats isolated-stack tuning by up to 48x.

desk verdict A useful framework with a misleading headline: the 1.50–48.41x claim is measured against a per-bandwidth objective that full-stack search trivially wins by cutting bandwidth to the floor. read the letter →

arxiv 2505.15020 v1 pith:WFGPR6V7 submitted 2025-05-21 cs.DC

Aditi Raju , Jared Ni , William Won , Changhai Man , Srivatsan Krishnan , Srinivas Sridharan , Amir Yazdanbakhsh , Tushar Krishna
show 1 more author
Vijay Janapa Reddi
This is my paper · ORCID
classification cs.DC
keywords full-stackco-designdistributedmachinelearningdesignspaceexplorationparametersetarchitectureagent-basedsearchcollectivecommunicationnetworktopologyoptimizationparallelizationstrategies
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that jointly searching workload parallelization, collective communication algorithms, network topology, and compute parameters finds substantially better distributed-training configurations than tuning any one design stack alone. To make this tractable, the paper introduces Parameter Set Architecture (PSA), a schema that separates the design space from the details of the search agent, and COSMIC, an environment that couples a detailed distributed-training simulator with several ML search agents. Across simulated clusters of 512 to 2048 NPUs and transformer models from 13B to 175B parameters, full-stack search beat isolated single-stack search by 1.50-48.41x while discovering eight distinct, non-obvious high-performance configurations. The best full-stack design also shifts with the optimization objective, which is evidence that the system layers are tightly coupled.

What carries the argument

The load-bearing abstraction is Parameter Set Architecture (PSA), a schema analogous to an instruction set architecture in that it defines the interface between a search agent and the underlying system. PSA has three parts: a parameter set covering workload, collective, network, and compute knobs; value ranges for each parameter; and constraints capturing cross-parameter dependencies such as the product of parallelization dimensions being bounded by the number of NPUs. A Parameter Set Scheduler reads this schema and automatically configures both the simulator environment (action space, observation space, constraints) and the search agent (parameter types, step sizes, reward function), so that a domain expert never touches agent internals. This decoupling is what allows the same framework to run workload-only, collective-only, network-only, or full-stack searches and to swap in different ML agents without code changes.

What would settle it

Run the paper's best full-stack configuration and its best single-stack configuration for the same model on a real 512-, 1024-, or 2048-NPU cluster, or at least on full-model simulation without layer truncation, and measure end-to-end training time; if the full-stack configuration does not maintain a comparable advantage over the best single-stack configuration, the central claim fails.

Watch

Extended reading notes

Core claim

The discovery is that cross-stack interactions dominate distributed-ML performance: the configurations found by full-stack optimization are qualitatively different from, and reliably better than, the ones found by optimizing any single stack. In the paper's experiments on GPT3-175B, restricting the search to workload-only, collective-only, or network-only knobs yields training runtimes 1.50-48.41x worse per bandwidth/NPU and 3.94-127.17x worse per network dollar cost than the full-stack result on 512- and 1024-NPU systems. On a 2048-NPU cluster, full-stack optimization beats workload-only optimization by 1.71-5.05x across global batch sizes from 1024 to 16384, with larger gains for the larger model. The full-stack optimum is objective-dependent: when the reward is runtime per bandwidth, the agent selects a topology and workload split that differ sharply from those chosen under a runtime-per-network-cost reward, and the collective algorithm choices change accordingly. Across four search algorithms, agents converge to the same optimal reward but reach it through different parameter settings, yielding eight non-obvious high-performance design points.

Load-bearing premise

The reported speedups assume the simulator's runtime estimates are faithful for the full models on real clusters, because the paper simulates only four layers per model and re-scales latency and memory, and no discovered configuration was run on real hardware.

Editorial extensions

If this is right

  • Design-space exploration for distributed ML should be treated as a joint search over workload, collective, network, and compute knobs, not as sequential or isolated tuning passes.
  • The optimal distributed system depends on the objective: runtime-per-bandwidth and runtime-per-network-cost reward functions lead to different topologies, workload splits, and collective algorithms.
  • The advantage of full-stack co-design grows with model scale: GPT3-175B shows larger relative improvements than ViT-Large on the same 2048-NPU system.
  • Because multiple agents converge to equal-reward but structurally different configurations, system designers can choose among near-optimal points by secondary criteria such as cost, complexity, or deployment constraints.
  • The PSA abstraction makes agent-agnostic search possible, so adding a new search algorithm family does not require reconfiguring the simulator interface.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the simulation-to-hardware transfer holds, the PSA abstraction could become a reusable interface for co-design across other simulators and real testbeds, letting different teams share a common design-space definition.
  • The equal-reward diversity of discovered configurations suggests a Pareto surface; a multi-objective version of the search could map the cost-latency tradeoff explicitly instead of collapsing it into a single reward scalar.
  • The layer-truncation approximation (four simulated layers re-scaled to the full model) is the most likely place for simulator error to enter; running one full-model validation on a real cluster would bound that error and is the natural next test.
  • Dynamic factors such as stragglers, faults, and varying batch compositions are not modeled, so cross-layer interactions may matter even more in production settings than in the static results reported here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces COSMIC, a framework that couples ASTRA-sim with ArchGym through a Parameter Set Architecture (PSA) abstraction, and uses ML search agents to co-optimize workload parallelization, collective algorithms, network topology/bandwidth, and compute parameters for distributed transformer training and inference. The authors report that full-stack optimization outperforms single-stack optimization by 1.50-48.41x on 'runtime per BW/NPU' and by 3.40-127.17x on 'runtime per network cost' across three simulated systems, and they present a scalability study plus two co-design use cases. The central claim is that jointly searching multiple design stacks yields substantially better configurations and reveals non-obvious design points.

Significance. If the results held under a correctly specified objective and at full model scale, COSMIC would be a useful addition to the distributed-ML design-space exploration literature: the PSA abstraction is a clean separation-of-concerns idea, the agent-agnostic integration is demonstrated with four search algorithms, and the observation that multiple distinct configurations achieve similar reward is practically valuable. These strengths are real but are currently overshadowed by two problems: the printed reward function does not match the stated objective, and the headline speedups are measured in a metric that full-stack search can improve by reducing bandwidth alone. The paper also provides no artifact, no error bars, and no hardware validation to support the simulation-based speedups, so the quantitative central claim is not yet supported.

major comments (4)
  1. [Section 5.4] The reward function for 'Perf per BW/NPU' is printed as reward = 1/sqrt((Latency x sum(BW per Dim) - 1)^2). Maximizing this reward drives Latency x sum(BW) toward 1, not toward 0; the prose says COSMIC minimizes the product of latency and bandwidth. The 'minus-one offset' does not prevent divide-by-zero; it introduces a reward singularity at product = 1. Because every result in Section 6 uses this reward, the formula and the prose cannot both be correct, and the quantitative claims cannot be interpreted until this discrepancy is resolved.
  2. [Section 6.1, Tables 5 and 6] The full-stack versus single-stack comparison is not apples-to-apples: full-stack search controls the bandwidth knobs, while workload-only, collective-only, and network-only baselines use the fixed bandwidths in Table 3. All reported full-stack configurations in Tables 5 and 6 select bandwidth 50 per dimension. For System 1 the fixed baseline sums to 650 units versus 200 for the full-stack configuration, and for System 2 the fixed baseline sums to 800 versus 200, giving full-stack a 3.25x or 4x reward advantage before any latency difference is considered. Thus the reported 1.50-48.41x improvements do not establish raw latency improvements; the paper must report latency or training time separately, or hold the network configuration fixed.
  3. [Section 4.4 and Table 2 footnote] The paper asserts that ASTRA-sim 'has been cross-validated against real system measurements' but gives no citation and no validation experiment. The evaluations simulate only 4 layers per model and then re-scale latency and memory to the full model, and no discovered configuration is validated on real hardware. The headline speedups and the claimed 'non-obvious configurations' therefore rest on the unverified assumption that 4-layer simulated rankings, after re-scaling, match full-model behavior. Please provide validation evidence, or explicitly characterize the sensitivity of the conclusions to the 4-layer approximation and the re-scaling procedure.
  4. [Abstract, Section 6.1, Conclusion] The central claim is stated without its metric qualifier. The abstract and conclusion say full-stack optimization delivers '1.50-48.41x higher performance', but Section 6.1 actually measures 'runtime per BW/NPU' or 'runtime per network dollar cost', not raw runtime. This overstates the finding even if the reward formula were corrected, because the denominator includes bandwidth or cost terms that full-stack search can deliberately reduce. The authors should either re-state the claim as 'performance per bandwidth/cost' or provide raw-latency results that support the stronger phrasing.
minor comments (5)
  1. [Section 6.4] The text refers to 'experiment 4' and to GPT3-175B training, but Table 6 defines Experiment 2 as inference (Chat and QA) and no Experiment 4 exists anywhere in the paper.
  2. [Table 5 caption] Table 5 reuses the caption 'System configurations used for evaluation' from Table 3; it should instead describe these as the discovered full-stack configurations, since the table reports search outputs rather than fixed baselines.
  3. [Table 1 and Section 3.2] The design-space count is not self-consistent: the constraint is written as product(DP, SP, PP) <= 1024, but the stated 286 options appear to count tuples with product exactly 1024, and multiplying the listed per-knob point counts does not give 7.69e13. Please recompute or clarify the counting procedure.
  4. [Section 5.4] The notation 'BW per Dim' is used for what Table 3 calls 'Bandwidth per Dim' and the reward expression sums these values, but the units and the 'per NPU' normalization are never defined; please clarify whether the sum is total network bandwidth or bandwidth per link, and why dividing by NPU count is omitted.
  5. [Throughout] There are several typographical and consistency issues: 'Cerberas' should be 'Cerebras', the framework name appears as both COSMIC and Cosmic, and Figure 9's parameter labels (a through o) are not mapped to a legend in the figure itself, making the discussion of which parameters vary hard to follow.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline '1.50-48.41x higher performance' is partially self-definitional: 'performance' is measured as a per-bandwidth reward whose denominator the full-stack search can shrink, while the workload-only and collective-only baselines cannot.

  1. self definitional [Section 5.4 (Optimization Objectives), Section 6.1 (Full-Stack Optimizations), Tables 3, 5 and 6]
    "reward(Perf per BW/NPU) = 1 / sqrt((Latency × Σ(BW per Dim)−1)^2). ... Cosmic aims to maximize ML performance while limiting the bandwidth allocated per NPU. To achieve this, Cosmic minimizes the product of ML execution latency and the total network bandwidth consumed per NPU. ... Bandwidth per Dim [50, 50, 50, 50]"

    The headline improvement is measured under the 'Perf per BW/NPU' reward, whose denominator contains Σ(BW per Dim). In Section 6.1, workload-only and collective-only searches keep the fixed Table 3 networks (System 1 bandwidth sums to 650; System 2 to 800), while full-stack search is free to choose bandwidth knobs, and every reported full-stack result (Tables 5 and 6) selects [50,50,50,50], summing to 200. Thus the full-stack reward receives a 3.25x (System 1) or 4x (System 2) advantage purely from the metric's denominator, before any latency difference. The quoted 1.50-48.41x 'higher performance' is therefore in part a restatement of the objective's definition rather than an emergent co-design gain.

full rationale

The paper's central result is a simulation measurement, not a theorem derived from assumptions that include the speedup. I found no load-bearing self-citation chain: ASTRA-sim [58] and ArchGym [23] are prior systems from overlapping author groups, but the paper does not invoke a uniqueness theorem, and these are open, externally checkable artifacts. Section 4.4's assertion that 'ASTRA-sim has been cross-validated against real system measurements' is, however, made without an in-paper citation and is an evidence gap rather than a circular step. The Table 2 footnote that only 4 layers are simulated and then re-scaled is likewise a validity limitation, not circularity. The significant circularity concern is the reward construction: because the performance metric is defined per total bandwidth and full-stack alone can reduce that bandwidth in the workload-only and collective-only comparisons, a substantial part of the reported improvement is forced by the metric's definition. This makes the headline claim partially self-definitional, though the full-stack search also has genuine independent content in jointly tuning parallelization, collectives, and topology. A more rigorous evaluation would report raw latency separately and use a reward that actually minimizes the stated product, not |Latency×ΣBW − 1|.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central claims rest on ASTRA-sim's simulator as ground truth, the roofline compute model, the 4-layer trace scaling, and the PSA abstraction. No external hardware validation or released artifacts are provided, so these are assumptions the reader must accept from the paper.

free parameters (2)
  • Simulated layers per model = 4 layers, then re-scaled
    Table 2 footnote: only 4 layers per model are simulated and latency/memory are re-scaled to the full model; this modeling choice affects all absolute performance numbers.
  • Memory capacity constraint = 24 GB per NPU
    Section 5.4: configurations with memory footprint exceeding 24 GB per NPU are discarded; chosen as a proxy for device memory and affects which configurations are considered valid.
assumptions (5)
  • domain assumption ASTRA-sim accurately models runtime of distributed training at scale
    Section 4.4 states ASTRA-sim has been cross-validated against real system measurements but gives no citation or experiment in this paper; all conclusions treat simulated latencies as ground truth.
  • domain assumption Roofline compute model with peak-perf, local-mem-bw, and memory-capacity captures device behavior
    Section 2.4 reduces compute devices to three parameters and uses a simple roofline model to estimate operator runtime.
  • ad hoc to paper 4-layer trace templates with symbolic shapes represent full transformer workloads after scaling
    Table 2 footnote simulates only 4 layers per model and re-scales latency and memory; this assumes linear or representative scaling of the full model.
  • domain assumption Network topology as stacking of Ring, Switch, and FC dimensions captures real cluster fabrics
    Section 2.3 abstracts real topologies using the multi-dimensional representation from ASTRA-sim, assuming this representation preserves performance-relevant behavior.
  • ad hoc to paper Reward functions in Section 5.4 encode the intended performance-per-resource objective
    The paper states it minimizes the product of latency and bandwidth, but the written equation maximizes inverse distance of that product to 1, so the objective needs clarification.
invented entities (1)
  • Parameter Set Architecture (PSA)
    purpose: Schema abstraction defining searchable parameters, ranges, and constraints, plus an automated scheduler that configures agents and environment.
    PSA is a software abstraction introduced by the paper; its value is demonstrated only through COSMIC's simulation results, not through independent external validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of COSMIC: Enabling Full-Stack Co-Design and Optimization of Distributed Machine Learning Systems." pith.science (2026). https://pith.science/paper/WFGPR6V7

@misc{pith2026250515020,
  author       = {Pith},
  title        = {Pith review of: COSMIC: Enabling Full-Stack Co-Design and Optimization of Distributed Machine Learning Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WFGPR6V7}},
  note         = {Machine review of arXiv:2505.15020}
}
read the original abstract

Large-scale machine learning models necessitate distributed systems, posing significant design challenges due to the large parameter space across distinct design stacks. Existing studies often focus on optimizing individual system aspects in isolation. This work challenges this limitation and introduces COSMIC, a full-stack distributed machine learning systems environment enabling end-to-end simulation and agent-based design space exploration. To facilitate efficient exploration and optimization across the entire stack, we introduce Parameter Set Architecture-an abstraction concept analogous to the instruction set architecture-abstracting away configuration complexities of agent-based search methods. Case studies demonstrate COSMIC's ability to consolidate parameters across multiple layers of design abstraction, discovering eight non-obvious high-performance system configurations across four transformer-based models with up to 175 billion parameters. By optimizing across the stack, COSMIC full-stack optimization delivers 1.50-48.41x higher performance compared to the isolated single-stack optimization.

Figures

Figures reproduced from arXiv: 2505.15020 by the authors.

Figure 1
Figure 1. Common parallelization strategies common in dis [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Network topology building blocks we considered to [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. (a) shows latency spread for training GPT3-175B [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Summary of (i) Psa to capture the full-stack distributed ML design space and (ii) ML-based Psa optimization frame￾work (Cosmic) to design new distributed ML infrastructures. 4.1 Design Principles Cosmic’s design is guided by three fundamental principles that to￾gether …
Figure 6
Figure 6. Figure 6: ML runtime per BW/NPU for the GPT3-175B work [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Normalized ML runtime per network dollar cost for [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Workload-only vs. full-stack DSE result by [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 10
Figure 10. Figure 10: Reward vs. step convergence curves for each ML [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference

    cs.LG 2025-08 conditional novelty 6.0 of 10

    An RL agent that co-optimizes parallelism degrees and per-operator sharding dimensions finds distributed inference strategies that beat random search and simulated annealing, and slightly outperform Megatron-LM heuris...

Reference graph

Works this paper leans on

63 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    Amazon. 2020. AWS Trainium. https://aws.amazon.com/machine-learning/ trainium

    work page 2020

  2. [2]

    AMD. 2023. AMD Instinct MI300 Series Accelerators. https://www.amd.com/en/ products/accelerators/instinct/mi300.html

    work page 2023

  3. [3]

    Anthropic. 2024. Introducing the next generation of Claude. https://www. anthropic.com/news/claude-3-family

    work page 2024

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    work page 2020

  5. [5]

    Cerebras. 2021. Cerebras Systems: Achieving Industry Best AI Performance Through A Systems Approach. https://cerebras.net/wp-content/uploads/2021/ 04/Cerebras-CS-2-Whitepaper.pdf

    work page 2021

  6. [6]

    Ernie Chan, Robert van de Geijn, William Gropp, and Rajeev Thakur. 2006. Collective Communication on Architectures that Support Simultaneous Com- munication over Multiple Links. In PPoPP

    work page 2006

  7. [7]

    M. Cho, U. Finkler, M. Serrano, D. Kung, and H. Hunter. 2019. BlueConnect: De- composing All-Reduce for Deep Learning on Heterogeneous Network Hierarchy. In MLSys

    work page 2019

  8. [8]

    Sajal Dash, Isaac Lyngaas, Junqi Yin, Xiao Wang, Romain Egele, Guojing Cong, Feiyi Wang, and Prasanna Balaprakash. 2023. Optimizing Distributed Training on Frontier for Large Language Models. In arXiv:2312.12705 [cs.DC]

    arXiv 2023

Show all 63 references
  1. [9]

    Dorigo and G

    M. Dorigo and G. Di Caro. 1999. Ant Colony Optimization: A New Meta-Heuristic. In CEC

    1999

  2. [10]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognit...

    2021 arXiv

  3. [11]

    Hadi Esmaeilzadeh, Soroush Ghodrati, Andrew Kahng, Joon Kyung Kim, Sean Kinzer, Sayak Kundu, Rohan Mahapatra, Susmita Dey Manasi, Sachin Sapatnekar, Zhiang Wang, and Ziqing Zeng. 2024. An Open-Source ML-Based Full-Stack Optimization Framework for Machine Learning Accelerators....

    2024

  4. [12]

    Clement, Neel Sun- daresan, and Chen Wu

    Spandan Garg, Roshanak Zilouchian Moghaddam, Colin B. Clement, Neel Sun- daresan, and Chen Wu. 2022. DeepPERF: A Deep Learning-Based Approach For Improving Software Performance. In arXiv:2206.13619 [cs.SE]

    2022 arXiv

  5. [13]

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. 2023. Photorealistic Video Generation with Diffusion Models. In arXiv:2312.06662 [cs.CV]

    2023 arXiv

  6. [14]

    Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. 2018. PipeDream: Fast and Efficient Pipeline Parallel DNN Training. In arXiv:1806.03377 [cs.DC]

    2018 arXiv

  7. [15]

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In NeurIPS

    2019

  8. [16]

    Sylvain Jeaugey. 2019. Massively Scale Your Deep Learning Training with NCCL 2.4. https://developer.nvidia.com/blog/massively-scale-deep-learning-training- nccl-2-4

    2019

  9. [17]

    Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Clifford Young, Xiang Zhou, Zongwei Zhou, and David A Patterson. 2023. TPU v4: An Optically Reconfigurable Supercomputer for Machine Learn...

    2023

  10. [18]

    Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson

    Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. 2020. A Domain-Specific Supercomputer for Training Deep neural networks. Commun. ACM (2020)

    2020

  11. [19]

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. 2021. Highly Accurate Protein Structure Prediction with Al- phaFold. Nature (2021)

    2021

  12. [20]

    Sheng-Chun Kao and Tushar Krishna. 2020. GAMMA: Automating the HW Mapping of DNN Models on Accelerators via Genetic Algorithm. In ICCAD

    2020

  13. [21]

    Sourabh Katoch, Sumit Singh Chauhan, and Vijay Kumar. 2021. A Review on Genetic Algorithm: Past, Present, and Future. Multimedia Tools Appl. (2021)

    2021

  14. [22]

    Benjamin Klenk, Nan Jiang, Greg Thorson, and Larry Dennison. 2020. An In-Network Architecture for Accelerating Shared-memory Multiprocessor Col- lectives. In ISCA

    2020

  15. [23]

    Srivatsan Krishnan, Amir Yazdanbakhsh, Shvetank Prakash, Jason Jabbour, Ikechukwu Uchendu, Susobhan Ghosh, Behzad Boroujerdian, Daniel Richins, Devashree Tripathy, Aleksandra Faust, and Vijay Janapa Reddi. 2023. Arch- Gym: An Open-Source Gymnasium for Machine Learning Assisted...

    2023

  16. [24]

    H. J. Kushner. 1964. A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise. Journal of Basic Engineering (1964)

    1964

  17. [25]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. In arXiv:2006.16668 [cs.CL]

    2020 arXiv

  18. [26]

    Chuan Li. 2020. OpenAI’s GPT-3 Language Model: A Technical Overview. https: //lambdalabs.com/blog/demystifying-gpt-3

    2020

  19. [27]

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala

  20. [28]

    Hao Lin, Ke Wu, Jie Li, Jun Li, and Wu-Jun Li. 2024. UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming. In arXiv:2307.16375 [cs.LG]

    2024 arXiv

  21. [29]

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, Lifang He, and Lichao Sun. 2024. Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models. In arXiv:2402.17177 [cs.CV]

    2024 arXiv

  22. [30]

    Wenyan Lu, Guihai Yan, Jiajun Li, Shijun Gong, Yinhe Han, and Xiaowei Li

  23. [31]

    Meta. 2023. Introducing LLaMA: A Foundational, 65-billion-Parameter Large Language Model. https://ai.meta.com/blog/large-language-model-llama-meta- ai

    2023

  24. [32]

    J. Močkus. 1975. On Bayesian Methods for Seeking the Extremum. Optimization Techniques IFIP Technical Conference (1975)

    1975

  25. [33]

    NVIDIA. 2020. NVIDIA A100 TENSOR CORE GPU. https://www.nvidia.com/en- us/data-center/a100

    2020

  26. [34]

    NVIDIA. 2023. NVIDIA H100 Tensor Core GPU. https://www.nvidia.com/en- us/data-center/h100

    2023

  27. [35]

    NVIDIA. 2024. NVIDIA Collective Communications Library (NCCL). https: //developer.nvidia.com/nccl

    2024

  28. [36]

    NVIDIA. 2024. NVLink and NVLink Switch. https://www.nvidia.com/en-us/data- center/nvlink/

    2024

  29. [37]

    NVIDIA. 2024. Parallelisms - NVIDIA Docs. https://docs.nvidia.com/nemo- framework/user-guide/latest/nemotoolkit/features/parallelisms.html

    2024

  30. [38]

    OpenAI. 2022. Introducing ChatGPT. https://openai.com/blog/chatgpt

    2022

  31. [39]

    KARL PEARSON. 1905. The Problem of the Random Walk. Nature (1905)

    1905

  32. [40]

    Sundar Pichai and Demis Hassabis. 2023. Introducing Gemini: Our Largest and Most Capable AI Model. https://blog.google/technology/ai/google-gemini-ai

    2023

  33. [41]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. In arXiv:1910.02054 [cs.LG]

    2020 arXiv

  34. [42]

    Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna

  35. [43]

    Saeed Rashidi, William Won, Sudarshan Srinivasan, Srinivas Sridharan, and Tushar Krishna. 2022. Themis: A Network Bandwidth-Aware Collective Sched- uling Policy for Distributed Training of DL Models. In ISCA

    2022

  36. [44]

    Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musu- vathi, Todd Mytkowicz, Jacob Nelson, Olli Saarikivi, and Rachee Singh. 2023. TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches. In NSDI

    2023

  37. [45]

    In ISPASS

    ASTRA-SIM: Enabling SW/HW Co-Design Exploration for Distributed DL Training Platforms. In ISPASS

  38. [46]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. In arXiv:1909.08053 [cs.CL]

    2020 arXiv

  39. [47]

    Gardner, Yiming Yang, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh

    Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob R. Gardner, Yiming Yang, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. 2024. Learning Performance-Improving Code Edits. In ICLR

    2024

  40. [48]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In arXiv:1701.06538 [cs.LG]

    2017 arXiv

  41. [49]

    Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhan- dari, Yuxiong He, and Abhinav Bhatele. 2023. A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training. In ICS

    2023

  42. [50]

    Mellanox Technologies. 2003. Introduction to InfiniBand. https://network.nvidia. com/pdf/whitepapers/IB_Intro_WP_190.pdf

    2003

  43. [51]

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. 2022. Make-A-Video: Text-to-Video Generation without Text-Video Data. In arXiv:2209.14792 [cs.CV]. A. Raju et al

    2022 arXiv

  44. [52]

    Rajeev Thakur, Rolf Rabenseifner, and William Gropp. 2005. Optimization of Collective Communication Operations in MPICH. Int. J. High Perform. Comput. Appl. 19, 1 (2005), 49–66. doi:10.1177/1094342005051521

    2005 doi

  45. [53]

    Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. 2024. Solving Olympiad Geometry without Human Demonstrations. Nature (2024)

    2024

  46. [54]

    Tesla. 2021. Tesla Dojo Technology. https://cdn.motor1.com/pdf-files/535242876- tesla-dojo-technology.pdf

    2021

  47. [55]

    Leslie G. Valiant. 1990. A Bridging Model for Parallel Computation. Commun. ACM 33, 8 (1990), 103–111. doi:10.1145/79173.79181

    1990

  48. [56]

    Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Manya Ghobadi, Zhihao Jia, Dheevatsa Mudigere, Ying Zhang, and Anthony Kewitsch. 2023. TopoOpt: Co- optimizing Network Topology and Parallelization Strategy for Distributed Train- ing Jobs. In NSDI

    2023

  49. [57]

    Amin Vahdat and Mark Lohmeyer. 2023. Enabling next-generation AI workloads: Announcing TPU v5p and AI Hypercomputer. https: //cloud.google.com/blog/products/ai-machine-learning/introducing-cloud- tpu-v5p-and-ai-hypercomputer

    2023

  50. [58]

    William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. 2023. ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale. In Proceedings of the 2023 IEEE International Symposium on Pe...

    2023

  51. [59]

    William Won, Saeed Rashidi, Sudarshan Srinivasan, and Tushar Krishna. 2024. LIBRA: Enabling Workload-aware Multi-dimensional Network Topology Opti- mization for Distributed Training of Large AI Models. In ISPASS

    2024

  52. [60]

    William Won, Midhilesh Elavazhagan, Sudarshan Srinivasan, Ajaya Durg, Samvit Kaul, Swati Gupta, and Tushar Krishna. 2024. TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learning. In arXiv:2304.05301 [cs.DC]

    2024 arXiv

  53. [63]

    Liangyu Zhao, Siddharth Pal, Tapan Chugh, Weiyang Wang, Jason Fantl, Prith- wish Basu, Joud Khoury, and Arvind Krishnamurthy. 2024. Efficient Direct- Connect Topologies for Collective Communications. In arXiv:2202.03356 [cs.NI]

    2024 arXiv

  54. [2017]

    FlexFlow: A Flexible Dataflow Accelerator Architecture for Convolutional Neural Networks. In HPCA

  55. [2020]

    In arXiv:2006.15704 [cs.DC]

    PyTorch Distributed: Experiences on Accelerating Data Parallel Training. In arXiv:2006.15704 [cs.DC]

    2006 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.