Pith. sign in

REVIEW 3 major objections 5 minor 101 references

Opus: Photonic Rail-Optimized Fabric in ML Datacenters

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By time-multiplexing optical circuits across parallelism phases, photonic rails can cut ML network power over 23× and cost 4× while adding under 7% to training time.

desk verdict A genuinely novel phase-multiplexed rail idea, but the headline overhead numbers rest on an unproven non-overlap assumption and a NIC firmware fix that the hardware doesn't deliver yet. read the letter →

arxiv 2602.12521 v3 pith:TE3CXUCK submitted 2026-02-13 cs.NI

Eric Ding , Barry Lyu , Bhaskar Kataria , Rachee Singh This is my paper · ORCID
classification cs.NI
keywords photoniccircuitswitchrail-optimizedfabricparallelism-drivenreconfigurationhybridparallelismopticaldatacenternetworkin-jobMLtrainingpowerefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Rail-optimized fabrics are the standard scale-out network for large-scale ML training, but their high-radix electrical switches consume significant power and cost. This paper argues that the rail abstraction can be preserved with optical circuit switches, which consume a fraction of the power, by reconfiguring the optics within each training step. The key is that the communication phases of different parallelism dimensions—data, pipeline, tensor, context, expert—do not overlap in time, leaving idle windows between phases. Opus, a control plane layered between the training framework and the collective communication library, reprograms the optical switch during these windows so each phase gets full-bandwidth, tailored connectivity without extra NICs. The paper reports that this design achieves over 23× network power reduction and 4× cost savings while adding less than 6.7% to iteration time at production-relevant reconfiguration latencies.

What carries the argument

The central mechanism is parallelism-driven rail reconfiguration: within a single training iteration, the optical circuit switch is reprogrammed at parallelism phase boundaries to present a circuit topology tailored to the upcoming collective, using the same physical ports for every phase. The load-bearing object is the communication window—the idle interval between the end of one phase's collectives and the start of the next phase's—which is formalized as the minimum over next-phase collectives of the slowest-rank start time minus the maximum end time of the previous phase. Opus exploits this window in two ways: on-demand reconfiguration at phase transitions, and speculative provisioning wh

What would settle it

Record collective traces of a hybrid-parallel LLM training step and check whether any idle window between a pipeline-parallel Send/Recv and the following data-parallel ReduceScatter is ever shorter than the OCS reconfiguration time; a single such overlap, or a window below the switching latency, would invalidate the reported overhead.

Watch

Extended reading notes

Core claim

Opus's central claim is that the all-to-all connectivity invariant of rail-optimized fabrics need not be physically provisioned at all times: it can be presented as an illusion by an application-aware control plane that reconfigures a single optical circuit switch between the communication phases of different parallelism dimensions. Because collectives from different parallelisms (e.g., data-parallel ReduceScatter and pipeline-parallel Send/Recv) are separated by data dependencies in the model's compute graph, there are idle windows—often milliseconds long—between phases. If the optical switch can be reprogrammed within such a window, a GPU's few physical NIC ports can be time-multiplexed ac

Load-bearing premise

The load-bearing premise is that the communication phases of different parallelism dimensions never overlap in time, so a single optical switch can be reprogrammed between them without stalling traffic—if DP and PP (or other) collectives overlap, the time-multiplexing mechanism collapses.

Editorial extensions

If this is right

  • The number of parallelisms a job can use is no longer bounded by the number of NIC ports per GPU; Opus's topology encoding supports up to 10 parallelism dimensions with only 2-degree ring connectivity.
  • At production-relevant OCS reconfiguration latencies (up to 100 ms), the training overhead stays below 7%—around 5% on current GPU clusters with provisioning, and lower at 10 ms.
  • Network power and cost scale with cluster size; at 2,048 GPUs the photonic rail shows 15–24× lower power and 3–4× lower cost than electrical rail fabrics, with the absolute savings growing as clusters grow.
  • The datapath becomes GPU→NIC→optical fiber→NIC→GPU, removing OEO conversions and switch ASIC processing, so bandwidth scaling is no longer limited by ASIC speed.
  • Opus works with existing training frameworks through a single backend flag, requiring no changes to model code or parallelism constructs, and its locking protocol ensures circuits are never torn down with traffic in flight.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit: the same phase-window mechanism could serve inference pipelines and RL post-training, whose prefill/decode and rollout/optimize phases also have structured idle windows.
  • The headline 23× power savings is measured against a fully electrical rail baseline; compared to hybrid fabrics that already use co-packaged optics, the relative gain would be smaller, so the figure is best read as the opportunity against today's standard deployment.
  • A testable consequence of the central assumption: the paper's window measurements come from three LLM configurations; workloads that aggressively overlap communication with compute (e.g., zero-bubble pipeline schedules) may shrink inter-phase windows below the OCS reconfiguration time, and measuring window distributions across a broader workload space would bound Opus's applicability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. Opus proposes replacing electrical rail switches in ML datacenter fabrics with optical circuit switches (OCSes), while retaining the rail abstraction through 'parallelism-driven rail reconfiguration.' The key idea is to time-multiplex a single set of physical ports across circuit configurations that are optimized for each parallelism phase (DP, PP, FSDP, etc.), reconfiguring the OCS during the idle windows between communication phases of different parallelism dimensions. The paper describes a control plane (shim, controller, network orchestrator), implements it as a PyTorch backend, and evaluates it on a small physical OCS testbed, on the Perlmutter supercomputer via emulation (up to 64 GPUs), and in simulation at up to 2,048 GPUs. The headline results are >23× network power reduction and >4× cost savings, with less than 6.7% training iteration-time overhead at OCS reconfiguration latencies up to 100 ms.

Significance. If the central claims hold, this would be a significant contribution: it is one of the first systems to make a concrete case for replacing electrical packet switches with OCSes inside a widely deployed rail-optimized topology, without changing the number of NICs per GPU or the job's parallelism strategy. The paper includes a real hardware testbed, a working control-plane implementation, open-source code, and a three-scale evaluation, which are strengths for reproducibility. The power and cost numbers, if substantiated with a transparent methodology, would be important for datacenter designers. However, the headline performance claims rest on two assumptions that are not fully validated: that communication phases of different parallelism dimensions are strictly non-overlapping, and that NIC firmware can be made to support fast link-up after circuit reconfiguration. The physical testbed only demonstrates ~3 s end-to-end reconfiguration, and the simulator enforces the non-overlap assumption rather than testing it.

major comments (3)
  1. [§3.2, Eq. (1)–(2), Fig. 3, and §5.3 (AstraSim backend)] The central time-multiplexing mechanism assumes that communication phases of different parallelism dimensions never overlap. The window definition in Eq. (1) presupposes a gap between the end of all comm_i in P1 and the start of all comm_j in P2. The empirical support is limited to three TorchTitan workloads with PP=2/FSDP=2 and PP=3/FSDP=2; zero-bubble pipeline schedules, MoE AllToAll/AllGather, and aggressive compute-communication overlap are not covered. Critically, the simulation backend in §5.3 'rejects reconfiguration requests while collectives are in flight,' so the simulator enforces the non-overlap assumption rather than testing it. If a collective from the next phase arrives before reconfiguration completes, Opus's lock stalls that collective for the full reconfiguration latency. Please add evidence for schedules with overlap (e.g., zero-bubble, MoE, interleaved microbatches),
  2. [§5.1, Fig. 9(c)–(d)] The hardware testbed does not demonstrate the production-relevant reconfiguration latencies (≤100 ms) used in the paper's headline. The measured end-to-end reconfiguration is dominated by the NIC firmware: the Polatis switch returns optical power within ~200 ms, but the Mellanox firmware takes ~3 s (or ~6 s with auto-negotiation) to report link-up. The paper attributes this to firmware assumptions and says fast link-up is available with firmware support, but no such firmware is demonstrated or simulated at the hardware layer. Consequently, the physical system validates the control plane only at ~3 s reconfiguration time, while the 100 ms results come from emulation/simulation. Please temper the claim that the physical testbed validates production-relevant performance, or add a concrete path (e.g., modified/emulated firmware behavior) to bring the NIC link-up time into the OCS switching r
  3. [§5.3, Figure 14, 'Cost and power'] The cost and power savings—4.27× cost and 23.86× power for H200, 3.17× and 15.44× for GB200—are central to the paper's contribution, but the methodology is not described. The text only cites [16–18,44,52,63] and states that fiber cables are excluded. There is no bill of materials, no unit power/cost table, no switch/transceiver counts, and no sensitivity analysis. As written, the savings factors are not reproducible. Please provide a component-level cost/power model, including OCS, transceivers, NICs, and switch ASICs, and show how the savings vary with the assumed OCS port count, link rate, and pricing source.
minor comments (5)
  1. [Abstract and §1] The abstract states 'less than 6% training overhead,' while §1 and §5.3 report 'less than 6.7%' and specific values such as 5.31% and 11.22%. Please reconcile the abstract with the empirical numbers.
  2. [Eq. (5) / Fig. 5] The formula for the number of windows is presented as an equation with symbols (n_layer, n_microbatch, PP) but the terms in the right-hand side are not individually derived or defined in the text. Please define all terms and give a brief derivation of each additive component.
  3. [§5.1 / Fig. 9] The RDMA RETRY_CNT=7 setting is mentioned in the text but not discussed as a potential limitation. Since the paper claims no transport modifications, please clarify whether this setting affects fault tolerance or timeout behavior during reconfiguration.
  4. [Figure 3 and 4] The axis labels and legends in Figures 3 and 4 are hard to read in the provided text; e.g., repeated '0481204812' tick labels and 'Rail 0 window break-down' should be 'breakdown.' Please improve figure clarity.
  5. [§5.3 / Table 2] The simulation baseline 'EPS' is described as having all links active that Opus could form, but the paper does not specify the exact EPS topology (e.g., rail-optimized vs. fat-tree). Since the power/cost comparison uses 'EPS Rail-Opt' in Figure 14, please clarify whether the performance baseline is the same rail-optimized EPS or a generic electrical fabric.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed power/cost and overhead results are derived from component models, measurements, and simulations whose inputs are not fitted to the claimed outputs; the self-citations are not load-bearing.

full rationale

I walked the paper's derivation chain and found no step in which a claimed prediction reduces by construction to its inputs. The headline quantitative results—over 23× power reduction, ~4× cost savings, and <6.7% iteration-time overhead—are produced by two independent mechanisms. The cost/power comparison is based on component counts and unit costs (e.g., "Cost and power exclude fiber cables. [16–18, 44, 52, 63]"), not on fitted values derived from the claimed savings. The iteration-time overhead is measured on a physical testbed, emulated on Perlmutter with injected reconfiguration delays, and simulated in AstraSim with Chakra traces; in all cases the reconfiguration latency is swept as an independent input (0–1000 ms) rather than tuned to reproduce the <6.7% number. The window measurements in §3.2 motivate the choice of reconfiguration latency target, but the simulation derives communication and compute times from model/workload configurations, so the overhead result is not an artifact of the window definition. The paper's central mechanism does rely on the empirical assumption that parallelism phases are non-overlapping, and the evidence for this is limited to three measured workloads; however, that is a generalizability/correctness concern, not circularity, because the assumption is stated as an observation and is not used to define the measured overhead. The citations to the authors' own prior work ([32] and [33]) appear in related-work and survey contexts (e.g., "A wide range of general datacenter fabric designs... [29, 30, 32, 33, 83, 85, 91]"), and are not invoked as proof of Opus's mechanism, uniqueness, or optimality. Accordingly, no load-bearing self-citation chain exists. The paper is a systems design with components, measurements, and simulations whose inputs are independent of the final claims, so the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. Its central claim rests on empirical assumptions about ML communication phase non-overlap and on hardware assumptions about OCS reconfiguration latency and NIC link-up speed. The free parameters are mostly swept or hand-chosen configuration values, not fitted to data. The most load-bearing assumption is the non-overlap of parallelism phases, which is measured for specific workloads but not guaranteed in general.

free parameters (3)
  • profiling_steps = 5
    The phase table is populated during the first 5 training iterations. This is a hand-chosen constant; different values could affect the accuracy of phase-boundary detection and the resulting reconfiguration overhead.
  • OCS reconfiguration latency in simulations = swept: 0–1000 ms
    The central overhead claims are produced by sweeping this parameter. It is not fitted, but the claim that 'production-relevant' latencies are ≤100 ms is a chosen target, not a measured hardware capability.
  • RDMA RETRY_CNT = 7
    Set to avoid QP error state during reconfiguration. A design choice; the paper does not study sensitivity to this value.
assumptions (4)
  • domain assumption Communication phases of different parallelisms do not overlap in time (there is a non-empty window between the end of one parallelism's last collective and the start of the next's first collective).
    This is the basis for time-multiplexing. It is empirically supported by Figure 3 for three workloads on Perlmutter, but is not guaranteed for all pipeline schedules, model sizes, or overlap strategies. The window-size equation in Section 3.2 assumes this ordering.
  • domain assumption OCS reconfiguration latency can be hidden within phase-transition windows (i.e., the windows are larger than the reconfiguration delay).
    The paper measures window sizes of at least ~1ms for 75% of transitions, but production-relevant latencies are taken as 100ms in the headline results. The hardware testbed shows 3s NIC firmware delay, so the hiding assumption depends on future firmware improvements.
  • domain assumption The parallelism phase structure is stable across training iterations (the phase table learned in the first 5 steps remains valid).
    Profiling assumes a deterministic communication schedule. Dynamic graphs, varying batch sizes, or failover reconfigurations could invalidate this. The paper acknowledges fault handling but not schedule variability.
  • domain assumption OCS radix is sufficient to connect all GPUs of a rail (e.g., 384-512 ports for large scale-up domains).
    Section 7 states up to 18K GPUs per rail with 512-port liquid-crystal OCSes, but this requires one port per GPU NIC and assumes the NIC has sufficient ports. The paper's own C1–C3 challenges are resolved by reconfiguration, but port count remains a hard limit that the paper does not deeply analyze.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Opus: Photonic Rail-Optimized Fabric in ML Datacenters." pith.science (2026). https://pith.science/paper/TE3CXUCK

@misc{pith2026260212521,
  author       = {Pith},
  title        = {Pith review of: Opus: Photonic Rail-Optimized Fabric in ML Datacenters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TE3CXUCK}},
  note         = {Machine review of arXiv:2602.12521}
}
abstract

Rail-optimized network fabrics have become the de facto datacenter scale-out fabric for large-scale ML training. However, the use of high-radix electrical switches to provide all-to-all connectivity in rails imposes substantial power and cost. We propose a rethinking of the rail abstraction by retaining its communication semantics, but realizing it using optical circuit switches. The key challenge is that optical switches support one-to-one connectivity at a time, limiting the fan-out of traffic in ML workloads using hybrid parallelisms. We overcome this through \emph{parallelism-driven rail reconfiguration}, which exploits the non-overlapping communication phases of different parallelism dimensions. This time-multiplexes a single set of physical ports across circuit configurations tailored to each phase within a training iteration. We design and implement Opus, a control plane that orchestrates this in-job reconfiguration of photonic rails at parallelism phase boundaries, and evaluate it on a physical OCS testbed, the Perlmutter supercomputer, and in simulation at up to 2,048 GPUs. Our results show that photonic rails can achieve over $23\times$ network power reduction and $4\times$ cost savings while incurring only modest training overhead at production-relevant OCS reconfiguration latencies.

Figures

Figures reproduced from arXiv: 2602.12521 by the authors.

Figure 1
Figure 1. Rail-optimized fabrics.We propose to replace packet [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Traffic in a training iteration with 3D parallelism. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Communication pattern for PP and FSDP in one [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: (a) CDF of window size from 10 iters in Exp 1. (b) Rail 0 window break-down based on traffic volume after the window [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Number of windows in one training iteration with different parallelisms. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Reconfiguration during the warm-up stage of rank [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Opus system architecture. topology identifier, topo_id𝑖 , which encodes the current con￾nectivity requirement of the job on rail 𝑖. Topology ID encoding. The topo_id is a decimal integer whose digit positions correspond to the ways (stages) of the job’s asymmetrical pa…
Figure 8
Figure 8. Figure 8: Translating parallelism shifts to topology recon [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Hardware testbed evaluation. (a) Physical testbed: 4 L40 GPU servers connected via a Polatis Series 6000 OCS forming [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Step latency vs. emulated OCS reconfiguration [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Control-plane overhead. (left) Config. 2 (64 GPUs): [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: LLaMA-80B on 128 H200 GPUs (DP=4, PP=4, TP=8). [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: GPT-80B on 512 GB200 GPUs (DP=4, PP=4, TP=32). [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 14
Figure 14. Figure 14: Performance, cost, and power scaling. DGX H200 [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

101 extracted references · 1 canonical work pages

  1. [86]

    Zhenguo Wu, Benjamin Klenk, Larry Dennison, and Keren Bergman

  2. [1]

    NVIDIA GB200 NVL72

    2025. NVIDIA GB200 NVL72. https://www.nvidia.com/en-us/ data-center/gb200-nvl72/. (2025). https://www.nvidia.com/en-us/ data-center/gb200-nvl72/ Accessed: 2026-02-07

    2025

  3. [2]

    Saksham Agarwal, Qizhe Cai, Rachit Agarwal, David Shmoys, and Amin Vahdat. 2024. Harmony: A congestion-free datacenter archi- tecture. In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). 329–343

    2024

  4. [3]

    Daniel Amir, Nitika Saran, Tegan Wilson, Robert Kleinberg, Vishal Shrivastav, and Hakim Weatherspoon. 2024. Shale: A practical, scalable oblivious reconfigurable network. InProceedings of the ACM SIGCOMM 2024 Conference. 449–464

    2024

  5. [4]

    Hitesh Ballani, Paolo Costa, Raphael Behrendt, Daniel Cletheroe, Istvan Haller, Krzysztof Jozwik, Fotini Karinou, Sophie Lange, Kai Shi, Benn Thomsen, et al. 2020. Sirius: A flat datacenter network with nanosecond optical switching. InProceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technolo...

    2020

  6. [5]

    Kaoutar Benyahya, Ariel Gomez Diaz, Junyi Liu, Vassily Lyutsarev, Marianna Pantouvaki, Kai Shi, Shawn Yohanes Siew, Hitesh Ballani, Thomas Burridge, Daniel Cletheroe, et al. 2025. Mosaic: Breaking the Optics versus Copper Trade-off with a Wide-and-Slow Architecture and MicroLEDs. InProceedings of the ACM SIGCOMM 2025 Conference. 234–247

    2025

  7. [6]

    Pankaj Berde, Matteo Gerola, Jonathan Hart, Yuta Higuchi, Masayoshi Kobayashi, Toshio Koide, Bob Lantz, Brian O’Connor, Pavlin Ra- doslavov, William Snow, et al . 2014. ONOS: towards an open, dis- tributed SDN OS. InProceedings of the third workshop on Hot topics in software defined networking. 1–6

    2014

  8. [7]

    Maciej Besta and Torsten Hoefler. 2014. Slim Fly: A Cost Effective Low- Diameter Network Topology. InSC ’14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 348–359. https://doi.org/10.1109/SC.2014.34

    doi:10.1109/sc.2014.34 2014

Show all 101 references
  1. [8]

    Broadcom Inc. 2025. BCM78909 51.2 -Tb/s Multilayer Co-Packaged Optics Switch. Online; accessed July 5, 2025. (2025). https: //www.broadcom.com/products/fiber-optic-modules-components/ co-packaged-optics/switches/bcm78909 A high-radix, high-bandwidth CPO switch supporting up to...

    2025

  2. [9]

    Broadcom Inc. 2025. Co -Packaged Optics (CPO). https://www. broadcom.com/info/optics/cpo. (2025). Accessed: 2025-07-03

    2025

  3. [10]

    Optical Systems Division

    Broadcom Inc. Optical Systems Division. 2021.SiPh Chiplets In Package (SCIP). Technical Report. Broadcom Inc., Irvine, CA, USA. https: //docs.broadcom.com/doc/siph-chiplets-in-package-scip OSD CPO SCIP_20211106 V5

    2021

  4. [11]

    Li Chen, Kai Chen, Zhonghua Zhu, Minlan Yu, George Porter, Chun- ming Qiao, and Shan Zhong. 2017. Enabling {Wide-Spread} Com- munications on Optical Fabric with{MegaSwitch}. In14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). 577–593

    2017

  5. [12]

    Weiwei Chu, Xinfeng Xie, Jiecao Yu, Jie Wang, Amar Phanishayee, Chunqiang Tang, Yuchen Hao, Jianyu Huang, Mustafa Ozdal, Jun Wang, et al. 2025. Scaling Llama 3 Training with Efficient Parallelism Strategies. InProceedings of the 52nd Annual International Symposium on Computer ...

    2025

  6. [13]

    Coherent Corp. 2025. Optical Circuit Switch (OCS). https://www. coherent.com/networking/optical-circuit-switch. (2025). Accessed: 2025-07-10; Based on press release published March 25,2024; Coher- ent’s liquid-crystal-based OCS architecture supports up to 300×300 ports and is ...

    2025

  7. [14]

    Nathan Farrington, George Porter, Sivasankar Radhakrishnan, Hamid Hajabdolali Bazzaz, Vikram Subramanya, Yeshaiahu Fainman, George Papen, and Amin Vahdat. 2010. Helios: a hybrid electri- cal/optical switch architecture for modular data centers. InProceedings of the ACM SIGCOMM...

    2010

  8. [15]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch trans- formers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research23, 120 (2022), 1–39

    2022

  9. [16]

    FS.COM. n.d.. Cisco Compatible 400GBASE -XDR4 QSFP-DD PAM4 1310nm 2km Module. https://www.fs.com/products/110530.html? attribute=94270&id=4477813. (n.d.). Accessed: 2025-07-02

    2025

  10. [17]

    FS.COM. n.d.. N9510 -64D 64-Port Ethernet L3 Data Center Switch (Broadcom Tomahawk-4, 64×400GbE). https://www.fs.com/products/ 149853.html. (n.d.). Accessed: 2025-07-02

    2025

  11. [18]

    FS.com Inc. 2025. NVIDIA/Mellanox MMA4Z00-NS Optical Transceiver Module. https://www.fs.com/products/229253.html. (2025). Product page, Accessed: 2026-02-06

    2025

  12. [19]

    Swapnil Gandhi, Mark Zhao, Athinagoras Skiadopoulos, and Christos Kozyrakis. 2024. Recycle: Resilient training of large dnns using pipeline adaptation. InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. 211–228

    2024

  13. [20]

    Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, et al . 2024. Rdma over eth- ernet for distributed training at meta scale. InProceedings of the ACM SIGCOMM 2024 Confer...

    2024

  14. [21]

    Alexandru M Gherghescu, Vlad-Andrei Bădoiu, Alexandru Agache, Mihai-Valentin Dumitru, Iuliu Vasilescu, Radu Mantu, and Costin Raiciu. 2024. I’ve Got 99 Problems But FLOPS Ain’t One. InProceedings of the 23rd ACM Workshop on Hot Topics in Networks. 195–204

    2024

  15. [22]

    Monia Ghobadi, Ratul Mahajan, Amar Phanishayee, Nikhil Deva- nur, Janardhan Kulkarni, Gireeja Ranade, Pierre-Alexandre Blanche, Houman Rastegarfar, Madeleine Glick, and Daniel Kilper. 2016. Pro- jecToR: Agile Reconfigurable Data Center Interconnect. InProceed- ings of the 2016...

    2016

  16. [23]

    Hamilton, Navendu Jain, Srikanth Kandula, Changhoon Kim, Parantap Lahiri, David A

    Albert Greenberg, James R. Hamilton, Navendu Jain, Srikanth Kandula, Changhoon Kim, Parantap Lahiri, David A. Maltz, Parveen Patel, and Sudipta Sengupta. 2009. VL2: A Scalable and Flexible Data Center Network. InProceedings of the ACM SIGCOMM 2009 Conference on 13 Data Communi...

    2009 doi

  17. [24]

    Das, Jon P

    Navid Hamedazimi, Zafar Qazi, Himanshu Gupta, Vyas Sekar, Samir R. Das, Jon P. Longtin, Himanshu Shah, and Ashish Tanwer. 2014. Fire- Fly: A Reconfigurable Wireless Data Center Fabric Using Free-Space Optics. InProceedings of the 2014 ACM Conference on SIGCOMM (SIG- COMM ’14)....

    2014

  18. [25]

    Vipul Harsh, Sangeetha Abdu Jyothi, and P Brighten Godfrey. 2020. Spineless data centers. InProceedings of the 19th ACM Workshop on Hot Topics in Networks. 67–73

    2020

  19. [26]

    Hewlett Packard Enterprise. 2021. HPE Cray EX Supercomputer Overview. https://www.hpe.com/psnow/doc/a50002546enw. (2021). Accessed: 2025-07-09

    2021

  20. [27]

    2010.{ZooKeeper}: Wait-free coordination for internet-scale systems

    Patrick Hunt, Mahadev Konar, Flavio P Junqueira, and Benjamin Reed. 2010.{ZooKeeper}: Wait-free coordination for internet-scale systems. In2010 USENIX Annual Technical Conference (USENIX ATC 10)

    2010

  21. [28]

    Insu Jang, Zhenning Yang, Zhen Zhang, Xin Jin, and Mosharaf Chowd- hury. 2023. Oobleck: Resilient distributed training of large models using pipeline templates. InProceedings of the 29th Symposium on Operating Systems Principles. 382–395

    2023

  22. [29]

    Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, et al. 2023. Tpu v4: An optically reconfigurable supercom- puter for machine learning with hardware support for embeddings. In Proceedings...

    2023

  23. [30]

    Mehrdad Khani, Manya Ghobadi, Mohammad Alizadeh, Ziyi Zhu, Madeleine Glick, Keren Bergman, Amin Vahdat, Benjamin Klenk, and Eiman Ebrahimi. [n. d.]. SiP-ML: High-Bandwidth Optical Network Interconnects for Machine Learning Training. InProceedings of the 2021 ACM SIGCOMM 2021 C...

    2021

  24. [31]

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro

  25. [32]

    Abhishek Vijaya Kumar, Arjun Devraj, Darius Bunandar, and Rachee Singh. 2024. A case for server-scale photonic connectivity. InProceed- ings of the 23rd ACM Workshop on Hot Topics in Networks (HotNets ’24). Association for Computing Machinery, New York, NY, USA, 290–299. https...

    2024

  26. [33]

    Abhishek Vijaya Kumar, Eric Ding, Arjun Devraj, Darius Bunandar, and Rachee Singh. 2025. Morphlux: Transforming Torus Fabrics for Efficient Multi-tenant ML. (2025). arXiv:cs.NI/2508.03674 https://arxiv. org/abs/2508.03674

    2025

  27. [34]

    ChonLam Lao, Minlan Yu, Aditya Akella, Jiamin Cao, Yu Guan, Pengcheng Zhang, Zhilong Zheng, Yichi Xu, Ennan Zhai, Dennis Cai, et al. 2024. TrainMover: Efficient ML Training Live Migration with No Memory Overhead.arXiv e-prints(2024), arXiv–2412

    2024

  28. [35]

    Cong Liang, Xiangli Song, Jing Cheng, Mowei Wang, Yashe Liu, Zhen- hua Liu, Shizhen Zhao, and Yong Cui. 2024. NegotiaToR: Towards A Simple Yet Effective On-demand Reconfigurable Datacenter Network. InProceedings of the ACM SIGCOMM 2024 Conference. 415–432

    2024

  29. [36]

    Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, et al. 2024. TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training.arXiv preprint arXiv:2410.06511 (2024)

    2024 arXiv

  30. [38]

    Xudong Liao, Yijun Sun, Han Tian, Xinchen Wan, Yilun Jin, Zilong Wang, Zhenghang Ren, Xinyang Huang, Wenxue Li, Kin Fai Tse, et al

  31. [39]

    Linux. [n. d.].ethtool(8) - Linux man page. https://linux.die.net/man/8/ ethtool

  32. [40]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al

  33. [41]

    InProceedings of the ACM SIGCOMM 2025 Conference

    Mixnet: A runtime reconfigurable optical-electrical fabric for distributed mixture-of-experts training. InProceedings of the ACM SIGCOMM 2025 Conference. 554–574

    2025

  34. [42]

    Lumentum Holdings Inc. 2025. Lumentum Optical Circuit Switch to Improve Next -Generation AI Data Center Scalabil- ity. https://www.lumentum.com/en/media-room/news-releases/ lumentum-optical-circuit-switch-improve-next-generation-ai-data-center. (26 March 2025). Accessed June 20, 2025

    2025

  35. [43]

    William M Mellette, Rob McGuinness, Arjun Roy, Alex Forencich, George Papen, Alex C Snoeren, and George Porter. 2017. Rotornet: A scalable, low-complexity, optical datacenter network. InProceedings of the Conference of the ACM Special Interest Group on Data Communica- tion. 267–280

    2017

  36. [44]

    NADDOD. 2025. NVIDIA Quantum-X800 XDR InfiniBand Switch, Q3400-RA. https://www.naddod.com/products/nvidia-networking/ 102612. (2025). Reseller price listing, Accessed: 2026-02-06

    2025

  37. [45]

    Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring attention with blockwise transformers for near-infinite context.arXiv preprint arXiv:2310.01889(2023)

    2023 arXiv

  38. [46]

    nEye Systems. 2025. nEye: Dismantling Network Walls to Build a Sustainable AI Future. https://www.neye.ai/. (2025). Optical circuit switch platform for AI datacenter networking. Accessed: 2025-02-02

    2025

  39. [47]

    2024.NVIDIA Firmware Tools (MFT) Docu- mentation

    NVIDIA. 2024.NVIDIA Firmware Tools (MFT) Docu- mentation. https://docs.nvidia.com/networking/display/ nvidia-firmware-tools-mft-documentation-v4-32-0.0.pdf

    2024

  40. [48]

    NVIDIA. 2025. Llama-3.1-405B DGXC Benchmarking Recipe. https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ dgxc-benchmarking/resources/llama31-405b-dgxc-benchmarking-a. (2025). Version 24.11.1, modified January 29, 2025

    2025

  41. [49]

    National Energy Research Scientific Computing Center (NERSC). 2025. Perlmutter Architecture — NERSC Documentation. https://docs.nersc. gov/systems/perlmutter/architecture/. (2025). Accessed: 2025-07-04

    2025

  42. [50]

    NVIDIA Corporation. 2022. Doubling all -to-all Performance with NCCL 2.12: Introducing PXN (PCI X NVLink). NVIDIA Developer Blog. (Feb. 2022). https://developer.nvidia.com/blog/ doubling-all2all-performance-with-nvidia-collective-communication-library-2-12/ Describes PXN, whic...

    2022

  43. [51]

    NVIDIA Corporation. 2024. ConnectX -7 400G Adapters Datasheet. https://resources.nvidia. com/en-us-accelerated-networking-resource-library/ connectx-7-datasheet. (2024). Accessed: 2025-07-02

    2024

  44. [52]

    NVIDIA Corporation. 2024. NVIDIA Q32xx and Q34xx XDR 800Gb/s InfiniBand Switch Systems User Manual. https://docs.nvidia.com/ networking/display/xdrswitcheshwum/specifications. (2024). Ac- cessed: 2026-02-06

    2024

  45. [53]

    2020.NVIDIA Collective Communication Library (NCCL): Creating a Communicator

    NVIDIA Corporation. 2020.NVIDIA Collective Communication Library (NCCL): Creating a Communicator. NVIDIA. https://docs.nvidia.com/ deeplearning/nccl/user-guide/docs/usage/communicators.html Ac- cessed July 6, 2025

    2020

  46. [54]

    NVIDIA Corporation. 2025. Co -Packaged Silicon Photonics Network- ing Switches. Online; accessed July 5,2025. (2025). https://www. nvidia.com/en-us/networking/products/silicon-photonics/ Describes NVIDIA’s co-packaged optics (CPO) switches with integrated silicon photonics

    2025

  47. [55]

    2025.NVIDIA Announces Spectrum -X Photonics, Co -Packaged Optics Networking Switches to Scale AI Factories to Millions of GPUs

    NVIDIA Corporation. 2025.NVIDIA Announces Spectrum -X Photonics, Co -Packaged Optics Networking Switches to Scale AI Factories to Millions of GPUs. Press Release. NVIDIA Corpora- tion, Santa Clara, CA, USA. https://nvidianews.nvidia.com/news/ nvidia-spectrum-x-co-packaged-opti...

    2025

  48. [56]

    2025.NVIDIA Collective Communications Library (NCCL)

    NVIDIA Corporation. 2025.NVIDIA Collective Communications Library (NCCL). NVIDIA Developer. https://developer.nvidia.com/nccl Version 2.x; MPI-compatible multi-GPU / multi-node collective communication library

    2025

  49. [57]

    NVIDIA Corporation. 2025. ConnectX-6 Dx Firmware Download. https://network.nvidia.com/support/firmware/connectx6dx/. (2025). Accessed: 2026-02-06. 14

    2025

  50. [58]

    2025.NVIDIA DGX SuperPOD

    NVIDIA Corporation. 2025.NVIDIA DGX SuperPOD. NVIDIA. https: //www.nvidia.com/en-us/data-center/dgx-superpod/ Full-stack data center platform scaling to tens of thousands of GPUs; includes compute, networking, storage, and software

    2025

  51. [59]

    NVIDIA Corporation. 2025. NVIDIA Ethernet Driver for Linux (mlnx_en). https://network.nvidia.com/products/ethernet-drivers/ linux/mlnx_en/. (2025). Accessed: 2026-02-06

    2025

  52. [60]

    2025.NVIDIA HGX Platform

    NVIDIA Corporation. 2025.NVIDIA HGX Platform. NVIDIA. https: //www.nvidia.com/en-us/data-center/hgx/ Reference architecture combining GPUs, NVLink/NVSwitch, networking, and AI/HPC soft- ware stack

    2025

  53. [61]

    2025.NVIDIA DGX H200 Datasheet

    NVIDIA Corporation. 2025.NVIDIA DGX H200 Datasheet. Datasheet. NVIDIA Corporation, Santa Clara, CA. https://resources.nvidia.com/ en-us-dgx-systems/dgx-h200-datasheet Includes specifications of the DGX H200 system, featuring 8×H200 GPUs, dual Xeon Platinum 8480C CPUs, 2 TB sys...

    2025

  54. [62]

    Jeremie Eliahou Ontiveros, Dylan Patel, and Wei Zhou

  55. [63]

    Polatis. 2023. Polatis Series 6000n Optical Switch Datasheet. https://www.redhelix.com/wp-content/uploads/2023/11/Polatis_ 6000n_Data_Sheet-rhl.pdf. (2023). Datasheet, Accessed: 2026-02-06

    2023

  56. [64]

    Polatis (a HUBER+SUHNER company). n.d.. Series 7000 - 384x384-port Software-Defined Optical Circuit Switch. https://www.polatis.com/ series-7000-384x384-port-software-controlled-optical-circuit-switch-sdn-enabled. asp. (n.d.). Accessed: 2025-07-01

    2025

  57. [65]

    2025.Rail Optimized Topology Val- idation

    NVIDIA Corporation. 2025.Rail Optimized Topology Val- idation. NVIDIA Networking, Santa Clara, CA. https: //docs.nvidia.com/networking/display/ibdiagnetusermanualv221/ Rail+Optimized+Topology+Validation Part of the ibdiagnet InfiniBand Fabric Diagnostic Tool User Manual; descr...

    2025

  58. [66]

    Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. 2023. Zero bubble pipeline parallelism.arXiv preprint arXiv:2401.10241(2023)

    2023 arXiv

  59. [67]

    SemiAnal- ysis

    xAI’s Colossus 2 - First Gigawatt Datacenter In The World, Unique RL Methodology, Capital Raise. SemiAnal- ysis. (Sept. 2025). https://newsletter.semianalysis.com/p/ xais-colossus-2-first-gigawatt-datacenter Accessed: 2026-01-23

    2025

  60. [68]

    Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. 2020. ASTRA-SIM: Enabling SW/HW Co-Design Exploration for Distributed DL Training Platforms. InIEEE International Sympo- sium on Performance Analysis of Systems and Software, ISPASS 2020, Boston, MA, ...

    2020

  61. [69]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He

  62. [70]

    Leon Poutievski, Omid Mashayekhi, Joon Ong, Arjun Singh, Mukar- ram Tariq, Rui Wang, Jianan Zhang, Virginia Beauregard, Patrick Con- ner, Steve Gribble, Rishi Kapoor, Stephen Kratzer, Nanfang Li, Hong Liu, Karthik Nagaraj, Jason Ornstein, Samir Sawhney, Ryohei Urata, Lorenzo V...

    2022

  63. [71]

    Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Jacob Nelson, Olli Saarikivi, and Rachee Singh. 2023. TACCL: Guiding Collective Algorithm Synthe- sis using Communication Sketches. In20th USENIX Symposium on Networked Systems Design...

    2023

  64. [72]

    Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan, Binzhang Fu, Xuemei Shi, Fangbo Zhu, Rui Miao, et al. 2024. Alibaba hpn: A data center network for large language model training. In Proceedings of the ACM SIGCOMM 2024 Conference. 691–706

    2024

  65. [73]

    Vishal Shrivastav, Asaf Valadarsky, Hitesh Ballani, Paolo Costa, Ki Suh Lee, Han Wang, Rachit Agarwal, and Hakim Weatherspoon. 2019. Shoal: A Network Architecture for Disaggregated Racks. In16th USENIX Symposium on Networked Systems Design and Implementa- tion (NSDI 19). USENI...

    2019

  66. [74]

    Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armis- tead, Roy Bannon, Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost, Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat. 2015. Jupiter Risin...

    2015

  67. [75]

    Ankit Singla, P Brighten Godfrey, and Alexandra Kolla. 2014. High throughput data center topology design. In11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14). 29–41

    2014

  68. [76]

    Peter Sanders, Jochen Speck, and Jesper Larsson Träff. 2009. Two-tree algorithms for full bandwidth broadcast, reduction and scan.Parallel Comput.35, 12 (2009), 581–594

    2009

  69. [77]

    Srinivas Sridharan, Taekyung Heo, Louis Feng, Zhaodong Wang, Matt Bergeron, Wenyin Fu, Shengbao Zheng, Brian Coutinho, Saeed Rashidi, Changhai Man, and Tushar Krishna. 2023. Chakra: Advancing Perfor- mance Benchmarking and Co-design using Standardized Execution Traces.arXiv pr...

    2023 arXiv

  70. [78]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi- billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053(2019)

    2019 arXiv

  71. [79]

    Rajeev Thakur and William D Gropp. 2003. Improving the perfor- mance of collective operations in MPICH. InEuropean Parallel Virtual Machine/Message Passing Interface Users’ Group Meeting. Springer, 257– 267. 15

    2003

  72. [80]

    Andersen, Michael Kaminsky, Konstantina Papagiannaki, T.S

    Guohui Wang, David G. Andersen, Michael Kaminsky, Konstantina Papagiannaki, T.S. Eugene Ng, Michael Kozuch, and Michael Ryan

  73. [81]

    Guohui Wang, David G Andersen, Michael Kaminsky, Konstantina Papagiannaki, TS Eugene Ng, Michael Kozuch, and Michael Ryan. 2010. c-Through: Part-time optics in data centers. InProceedings of the ACM SIGCOMM 2010 Conference. 327–338

    2010

  74. [82]

    Brighten Godfrey

    Ankit Singla, Chi-Yao Hong, Lucian Popa, and P. Brighten Godfrey

  75. [83]

    2023.{TopoOpt}: Co-optimizing network topology and parallelization strategy for distributed training jobs

    Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Manya Ghobadi, Zhihao Jia, Dheevatsa Mudigere, Ying Zhang, and Anthony Kewitsch. 2023.{TopoOpt}: Co-optimizing network topology and parallelization strategy for distributed training jobs. In20th USENIX Symposium on Networked Systems...

    2023

  76. [84]

    Chuan Wu. 2025. HybridFlow: A Flexible and Efficient RLHF Frame- work.EuroSys 2025 (30/03/2025-03/04/2025, Rotterdam)(2025)

    2025

  77. [85]

    Nouamane Tazi, Ferdinand Mom, Haojun Zhao, Phuc Nguyen, Mo- hamed Mekkouri, Leandro Werra, and Thomas Wolf. 2025. The Ultra- Scale Playbook: Training LLMs on GPU Clusters. https://huggingface. co/spaces/nanotron/ultrascale-playbook. (2025). Accessed: 2025-05-16

    2025

  78. [87]

    Zhenguo Wu, Liang Yuan Dai, Yuyang Wang, Songli Wang, and Keren Bergman. 2024. Flexible silicon photonic architecture for accelerating distributed deep learning.Journal of Optical Communications and Networking16, 2 (2024), A157–A168

    2024

  79. [88]

    Eric P Xing, Qirong Ho, Wei Dai, Jin-Kyu Kim, Jinliang Wei, Seunghak Lee, Xun Zheng, Pengtao Xie, Abhimanu Kumar, and Yaoliang Yu

  80. [89]

    Sharada Yeluri. 2023. Optimizing Power Consumption in High- End Routers. https://www.linkedin.com/posts/sharada-yeluri_ sharadayeluriblogs-junipernetworks-routers-activity. (22 June 2023)

    2023

  81. [90]

    Weiyang Wang, Manya Ghobadi, Kayvon Shakeri, Ying Zhang, and Naader Hasani. 2024. Rail-only: A low-cost high-performance network for training LLMs with trillion parameters. In2024 IEEE Symposium on High-Performance Interconnects (HOTI). IEEE, 1–10

    2024

  82. [91]

    Yazhou Zu, Alireza Ghaffarkhah, Hoang-Vu Dang, Brian Towles, Steven Hand, Safeen Huda, Adekunle Bello, Alexander Kolbasov, Arash Rezaei, Dayou Du, Steve Lacy, Hang Wang, Aaron Wisner, Chris Lewis, and Henri Bahini. 2024. Resiliency at Scale: Managing Google’s TPUv4 Machine Lea...

    2024

  83. [93]

    Zhenguo Wu, Liang Yuan Dai, Ziyi Zhu, Asher Novick, Madeleine Glick, and Keren Bergman. 2023. SiP Architecture For Accelerat- ing Collective Communication in Distributed Deep Learning. In2023 Optical Fiber Communications Conference and Exhibition (OFC). 1–3. https://doi.org/10...

    2023 doi

  84. [95]

    InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

    ACTINA: Adapting Circuit-Switching Techniques for AI Net- working Architectures. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1211–1222

  85. [100]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al

  86. [101]

    arXiv preprint arXiv:2304.11277(2023)

    Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277(2023)

    2023 arXiv

  87. [2010]

    InProceedings of the ACM SIGCOMM 2010 Conference (SIGCOMM ’10)

    C-Through: Part-Time Optics in Data Centers. InProceedings of the ACM SIGCOMM 2010 Conference (SIGCOMM ’10). Association for Computing Machinery, New York, NY, USA, 327–338. https://doi.org/ 10.1145/1851182.1851222

    2010

  88. [2012]

    In9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12)

    Jellyfish: Networking Data Centers Randomly. In9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12). USENIX Association, San Jose, CA, 225–238. https://www.usenix. org/conference/nsdi12/technical-sessions/presentation/singla

  89. [2015]

    InProceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

    Petuum: A new platform for distributed machine learning on big data. InProceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1335–1344

  90. [2020]

    InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 3505–3506

  91. [2023]

    Proceedings of Machine Learning and Systems5 (2023), 341–353

    Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems5 (2023), 341–353

    2023

  92. [2024]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  93. [2025]

    mFabric: An Efficient and Scalable Fabric for Mixture-of-Experts Training.arXiv preprint arXiv:2501.03905(2025)

    2025 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.