SAKURAONE: Empowering Transparent and Open AI Platforms through Private-Sector HPC Investment in Japan

Fumikazu Konishi

REVIEW 2 major objections 5 minor 9 references

SAKURAONE: Empowering Transparent and Open AI Platforms through Private-Sector HPC Investment in Japan

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SAKURAONE, a 100-node, 800-GPU cluster built entirely on open SONiC-based Ethernet, ranks 49th in the ISC 2025 TOP500 and is stated to be the only top-100 system using a fully open Ethernet fabric.

desk verdict A credible system description and benchmark report for an 800-GPU SONiC/Ethernet cluster; the 'only top-100' claim needs an audit, but the measurements stand on their own. read the letter →

arxiv 2507.02124 v1 pith:OPVDKD7S submitted 2025-07-02 cs.DC cs.NI

Fumikazu Konishi This is my paper · ORCID

classification cs.DCcs.NI

keywords HighperformancecomputingGPUclusterSONiCEthernetinterconnectRoCEv2TOP500HPL-MxPIO500

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports the architecture and benchmark results of SAKURAONE, a 100-node, 800-GPU HPC cluster operated by a Japanese private-sector internet company. The central claim is that a fully open networking stack—800 GbE Ethernet running the SONiC operating system on commodity switch hardware—can support a globally competitive AI/HPC system. SAKURAONE ranked 49th in the ISC 2025 TOP500 list and is stated to be the only system in the top 100 using a fully open SONiC-based Ethernet fabric. If correct, this would show that proprietary interconnects are not required for top-tier performance, lowering cost and vendor lock-in for AI infrastructure.

What carries the argument

The load-bearing element is the open interconnect stack: white-box Edgecore AIS800-64O switches powered by Broadcom Tomahawk 5 ASICs and running SONiC, arranged in a two-pod, rail-optimized leaf-spine topology with full bisection bandwidth at 800 GbE. Each of the 100 nodes' eight H100 GPUs is paired to a dedicated 400 GbE NIC with NUMA-local PCIe connectivity, and RoCEv2 provides the RDMA transport that makes Ethernet competitive with InfiniBand for collective communication. This stack is what the paper uses to prove that open technologies can carry the communication load of a top-50 system.

What would settle it

Check the ISC 2025 TOP500 data for every system ranked 1–100 and inspect the stated interconnect technology; if any other entry uses a fully open SONiC-based Ethernet network, the paper's uniqueness claim is false. A more direct test would be to run the same HPL run (N = 2,706,432, 16×49 grid) on SAKURAONE and confirm the reported 33.95 PFLOPS.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a large-scale AI/HPC cluster built on open, vendor-neutral networking can reach the upper ranks of the TOP500. SAKURAONE delivers 33.95 PFLOPS on HPL, 396.3 TFLOPS on HPCG, and 339.86 PFLOPS on the FP8 HPL-MxP benchmark, and the authors attribute this competitiveness to a full-bisection rail-optimized topology in which every GPU has a direct 400 GbE link to a leaf switch, with 800 GbE spine links and RoCEv2 providing lossless transport. The entire fabric runs on SONiC over Broadcom Tomahawk 5 switches, a combination the paper says appears nowhere else in the top 100.

Load-bearing premise

The claim that SAKURAONE is the only top-100 system with a fully open SONiC-based Ethernet fabric rests on the interconnect details recorded for all other top-100 systems, which the paper does not independently audit; if that external record is incomplete or miscategorized, the uniqueness claim could fail.

Editorial extensions

If this is right

Open Ethernet plus RoCEv2 can carry the communication of an 800-GPU LLM training cluster at top-100 HPL rank, so proprietary fabrics are not a strict requirement.
The rail-optimized full-bisection layout with per-GPU 400 GbE links is a reproducible recipe for other private-sector AI clusters.
The HPL-MxP result of 339.86 PFLOPS in FP8 shows the same system is strongly suited to low-precision AI workloads, not just double-precision HPC.
The IO500 results (rank 9 in the 10-node production category) show the Lustre/DDN storage stack scales with node count, with metadata improving but bandwidth saturating.
The lower HPCG efficiency (about 0.8% of HPL) indicates latency-sensitive sparse workloads remain a weaker point of Ethernet-based systems, pointing to MPI and NUMA tuning as next steps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's claims, if the uniqueness holds we can expect more top-100 systems to adopt SONiC-based Ethernet as RoCEv2 matures, which is a testable trend over the next TOP500 editions.
The cost advantage of white-box SONiC switching could let smaller organizations enter the top-100 class without vendor lock-in, shifting the economics of national AI infrastructure; the paper's Japanese private-sector context suggests this is the intended model.
The 0.8% HPCG/HPL ratio implies the current design is tuned for throughput-bound AI training rather than latency-bound HPC, so a hybrid fabric with a small low-latency tier would be a natural next experiment.
Since the HPL-MxP benchmark used optimized FP8 binaries, the AI performance number reflects software tuning as much as hardware; comparing against a vendor-provided reference implementation would isolate the network's contribution.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a useful, honest system paper. It gives the first detailed public description of an 800-GPU H100 cluster built on a rail-optimized SONiC/Tomahawk 5 Ethernet fabric, with reproducible benchmark parameters and results that are internally consistent. The TOP500 #49 HPL and #12 HPL-MxP numbers are credible given the hardware. The paper's real value is as a data point: a commercially operated, open-stack Ethernet system can sit in the top 50. That's worth having on record.

What's new is the specific system, not the components. The topology, RoCEv2, SONiC, and the benchmarks are all established. The paper's contribution is the integration story and the measured outcomes. It does that well: the configuration tables are detailed, process grids and matrix sizes are given, and the IO500 comparison across 10 and 96 nodes is informative.

The soft spots are minor and one is a bit more than minor. The HPCG paragraph has a unit slip: 437.4 GFLOP/s vs 404.96 TFLOP/s. Table 8 makes clear the intended numbers are in TFLOP/s, so this is a typo, not a red flag. The more substantive issue is the 'only system within the top 100' claim. The TOP500 interconnect field is too coarse to confirm switch operating systems, and the paper provides no audit of the other 99 systems. I agree with the stress-test note: this claim is asserted, not demonstrated. It should be softened to 'among the first' or backed by a table of interconnects for the top 100. The rest of the paper doesn't depend on the exclusivity claim, so this is a fixable weakness, not a load-bearing one.

I also note the paper explicitly acknowledges the lack of energy measurements and plausible performance-efficiency tradeoffs for HPCG on Ethernet; that kind of candor is welcome.

Who should read this: system architects and procurement folks weighing InfiniBand versus open Ethernet, and anyone tracking the SONiC ecosystem in HPC. It is not a methods paper and doesn't need deep theoretical review, but it does deserve a serious referee to check the numbers and push on the exclusivity claim. I'd accept it for peer review.

Referee Report

2 major / 5 minor

Summary. The paper describes SAKURAONE, a 100-node/800-GPU HPC cluster operated by SAKURA Internet in Japan, and reports its performance on four standardized benchmarks: HPL (33.95 PFLOPS, ranked 49th on the June 2025 TOP500), HPCG (396.295 TFLOP/s), HPL-MxP (339.86 PFLOPS in FP8), and IO500 (181.91 score in the 10-node production category). The system uses an 800GbE Ethernet interconnect with RoCEv2, SONiC-based white-box switching, and a rail-optimized topology. The authors argue that the system is the only TOP100 system with a fully open SONiC-based Ethernet stack and that this demonstrates the viability of open, vendor-neutral networking for competitive AI/HPC systems. The paper includes detailed benchmark parameters, hardware descriptions, and system software information.

Significance. If the claims hold, the paper provides a useful data point that an open, vendor-neutral Ethernet fabric can support a globally competitive GPU cluster. The benchmark results appear credible because they are drawn from external, standardized benchmarks with detailed parameter tables, and the reported numbers are consistent with the capabilities of 800 H100 GPUs. The paper's main asserted novelty, however, rests on the claim of uniqueness among TOP500 top-100 systems, which is not supported by an audit. The user-facing benchmark parameters are otherwise sufficiently detailed to permit reproduction, which is a strength. No fitted parameters or circular derivations are involved in the performance claims.

major comments (2)

[Abstract; Section 5, page 11] The abstract and Discussion state that SAKURAONE is 'the only system within the top 100' using a fully open SONiC-based 800GbE networking stack, and that it 'currently holds the highest global ranking' among such systems. This is a universal empirical claim about the other 99 TOP500 systems, but the paper provides no audit, table, or reproducible method showing that no other top-100 system uses a SONiC-based Ethernet fabric. The TOP500 'Interconnect' field does not record switch operating systems, so the cited source cannot settle the claim. Please either supply an explicit enumeration of the interconnect and switch-OS information for all top-100 systems, with sources, or qualify the claim explicitly as 'to the best of the authors' knowledge' and remove it from the abstract as a headline result.
[Section 4, Tables 7–9] The HPL, HPCG, and HPL-MxP runs use 784, 784, and 768 processes, respectively, although the system is described as having 100 compute nodes and 800 GPUs. The paper never explains why fewer than all 800 GPUs are used, nor whether the TOP500 submission corresponds to the full 100-node machine. This matters for the headline Rmax figure, the per-GPU FLOPS values, and the claim that an '800-GPU system' was ranked 49th. Please state the exact node/GPU count used for each benchmark, the reason for any exclusion, and confirm that the TOP500 submission matches the full production system or report the system configuration that was actually benchmarked.

minor comments (5)

[Section 4, HPCG paragraph] The text reads 'the raw computational throughput reached 437.4 GFLOP/s, and after accounting for convergence overhead, the performance was reduced to 404.96 TFLOP/s.' Given Table 8, both intermediate quantities should be in TFLOP/s (437.4 TFLOP/s and 404.96 TFLOP/s). Please correct the unit inconsistency and use the same units in the text and the table.
[Table 3] Table 3 is internally inconsistent and difficult to read: the 'Total' row shows 1, 1, 4, 4, 10 for the years 2020–2024 plus total, while the row totals sum to 20 and each year's top-10 should sum to 10. Please reformat the table so that each year sums to 10 and clarify what is being counted.
[Throughout] There are several typographical issues: 'Luster' should be 'Lustre' in Section 1, Table 1 contains 'Interconnet', Table 4 contains 'grdade', and Figure 2 contains a duplicate label 'OSK=GPU01'. Please correct these.
[Section 4, IO500] The text credits the '10 Node Production' category with 1,280 client processes and a score of 181.91, but Table 10 compares a 10-node and a 96-node configuration without identifying which configuration is the submitted production result. Please clarify the relationship between the two configurations and the IO500 submission.
[Section 2.2] The historical claim that seven of the top ten systems in the November 2024 TOP500 use GbE-based interconnects should cite the specific TOP500 release and list the systems counted; the current text relies on the ambiguous Table 3.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports externally standardized benchmark measurements and hardware specifications, with no fitted parameters, predicted quantities, or load-bearing self-citations.

full rationale

This paper is a system description and benchmark report, not a derivation. The performance numbers (HPL Rmax 33.95 PFLOPS, HPCG 396.295 TFLOPS, HPL-MxP 339.86 PFLOPS, IO500 scores) are measurements produced by standardized external benchmarks with published methodology and independent validation criteria. There are no fitted parameters, no model predictions, and no derived quantities that reduce to the paper's own inputs. The uniqueness claim that SAKURAONE is the only top-100 TOP500 system with a fully open SONiC-based 800GbE stack is an empirical assertion about an external dataset; whether that dataset is sufficiently detailed to support the claim is a verification concern, not a circularity concern, because the claim is not derived from the paper's own definitions or equations. The references cited are external benchmarking and networking studies, and the author does not rely on self-citations for any load-bearing argument. Therefore, no significant circularity is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's central claims rest on external benchmark results and several domain assumptions about interconnect performance and TOP500 data. No free parameters are fitted to data, and no new entities are introduced. The main assumptions are that RoCEv2 performance is comparable to InfiniBand (citing Pichetti et al.), that the rail-optimized topology delivers full bisection bandwidth, and that TOP500 entries accurately record other systems' interconnects.

assumptions (3)

domain assumption RoCEv2 delivers performance comparable to InfiniBand except for latency, per Pichetti et al. 2024
Used in Section 2.2 to justify choosing Ethernet/RoCEv2 over InfiniBand for the interconnect.
domain assumption The rail-optimized topology with 800GbE leaf-spine links provides full bisection bandwidth
The paper asserts full bisection bandwidth in the abstract and Section 2.2, but provides only a schematic (Figure 2) without port-count or measured-bandwidth verification.
domain assumption TOP500 interconnect records for other systems are complete and accurate
The exclusivity claim in Section 5 (only top-100 system with fully open SONiC Ethernet) depends on the correctness of the TOP500 dataset for all other systems; no audit is shown.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SAKURAONE: Empowering Transparent and Open AI Platforms through Private-Sector HPC Investment in Japan." pith.science (2026). https://pith.science/paper/OPVDKD7S

@misc{pith2026250702124,
  author       = {Pith},
  title        = {Pith review of: SAKURAONE: Empowering Transparent and Open AI Platforms through Private-Sector HPC Investment in Japan},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OPVDKD7S}},
  note         = {Machine review of arXiv:2507.02124}
}

read the original abstract

SAKURAONE is a managed high performance computing (HPC) cluster developed and operated by the SAKURA Internet Research Center. It reinforces the ``KOKARYOKU PHY'' configuration of bare-metal GPU servers and is designed as a cluster computing resource optimized for advanced workloads, including large language model (LLM) training. In the ISC 2025 edition of the TOP500 list, SAKURAONE was ranked \textbf{49th} in the world based on its High Performance Linpack (HPL) score, demonstrating its global competitiveness. In particular, it is the \textbf{only system within the top 100} that employs a fully open networking stack based on \textbf{800~GbE (Gigabit Ethernet)} and the \textbf{SONiC (Software for Open Networking in the Cloud)} operating system, highlighting the viability of open and vendor-neutral technologies in large-scale HPC infrastructure. SAKURAONE achieved a sustained performance of 33.95~PFLOP/s on the HPL benchmark (Rmax), and 396.295~TFLOP/s on the High Performance Conjugate Gradient (HPCG) benchmark. For the HPL-MxP benchmark, which targets low-precision workloads representative of AI applications, SAKURAONE delivered an impressive 339.86~PFLOP/s using FP8 precision. The system comprises 100 compute nodes, each equipped with eight NVIDIA H100 GPUs. It is supported by an all-flash Lustre storage subsystem with a total physical capacity of 2~petabytes, providing high-throughput and low-latency data access. Internode communication is enabled by a full-bisection bandwidth interconnect based on a Rail-Optimized topology, where the Leaf and Spine layers are interconnected via 800~GbE links. This topology, in combination with RoCEv2 (RDMA over Converged Ethernet version 2), enables high-speed, lossless data transfers and mitigates communication bottlenecks in large-scale parallel workloads.

Figures

Figures reproduced from arXiv: 2507.02124 by the authors.

**Figure 1.** SAKURAONE System Overview Within Japan, Takano et al. [2024] and Endo et al. [2023] shared high-performance computing services such as ABCI3.0, operated by the National Institute of Advanced Industrial Science and Technology (AIST), and TSUBAME4.0 , provided by the Tokyo Institute of Science, have been made available to industrial users. However, these systems are typically shared with academic users, making it diff… view at source ↗

**Figure 2.** SAKURAONE Network Overview [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 4 canonical work pages

[1]

Abci 3.0: Evolution of the leading ai infrastructure in japan, 2024

Ryousei Takano, Shinichiro Takizawa, Yusuke Tanimura, Hidemoto Nakada, and Hirotaka Ogawa. Abci 3.0: Evolution of the leading ai infrastructure in japan, 2024. URL https://arxiv.org/abs/2411.09134

arXiv 2024
[2]

Challenges in computing resource sharing towards next-gen interactive accelerated hpc

Toshio Endo, Shohei Minami, Akihiro Nomura, Hiroki Ohtsuji, Jun Kato, Masahiro Miwa, Eiji Yoshida, Tomoya Yuki, and Ryuichi Sakamoto. Challenges in computing resource sharing towards next-gen interactive accelerated hpc. In High Performance Computing. ISC High Performance 2024 International Workshops: Hamburg, Germany, May 12–16, 2024, Revised Selected Pa...

work page doi:10.1007/978-3-031-73716-9_16 2024
[3]

Benchmarking ethernet interconnect for hpc/ai workloads

Lorenzo Pichetti, Daniele De Sensi, Karthee Sivalingam, Stepan Nassyr, Daniele Cesarini, Matteo Turisini, Dirk Pleiter, Aldo Artigiani, and Flavio Vella. Benchmarking ethernet interconnect for hpc/ai workloads. In SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 869--875, 2024. doi:1...

arXiv 2024
[4]

Efficient training of large language models on distributed infrastructures: A survey, 2024

Jiangfei Duan, Shuo Zhang, Zerui Wang, Lijuan Jiang, Wenwen Qu, Qinghao Hu, Guoteng Wang, Qizhen Weng, Hang Yan, Xingcheng Zhang, Xipeng Qiu, Dahua Lin, Yonggang Wen, Xin Jin, Tianwei Zhang, and Peng Sun. Efficient training of large language models on distributed infrastructures: A survey, 2024. URL https://arxiv.org/abs/2407.20018

arXiv 2024
[5]

Rail-only: A low-cost high-performance network for training llms with trillion parameters

Weiyang Wang, Manya Ghobadi, Kayvon Shakeri, Ying Zhang, and Naader Hasani. Rail-only: A low-cost high-performance network for training llms with trillion parameters. In 2024 IEEE Symposium on High-Performance Interconnects (HOTI), page 1–10. IEEE, August 2024. doi:10.1109/hoti63208.2024.00013. URL http://dx.doi.org/10.1109/HOTI63208.2024.00013

arXiv 2024
[6]

Dongarra, Piotr Luszczek, and Antoine Petitet

Jack J. Dongarra, Piotr Luszczek, and Antoine Petitet. The linpack benchmark: Past, present, and future. http://www.netlib.org/utk/people/JackDongarra/PAPERS/hpl.pdf, 2001. Online, December 2001

work page 2001
[7]

Toward a new metric for ranking high performance computing systems

Michael Allen Heroux and Jack Dongarra. Toward a new metric for ranking high performance computing systems. Technical report, Sandia National Lab. (SNL-NM), Albuquerque, NM (United States); Univ. of Tennessee, Knoxville, TN (United States), 06 2013. URL https://www.osti.gov/biblio/1089988

work page arXiv 2013
[8]

Azzam Haidar, Stanimire Tomov, Jack Dongarra, and Nicholas J. Higham. Harnessing gpu tensor cores for fast fp16 arithmetic to speed up mixed-precision iterative refinement solvers. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC '18. IEEE Press, 2019. doi:10.1109/SC.2018.00050. URL https...

arXiv 2019

Show all 9 references

[9]

Establishing the io-500 benchmark

J Kunkel, John Bent, Jay Lofstead, and George S Markomanolis. Establishing the io-500 benchmark. White Paper, 2016

2016

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Abci 3.0: Evolution of the leading ai infrastructure in japan, 2024

Ryousei Takano, Shinichiro Takizawa, Yusuke Tanimura, Hidemoto Nakada, and Hirotaka Ogawa. Abci 3.0: Evolution of the leading ai infrastructure in japan, 2024. URL https://arxiv.org/abs/2411.09134

arXiv 2024

[2] [2]

Challenges in computing resource sharing towards next-gen interactive accelerated hpc

Toshio Endo, Shohei Minami, Akihiro Nomura, Hiroki Ohtsuji, Jun Kato, Masahiro Miwa, Eiji Yoshida, Tomoya Yuki, and Ryuichi Sakamoto. Challenges in computing resource sharing towards next-gen interactive accelerated hpc. In High Performance Computing. ISC High Performance 2024 International Workshops: Hamburg, Germany, May 12–16, 2024, Revised Selected Pa...

work page doi:10.1007/978-3-031-73716-9_16 2024

[3] [3]

Benchmarking ethernet interconnect for hpc/ai workloads

Lorenzo Pichetti, Daniele De Sensi, Karthee Sivalingam, Stepan Nassyr, Daniele Cesarini, Matteo Turisini, Dirk Pleiter, Aldo Artigiani, and Flavio Vella. Benchmarking ethernet interconnect for hpc/ai workloads. In SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 869--875, 2024. doi:1...

arXiv 2024

[4] [4]

Efficient training of large language models on distributed infrastructures: A survey, 2024

Jiangfei Duan, Shuo Zhang, Zerui Wang, Lijuan Jiang, Wenwen Qu, Qinghao Hu, Guoteng Wang, Qizhen Weng, Hang Yan, Xingcheng Zhang, Xipeng Qiu, Dahua Lin, Yonggang Wen, Xin Jin, Tianwei Zhang, and Peng Sun. Efficient training of large language models on distributed infrastructures: A survey, 2024. URL https://arxiv.org/abs/2407.20018

arXiv 2024

[5] [5]

Rail-only: A low-cost high-performance network for training llms with trillion parameters

Weiyang Wang, Manya Ghobadi, Kayvon Shakeri, Ying Zhang, and Naader Hasani. Rail-only: A low-cost high-performance network for training llms with trillion parameters. In 2024 IEEE Symposium on High-Performance Interconnects (HOTI), page 1–10. IEEE, August 2024. doi:10.1109/hoti63208.2024.00013. URL http://dx.doi.org/10.1109/HOTI63208.2024.00013

arXiv 2024

[6] [6]

Dongarra, Piotr Luszczek, and Antoine Petitet

Jack J. Dongarra, Piotr Luszczek, and Antoine Petitet. The linpack benchmark: Past, present, and future. http://www.netlib.org/utk/people/JackDongarra/PAPERS/hpl.pdf, 2001. Online, December 2001

work page 2001

[7] [7]

Toward a new metric for ranking high performance computing systems

Michael Allen Heroux and Jack Dongarra. Toward a new metric for ranking high performance computing systems. Technical report, Sandia National Lab. (SNL-NM), Albuquerque, NM (United States); Univ. of Tennessee, Knoxville, TN (United States), 06 2013. URL https://www.osti.gov/biblio/1089988

work page arXiv 2013

[8] [8]

Azzam Haidar, Stanimire Tomov, Jack Dongarra, and Nicholas J. Higham. Harnessing gpu tensor cores for fast fp16 arithmetic to speed up mixed-precision iterative refinement solvers. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC '18. IEEE Press, 2019. doi:10.1109/SC.2018.00050. URL https...

arXiv 2019