Pith. sign in

REVIEW 4 major objections 4 minor 100 references

FlexiNS: A SmartNIC-Centric, Line-Rate and Flexible Network Stack

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FlexiNS claims an off-path SmartNIC can run a software-defined network stack at full-duplex line rate.

desk verdict A serious SmartNIC network stack paper with a genuinely new combination of mechanisms and plausible measurements, but the headline full-duplex line-rate claim rests on an experiment that only shows TX throughput, so it needs a real bidirectional test before the central claim is accepted. read the letter →

arxiv 2504.18432 v1 pith:PJUEUGFT submitted 2025-04-25 cs.NI

Xuzheng Chen , Jie Zhang , Baolin Zhu , Xueying Zhu , Zhongqing Chen , Shu Ma , Lingjun Zhu , Chao Shi
show 2 more authors
Yin Zhang Zeke Wang
This is my paper · ORCID
classification cs.NI
keywords SmartNICnetworkstackoffloadline-rateprocessingRDMABlueField-3in-cacheRXDMAnotificationtransportprogrammability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to resolve a three-way tension in data-center networking: CPU-based stacks are programmable but consume too many cores and too much memory bandwidth, hardware-offloaded NICs are fast but fixed-function, and naive offloading to a SmartNIC's Arm processor is flexible but bounded by Arm memory and link bandwidth. FlexiNS claims to break this trade-off by keeping packet payloads out of Arm memory entirely: Arm cores build headers and run transport logic while NIC DMA engines move data directly between host memory and the wire, and received packets are processed in cache and then invalidated rather than written back. If the claim holds, cloud providers can get RDMA-like throughput while writing their own transport protocols in ordinary software. The prototype on an off-the-shelf BlueField-3 reports full-duplex line rate, $2.2\times$ higher block-storage IOPS than a microkernel stack, and $1.3\times$ higher KVCache transfer throughput than hardware-offloaded RDMA.

What carries the argument

Four mechanisms carry the design. The header-only offloading TX path constructs custom headers on Arm and uses a shadow memory region so the NIC fetches host payloads directly. The unlimited-working-set in-cache processing RX path keeps received packets in the Arm LLC, transfers payloads to the host by DMA, then invalidates the cache lines so no write-back occurs. The DMA-only notification pipe replaces slow emulated MMIO and doorbell round-trips with lockless producer-consumer DMA queues. A programmable offloading engine lets cloud providers attach custom functions to registered opcodes. The shadow memory region is the load-bearing piece: it maps host virtual addresses to Arm virtual addresses so the Arm can build work-queue entries without touching payload bytes.

What would settle it

Monitor Arm DRAM write traffic with hardware performance counters while FlexiNS receives at full rate with an RX working set several times larger than the Arm LLC. If write-back traffic approaches the naive DMA baseline as the working set grows, the unlimited-working-set in-cache RX path is not preventing evictions. A second check: run the same binary on a SmartNIC whose cache-invalidate opcode is not fused with DMA, and see whether line-rate throughput collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that an off-path SmartNIC, a NIC with its own Arm processor sitting beside the data path, can run a programmable network stack at full-duplex line rate, a capability previously split between CPU stacks (flexible but slow) and hardware-offloaded NICs (fast but fixed). FlexiNS achieves this by never letting packet payloads cross the Arm: on transmit the Arm builds only headers while NIC hardware fetches the host payload directly, and on receive packets are processed in the Arm's last-level cache and then explicitly invalidated instead of being written back to memory. On a BlueField-3 prototype it reports full-duplex line rate, single-connection throughput comparable to a ConnectX-7 RDMA NIC, $1.4\times$ lower latency than the microkernel-based Snap stack, $2.2\times$ higher IOPS for disaggregated block storage, and $1.3\times$ higher throughput for KVCache transfer.

Load-bearing premise

The whole design relies on BlueField-3 hardware doing two things exactly as assumed: letting the NIC fetch host payloads through a shadow address mapping without engaging Arm memory bandwidth, and invalidating processed cache lines without writing them back; on another NIC, or under full-duplex stress, either of those could fail and the line-rate claim would not transfer.

Editorial extensions

If this is right

  • A single off-path SmartNIC can deliver full-duplex line rate while leaving host CPU and host memory bandwidth almost untouched.
  • Transport protocols written in C/C++, including RoCEv2, Solar, and custom congestion control, can run at RDMA-comparable throughput, letting cloud operators ship protocol changes at software speed.
  • Latency-sensitive MTU-sized two-sided operations can be optimized to within about $2\,\mu$s of a hardware RDMA NIC while keeping programmability.
  • Applications using standard RDMA IBV verbs can run on FlexiNS without modification, easing adoption by existing RDMA-based systems.
  • The programmable offloading engine can move application work such as linked-list traversal and batched reads onto the SmartNIC, cutting latency $1.7\times$ and raising single-connection read throughput $3.5\times$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the hardware assumptions hold on future SmartNICs, the same design should scale toward 800 GbE with only about 2 MB of LLC budget for the RX path, as the paper's cache-sizing argument implies.
  • The header-only TX and in-cache RX split suggests a general offload principle: any data-plane processing that does not touch payload bytes can stay on the Arm, while all payload movement belongs to DMA engines; applying it to encryption, compression, or storage offload is a natural extension the paper does not develop.
  • A direct testable consequence is that FlexiNS's RX path should show near-zero Arm DRAM write traffic under an LLC-exceeding working set; measuring that would separate the mechanism from the aggregate throughput numbers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents FlexiNS, a network stack running on the Arm cores of an off-path BlueField-3 SmartNIC, with the goal of combining software transport programmability with line-rate packet processing. The main mechanisms are a header-only TX path that uses a host-to-Arm shadow memory mapping so the NIC fetches payloads directly from host memory, an RX path that processes packets in the Arm LLC and uses explicit cache invalidation to avoid write-backs, a DMA-only notification pipe between host and Arm, and a programmable offloading engine with a small API. The authors prototype the stack in about 11,000 lines of code, expose an IBV-verbs-compatible interface, and evaluate it against an RDMA NIC baseline (ConnectX-7), a self-built Snap-style microkernel stack, a Solar-CPU baseline, and Mooncake. The central claims are that FlexiNS sustains full-duplex line rate, achieves single-flow throughput comparable to an RDMA NIC, and delivers 2.2x higher IOPS than the microkernel baseline in block storage and 1.3x higher throughput than Mooncake RDMA in KVCache transfer.

Significance. If the headline claims hold, the paper would make a strong contribution: it would demonstrate that a software-defined transport can run on an off-path SmartNIC at line rate while keeping host CPU and memory overhead near zero, and that this can be done without sacrificing ecosystem compatibility. The paper is especially valuable for its concrete design details: the shadow memory region, explicit cache invalidation on the Arm, the shared send queue, and the DMA-only notification pipe are all described at a level that would allow replication. The programmable offloading engine is also a useful concrete step beyond fixed-function RDMA. The evaluation covers throughput, latency, memory-bandwidth effects, and two end-to-end workloads, which is substantial. However, the absence of artifacts, the reliance on an author-built unvalidated Snap baseline, and the ambiguity of the full-duplex and packet-spraying experiments mean the central quantitative claims are not yet fully supported; the paper is promising but needs revision before the claims can be accepted at face value.

major comments (4)
  1. [§5.3, Figure 13] The full-duplex line-rate claim made in the abstract and Section 1 is not directly demonstrated. Figure 13 reports only aggregate TX throughput while a 400 Gbps RX flow is inserted; it does not report the achieved RX throughput, packet loss, or whether TX and RX share the same 200GbE port or use separate ports. This matters because the B3220 has two 200GbE ports and the preceding paragraph states that two connections are used to cover both ports, so an RX flow on the second port would not demonstrate full-duplex operation on a single link. Please report per-port TX and RX throughput and loss, and state explicitly the port configuration used in Figure 13.
  2. [§5.7, Figure 18] The claimed 1.3x throughput improvement over Mooncake RDMA in KVCache transfer is not an isolated comparison of the network stacks. The FlexiNS configuration adds packet spraying via dynamic source UDP port variation and bonds both physical ports, while Mooncake uses its default QP selection, which the text says causes hash collision and underutilization. The speedup could therefore come from multipath utilization rather than from FlexiNS's stack design. Please compare FlexiNS with and without packet spraying, or compare Mooncake with an equivalent multi-path configuration, to attribute the 1.3x improvement.
  3. [§5.1, §5.2] The Snap baseline is author-built from the paper's description rather than taken from a public implementation, and no validation against published Snap performance numbers is provided. Since the claim 'up to 3.5x higher throughput than Snap' is one of the headline results, this baseline needs more support. Please provide the configuration details, any tuning parameters, and ideally a comparison against the original Snap numbers or a released artifact, and report variance across repeated runs for all key figures.
  4. [§3.3, §5.4] The unlimited-working-set RX path rests on a specific hardware behavior: that explicit cache invalidation prevents write-backs and that incoming DMA writes can reuse invalidated cachelines without eviction. The paper asserts this from the BF3 prototype but does not directly measure the cache behavior (e.g., cache miss rates, write-back counts, or invalidation cost). Since this mechanism is the core reason the design reaches line rate and the claim is hardware-specific, please add a direct characterization or at least a clear statement of which BF3 capabilities the result depends on and why they should hold on other off-path SmartNICs.
minor comments (4)
  1. [General] There are several typos and inconsistencies, including 'Evalution' in the Section 5 heading, 'Conclusioin' in Section 7, 'simontaneously' in Section 2, 'AES encryptio' in Section 3.2, and an observation list in Section 5.4 that introduces a 'Third' point after stating 'We have two observations.' These should be cleaned up.
  2. [§5.3] The text says 'since BF3 has two ports and each connection only uses one port, we use two connections' and then refers to 'line rate' for the aggregated throughput. Please clarify whether 'line rate' means 400 Gbps aggregate over two 200GbE ports or per-port 200 Gbps, and make the y-axis labels explicit about aggregation.
  3. [§5.2, Figure 11] The sentence 'FlexiNS also keeps minimal host memory bandwidth occupation and reduces the 2.7× bandwidth compared with Snap' is grammatically unclear; please rewrite to state which system has 2.7x lower or higher bandwidth.
  4. [§5.6] The programmable offloading engine section would benefit from a brief discussion of the overhead of registering an opcode and the isolation between the engine cores and the network stack cores, especially since the engine is described as using dedicated Arm cores.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are supported by comparative measurements against external baselines, not by fitted parameters or self-referential definitions.

full rationale

FlexiNS is a systems paper whose central claims ('full-duplex line-rate', 'comparable single-flow throughput with RDMA NIC', '2.2x higher throughput than microkernel-based baseline', '1.3x higher throughput than hardware-offloaded baseline') are empirical comparisons against external systems: RNIC (ConnectX-7), a reconstructed Snap-like microkernel baseline, Solar-CPU, and Mooncake. There is no mathematical derivation chain in which an output is defined in terms of an input, and no parameter fitting step that is later renamed as a prediction. The design components (header-only TX, in-cache RX, DMA-only notification, offloading engine) are argued from architectural bottlenecks and then measured; the bottleneck analysis itself is not circular because the claimed bottleneck (Arm-NIC switch link contention, Arm memory bandwidth exhaustion) is independently demonstrated with a naive implementation and hardware measurement. Self-citations appear in the references ([9], the LibR benchmark tool, and [10], a prior characterization of off-path SmartNIC datapath accelerators), but they are not load-bearing in the sense of defining the paper's results: [10] is used for a hardware bandwidth figure and [9] is a benchmark harness, both of which are externally measurable platform facts rather than unverified premises imported to force the conclusion. The closest concern to circularity is the full-duplex line-rate evaluation in Figure 13, where only TX throughput is plotted while a 400 Gbps RX flow is inserted; this is an evidentiary gap about whether full-duplex was directly demonstrated, not a circular reduction. No equation, definition, or fitted parameter makes the claimed result equivalent to its own input by construction. Therefore, under the requirement to exhibit a specific reduction before claiming circularity, the correct finding is no significant circularity with score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

There are no fitted mathematical parameters. The unproved premises are hardware capabilities of the BlueField-3, such as shadow memory mapping, DDIO to the Arm LLC, cache invalidation, and DMA engines, plus workload assumptions such as header-only processing and bounded packet bursts. These are stated in the design sections and are not independently validated beyond the prototype measurements.

assumptions (4)
  • domain assumption The network stack never needs to read or modify payload bytes; it only processes headers.
    Section 3.2 states this as the key insight for the header-only TX path; if false, the payload would have to cross Arm memory and the bandwidth claim fails.
  • domain assumption BF3's NIC can use an Arm-side shadow memory region to fetch host payloads directly and merge them with Arm-built headers at line rate.
    Required by Section 3.2 and Figure 7; only shown on the FlexiNS prototype, not characterized independently.
  • domain assumption BF3 supports DDIO into the Arm LLC and an explicit cache-invalidation opcode that discards clean cache lines without write-back.
    Required by Section 3.3; Appendix A.2 says the invalidation uses a BF3-specific NIC opcode, and the authors note they had to discuss a future combined DMA and invalidation opcode with Mellanox engineers.
  • domain assumption The required LLC size can be computed as bandwidth times average packet processing time, and bursts will not exceed the remaining cache.
    Section 3.3 computes 500KB for 400 Gbps and 10us average processing time; if processing time or burst distribution has heavy tails, packets will be evicted and memory bandwidth will be consumed.
invented entities (1)
  • Shadow memory region
    purpose: Exposes host payload buffers to Arm-side WQE construction without copying payload into Arm memory.
    Introduced in Section 3.2 as a new software and hardware abstraction: Arm virtual addresses that map to host payload buffers without physical Arm memory. It exists only inside FlexiNS; no independent measurement is provided outside the prototype.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FlexiNS: A SmartNIC-Centric, Line-Rate and Flexible Network Stack." pith.science (2026). https://pith.science/paper/PJUEUGFT

@misc{pith2026250418432,
  author       = {Pith},
  title        = {Pith review of: FlexiNS: A SmartNIC-Centric, Line-Rate and Flexible Network Stack},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PJUEUGFT}},
  note         = {Machine review of arXiv:2504.18432}
}
abstract

As the gap between network and CPU speeds rapidly increases, the CPU-centric network stack proves inadequate due to excessive CPU and memory overhead. While hardware-offloaded network stacks alleviate these issues, they suffer from limited flexibility in both control and data planes. Offloading network stack to off-path SmartNIC seems promising to provide high flexibility; however, throughput remains constrained by inherent SmartNIC architectural limitations. To this end, we design FlexiNS, a SmartNIC-centric network stack with software transport programmability and line-rate packet processing capabilities. To grapple with the limitation of SmartNIC-induced challenges, FlexiNS introduces: (a) a header-only offloading TX path; (b) an unlimited-working-set in-cache processing RX path; (c) a high-performance DMA-only notification pipe; and (d) a programmable offloading engine. We prototype FlexiNS using Nvidia BlueField-3 SmartNIC and provide out-of-the-box RDMA IBV verbs compatibility to users. FlexiNS achieves 2.2$\times$ higher throughput than the microkernel-based baseline in block storage disaggregation and 1.3$\times$ higher throughput than the hardware-offloaded baseline in KVCache transfer.

Figures

Figures reproduced from arXiv: 2504.18432 by the authors.

Figure 1
Figure 1. Comparison of different network stack designs. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of different stacks achieved throughput [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. High-level system architecture and TX/RX data [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (11 more)
Figure 6
Figure 6. Figure 6: Comparison of TX path strategies. constructs the corresponding packet header and merges it with the payload into a contiguous buffer ( 3 ). At last, the Arm transmits the assembled packet to the network ( 4 ). This approach benefits from standard IBV verbs, making it s…
Figure 7
Figure 7. Figure 7: Architecture of Shadow Memory Region and High [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Comparison of RX path strategies [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Architecture of unlimited-working-set in-cache pro [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: RDMA SEND/WRITE throughput and latency between a pair of connections on different hosts. (a) Aggreagate WRITE throughput (b) Host memory bandwidth occupation [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Aggreagate RDMA WRITE throughput of multiple [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Comparison of RDMA WRITE throughput under 2 [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 13
Figure 13. Figure 13: Using 8 connections and 2KB RDMA WRITE payload to achieve each TX design upper limit, and insert a 400Gbps RX flow at the 1 second that lasts 5 seconds. with Snap. This efficiency is attributed to its SmartNIC-based packet handling and zero-overhead on the host CPU. 5…
Figure 16
Figure 16. Figure 16: Programmable offloading functions performance. [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]
Figure 17
Figure 17. Figure 17: Performance comparison of four approaches when [PITH_FULL_IMAGE:figures/full_fig_p012_17.png]
Figure 18
Figure 18. Figure 18: Throughput comparison of KVCache transfer. [PITH_FULL_IMAGE:figures/full_fig_p012_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

100 extracted references · 79 canonical work pages

  1. [1]

    Idio: Network-driven, inbound net- work data orchestration on server processors

    Mohammad Alian, Siddharth Agarwal, Jongmin Shin, Neel Patel, Yifan Yuan, Daehoon Kim, Ren Wang, and Nam Sung Kim. Idio: Network-driven, inbound net- work data orchestration on server processors. In MI- CRO. IEEE, 2022

    2022

  2. [2]

    Skip it: Take control of your cache! In ASPLOS, 2024

    Shashank Anand, Michal Friedman, Michael Giardino, and Gustavo Alonso. Skip it: Take control of your cache! In ASPLOS, 2024

    2024

  3. [3]

    Enabling programmable transport protocols in{High-Speed}{NICs}

    Mina Tahmasbi Arashloo, Alexey Lavrov, Manya Ghobadi, Jennifer Rexford, David Walker, and David Wentzlaff. Enabling programmable transport protocols in{High-Speed}{NICs}. In NSDI, 2020

    2020

  4. [4]

    Empowering azure storage with {RDMA}

    Wei Bai, Shanim Sainul Abdeen, Ankit Agrawal, Kr- ishan Kumar Attre, Paramvir Bahl, Ameya Bhagat, Gowri Bhaskara, Tanya Brokhman, Lei Cao, Ahmad Cheema, et al. Empowering azure storage with {RDMA}. In NSDI, 2023

    2023

  5. [5]

    BCM958804-PS1100R

    Broadcom. BCM958804-PS1100R. https: //gtmteknoloji.com/wp-content/uploads/ 2020/08/PS1100R-PB100.pdf, 2020

    2020

  6. [6]

    Understanding host network stack overheads

    Qizhe Cai, Shubham Chaudhary, Midhul Vuppalapati, Jaehyun Hwang, and Rachit Agarwal. Understanding host network stack overheads. In SIGCOMM, 2021

    2021

  7. [7]

    Towards µ s tail latency and terabit ethernet: disaggregating the host network stack

    Qizhe Cai, Midhul Vuppalapati, Jaehyun Hwang, Chris- tos Kozyrakis, and Rachit Agarwal. Towards µ s tail latency and terabit ethernet: disaggregating the host network stack. In SIGCOMM, 2022

    2022

  8. [8]

    Mp-rdma: enabling rdma with multi-path transport in datacenters

    Guo Chen, Yuanwei Lu, Bojie Li, Kun Tan, Yongqiang Xiong, Peng Cheng, Jiansong Zhang, and Thomas Moscibroda. Mp-rdma: enabling rdma with multi-path transport in datacenters. TON, 2019

    2019

Show all 100 references
  1. [9]

    Libr: Yet another rdma perftest

    Xuzheng Chen and Jie Zhang. Libr: Yet another rdma perftest. https://github.com/carlzhang4/libr, 2025

    2025

  2. [10]

    Demystifying datapath acceler- ator enhanced off-path smartnic

    Xuzheng Chen, Jie Zhang, Ting Fu, Yifan Shen, Shu Ma, Kun Qian, Lingjun Zhu, Chao Shi, Yin Zhang, Ming Liu, et al. Demystifying datapath acceler- ator enhanced off-path smartnic. arXiv preprint arXiv:2402.03041, 2024

    2024 arXiv

  3. [11]

    Data Plane Development Kit

    DPDK. Data Plane Development Kit. https://www. dpdk.org/, 2025

    2025

  4. [12]

    In NSDI, 2014

    Aleksandar Dragojevi´c, Dushyanth Narayanan, Miguel Castro, and Orion Hodson.{FaRM}: Fast remote mem- ory. In NSDI, 2014

    2014

  5. [13]

    No com- promises: distributed transactions with consistency, availability, and performance

    Aleksandar Dragojevi ´c, Dushyanth Narayanan, Ed- mund B Nightingale, Matthew Renzelmann, Alex Shamis, Anirudh Badam, and Miguel Castro. No com- promises: distributed transactions with consistency, availability, and performance. In SOSP, 2015

    2015

  6. [14]

    Make the most out of last level cache in intel processors

    Alireza Farshin, Amir Roozbeh, Gerald Q Maguire Jr, and Dejan Kosti´c. Make the most out of last level cache in intel processors. In EuroSys, 2019

    2019

  7. [15]

    Reexamining direct cache access to optimize {I/O} intensive applications for multi- hundred-gigabit networks

    Alireza Farshin, Amir Roozbeh, Gerald Q Maguire Jr, and Dejan Kosti´c. Reexamining direct cache access to optimize {I/O} intensive applications for multi- hundred-gigabit networks. In ATC, 2020

    2020

  8. [16]

    Caladan: Mitigating interference at mi- crosecond timescales

    Joshua Fried, Zhenyuan Ruan, Amy Ousterhout, and Adam Belay. Caladan: Mitigating interference at mi- crosecond timescales. In OSDI, 2020

    2020

  9. [17]

    Rdma over ethernet for distributed training at meta scale

    Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, et al. Rdma over ethernet for distributed training at meta scale. In SIGCOMM, 2024

    2024

  10. [18]

    perftest: Infiniband verbs performance tests

    sshaulnv gilr8, HassanKhadour. perftest: Infiniband verbs performance tests. https://github.com/ linux-rdma/perftest, 2025

    2025

  11. [19]

    Google Falcon

    Google Falcon. Google Falcon. https://github. com/opencomputeproject/OCP-NET-Falcon, 2024

    2024

  12. [20]

    Rdma over commodity ethernet at scale

    Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitu Padhye, and Marina Lipshteyn. Rdma over commodity ethernet at scale. InSIGCOMM, 2016

    2016

  13. [21]

    Roud: Scal- able rdma over ud in lossy data center networks

    Zhiqiang He, Yuxin Chen, and Bei Hua. Roud: Scal- able rdma over ud in lossy data center networks. In CCGrid, 2023

    2023

  14. [22]

    Masq: Rdma for virtual private cloud

    Zhiqiang He, Dongyang Wang, Binzhang Fu, Kun Tan, Bei Hua, Zhi-Li Zhang, and Kai Zheng. Masq: Rdma for virtual private cloud. In SIGCOMM, 2020

    2020

  15. [23]

    Infiniband architecture

    Infiniband. Infiniband architecture. https://www. infinibandta.org/, 2008

    2008

  16. [24]

    RoCEv2 Update from the IBTA

    Infiniband. RoCEv2 Update from the IBTA. https://www.openfabrics.org/images/ eventpresos/workshops2014/DevWorkshop/ presos/Wednesday/pdf/02_RoCEv2forOFA.pdf, 2014

    2014

  17. [25]

    Intel Memory Latency Checker

    Intel. Intel Memory Latency Checker. https://www.intel.com/content/www/ 13 us/en/developer/articles/tool/ intelr-memory-latency-checker.html , 2024

    2024

  18. [26]

    Intel ® Infrastructure Processing Unit

    Intel. Intel ® Infrastructure Processing Unit. https://www.intel.com/content/www/us/en/ products/details/network-io/ipu.html, 2024

    2024

  19. [27]

    JEDEC DDR5 compare

    JEDEC. JEDEC DDR5 compare. https://www. jedec.org/category/technology-focus-area/ main-memory-ddr3-ddr4-sdram , 2024

    2024

  20. [28]

    Os2g: A high- performance dpu offloading architecture for gpu-based deep learning with object storage

    Zhen Jin, Yiquan Chen, Mingxu Liang, Yijing Wang, Guoju Fang, Ao Zhou, Keyao Zhang, Jiexiong Xu, Wenhai Lin, Yiquan Lin, et al. Os2g: A high- performance dpu offloading architecture for gpu-based deep learning with object storage. In ASPLOS, 2025

    2025

  21. [29]

    Shinjuku: Preemptive scheduling for{µsecond-scale} tail latency

    Kostis Kaffes, Timothy Chong, Jack Tigar Humphries, Adam Belay, David Mazières, and Christos Kozyrakis. Shinjuku: Preemptive scheduling for{µsecond-scale} tail latency. In NSDI, 2019

    2019

  22. [30]

    Design guidelines for high performance{RDMA} systems

    Anuj Kalia, Michael Kaminsky, and David G Ander- sen. Design guidelines for high performance{RDMA} systems. In ATC, 2016

    2016

  23. [31]

    {FaSST}: Fast, scalable and simple distributed transactions with two-sided rdma datagram rpcs

    Anuj Kalia, Michael Kaminsky, and David G Ander- sen. {FaSST}: Fast, scalable and simple distributed transactions with two-sided rdma datagram rpcs. In OSDI, 2016

    2016

  24. [32]

    Cdpu: Co-designing compression and decompression processing units for hyperscale systems

    Sagar Karandikar, Aniruddha N Udipi, Junsun Choi, Joonho Whangbo, Jerry Zhao, Svilen Kanev, Edwin Lim, Jyrki Alakuijala, Vrishab Madduri, Yakun Sophia Shao, et al. Cdpu: Co-designing compression and decompression processing units for hyperscale systems. In ISCA, 2023

    2023

  25. [33]

    Tas: Tcp acceleration as an os service

    Antoine Kaufmann, Tim Stamler, Simon Peter, Naveen Kr Sharma, Arvind Krishnamurthy, and Thomas Anderson. Tas: Tcp acceleration as an os service. In EuroSys, 2019

    2019

  26. [34]

    In NSDI, 2019

    Daehyeok Kim, Tianlong Yu, Hongqiang Harry Liu, Yibo Zhu, Jitu Padhye, Shachar Raindel, Chuanxiong Guo, Vyas Sekar, and Srinivasan Seshan.{FreeFlow}: Software-based virtual{RDMA} networking for con- tainerized clouds. In NSDI, 2019

    2019

  27. [35]

    Linefs: Efficient smart- nic offload of a distributed file system with pipeline parallelism

    Jongyul Kim, Insu Jang, Waleed Reda, Jaeseong Im, Marco Canini, Dejan Kosti´c, Youngjin Kwon, Simon Peter, and Emmett Witchel. Linefs: Efficient smart- nic offload of a distributed file system with pipeline parallelism. In SOSP, 2021

    2021

  28. [36]

    Rearchitecting the tcp stack for i/o-offloaded content delivery

    Taehyun Kim, Deondre Martin Ng, Junzhi Gong, Youngjin Kwon, Minlan Yu, and KyoungSoo Park. Rearchitecting the tcp stack for i/o-offloaded content delivery. In NSDI, 2023

    2023

  29. [37]

    Understanding {RDMA} microarchitecture resources for performance isolation

    Xinhao Kong, Jingrong Chen, Wei Bai, Yechen Xu, Mahmoud Elhaddad, Shachar Raindel, Jitendra Padhye, Alvin R Lebeck, and Danyang Zhuo. Understanding {RDMA} microarchitecture resources for performance isolation. In NSDI, 2023

    2023

  30. [38]

    Splitrpc: A {Control+ Data} path splitting rpc stack for ml inference serving

    Adithya Kumar, Anand Sivasubramaniam, and Timo- thy Zhu. Splitrpc: A {Control+ Data} path splitting rpc stack for ml inference serving. POMACS, 2023

    2023

  31. [39]

    Swift: Delay is simple and ef- fective for congestion control in the datacenter

    Gautam Kumar, Nandita Dukkipati, Keon Jang, Has- san MG Wassel, Xian Wu, Behnam Montazeri, Yao- gong Wang, Kevin Springborn, Christopher Alfeld, Michael Ryan, et al. Swift: Delay is simple and ef- fective for congestion control in the datacenter. In SIGCOMM, 2020

    2020

  32. [40]

    A quan- titative analysis and guidelines of data streaming ac- celerator in modern intel xeon scalable processors

    Reese Kuper, Ipoom Jeong, Yifan Yuan, Ren Wang, Narayan Ranganathan, Nikhil Rao, Jiayu Hu, Sanjay Kumar, Philip Lantz, and Nam Sung Kim. A quan- titative analysis and guidelines of data streaming ac- celerator in modern intel xeon scalable processors. In ASPLOS, 2024

    2024

  33. [41]

    From rdma to rdca: Toward high-speed last mile of data center networks using remote direct cache access

    Qiang Li, Qiao Xiang, Derui Liu, Yuxin Wang, Hao- nan Qiu, Xiaoliang Wang, Jie Zhang, Ridi Wen, Hao- hao Song, Gexiao Tian, et al. From rdma to rdca: Toward high-speed last mile of data center networks using remote direct cache access. arXiv preprint arXiv:2211.05975, 2022

    2022 arXiv

  34. [42]

    Hpcc: High precision congestion control

    Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng, Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, et al. Hpcc: High precision congestion control. In SIGCOMM, 2019

    2019

  35. [43]

    Deepseek- v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek- v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  36. [44]

    Fuyao: Dpu-enabled direct data transfer for serverless computing

    Guowei Liu, Laiping Zhao, Yiming Li, Zhaolin Duan, Sheng Chen, Yitao Hu, Zhiyuan Su, and Wenyu Qu. Fuyao: Dpu-enabled direct data transfer for serverless computing. In ASPLOS, 2024

    2024

  37. [45]

    Offloading distributed applications onto smartnics using ipipe

    Ming Liu, Tianyi Cui, Henry Schuh, Arvind Krishna- murthy, Simon Peter, and Karan Gupta. Offloading distributed applications onto smartnics using ipipe. In SIGCOMM. 2019

    2019

  38. [46]

    Harmonic: Hardware-assisted{RDMA} performance isolation for public clouds

    Jiaqi Lou, Xinhao Kong, Jinghan Huang, Wei Bai, Nam Sung Kim, and Danyang Zhuo. Harmonic: Hardware-assisted{RDMA} performance isolation for public clouds. In NSDI, 2024. 14

    2024

  39. [47]

    Snap: A microkernel approach to host networking

    Michael Marty, Marc de Kruijf, Jacob Adriaens, Christopher Alfeld, Sean Bauer, Carlo Contavalli, Michael Dalton, Nandita Dukkipati, William C Evans, Steve Gribble, et al. Snap: A microkernel approach to host networking. In SOSP, 2019

    2019

  40. [48]

    ConnectX ®-6 DX Card

    Mellanox. ConnectX ®-6 DX Card. https: //www.nvidia.com/content/dam/en-zz/ Solutions/networking/ethernet-adapters/ connectX-6-dx-datasheet.pdf , 2022

    2022

  41. [49]

    ConnectX ®-7 EN Card

    Mellanox. ConnectX ®-7 EN Card. https: //www.nvidia.com/content/dam/en-zz/ Solutions/networking/ethernet-adapters/ connectx-7-datasheet-Final.pdf , 2023

    2023

  42. [50]

    ConnectX ®-8 EN Card

    Mellanox. ConnectX ®-8 EN Card. https://resources.nvidia.com/ en-us-accelerated-networking-resource-library/ connectx-datasheet-c, 2024

    2024

  43. [51]

    From luna to solar: the evolutions of the compute-to-storage networks in alibaba cloud

    Rui Miao, Lingjun Zhu, Shu Ma, Kun Qian, Shu- jun Zhuang, Bo Li, Shuguang Cheng, Jiaqi Gao, Yan Zhuang, Pengcheng Zhang, et al. From luna to solar: the evolutions of the compute-to-storage networks in alibaba cloud. In SIGCOMM, 2022

    2022

  44. [52]

    Revisiting network support for rdma

    Radhika Mittal, Alexander Shpiner, Aurojit Panda, Ei- tan Zahavi, Arvind Krishnamurthy, Sylvia Ratnasamy, and Scott Shenker. Revisiting network support for rdma. In SIGCOMM, 2018

    2018

  45. [53]

    AccelTCP: Accelerat- ing network applications with stateful TCP offloading

    YoungGyoun Moon, SeungEon Lee, Muhammad Asim Jamshed, and KyoungSoo Park. AccelTCP: Accelerat- ing network applications with stateful TCP offloading. In NSDI, 2020

    2020

  46. [54]

    Netkernel: Making network stack part of the virtualized infrastructure

    Zhixiong Niu, Qiang Su, Peng Cheng, Yongqiang Xiong, Dongsu Han, Keith Winstein, Chun Jason Xue, and Hong Xu. Netkernel: Making network stack part of the virtualized infrastructure. TON, 2021

    2021

  47. [55]

    Storm: a fast transactional dataplane for remote data structures

    Stanko Novakovic, Yizhou Shan, Aasheesh Kolli, Michael Cui, Yiying Zhang, Haggai Eran, Boris Pis- menny, Liran Liss, Michael Wei, Dan Tsafrir, et al. Storm: a fast transactional dataplane for remote data structures. In SYSTOR, 2019

    2019

  48. [56]

    NVIDIA BLUEFIELD-2 DPU

    Nvidia. NVIDIA BLUEFIELD-2 DPU. https://www.nvidia.com/content/dam/ en-zz/Solutions/Data-Center/documents/ datasheet-nvidia-bluefield-2-dpu.pdf , 2022

    2022

  49. [57]

    DOCA Software Framework

    NVIDIA. DOCA Software Framework. https:// developer.nvidia.com/networking/doca, 2024

    2024

  50. [58]

    Link Aggregation

    Nvidia. Link Aggregation. https: //docs.nvidia.com/networking/display/ bluefielddpubspv422/link+aggregation, 2024

    2024

  51. [59]

    Nvidia. NVIDIA. https://docs.nvidia.com/ doca/sdk/mmap+advise/index.html, 2024

    2024

  52. [60]

    NVIDIA BLUEFIELD-3 DPU

    Nvidia. NVIDIA BLUEFIELD-3 DPU. https://www.nvidia.com/content/dam/ en-zz/Solutions/Data-Center/documents/ datasheet-nvidia-bluefield-3-dpu.pdf , 2024

    2024

  53. [61]

    NVIDIA DevX

    Nvidia. NVIDIA DevX. https://github.com/ Mellanox/devx, 2024

    2024

  54. [62]

    Cache Invalidate Operation

    NVIDIA. Cache Invalidate Operation. https://docs. nvidia.com/doca/sdk/mmap+advise/index. html#src-3543226633_id-.MmapAdvisev2.10. 0-CacheInvalidateOperation, 2025

    2025

  55. [63]

    Shenango: Achieving high{CPU} efficiency for latency-sensitive datacenter workloads

    Amy Ousterhout, Joshua Fried, Jonathan Behrens, Adam Belay, and Hari Balakrishnan. Shenango: Achieving high{CPU} efficiency for latency-sensitive datacenter workloads. In NSDI, 2019

    2019

  56. [64]

    Splitwise: Efficient generative llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bian- chini. Splitwise: Efficient generative llm inference using phase splitting. In ISCA. IEEE, 2024

    2024

  57. [65]

    The design and implementation of open{vSwitch}

    Ben Pfaff, Justin Pettit, Teemu Koponen, Ethan Jack- son, Andy Zhou, Jarno Rajahalme, Jesse Gross, Alex Wang, Joe Stringer, Pravin Shelar, et al. The design and implementation of open{vSwitch}. In NSDI, 2015

    2015

  58. [66]

    {ShRing}: Networking with shared receive rings

    Boris Pismenny, Adam Morrison, and Dan Tsafrir. {ShRing}: Networking with shared receive rings. In OSDI, 2023

    2023

  59. [67]

    Zygos: Achieving low tail latency for microsecond- scale networked tasks

    George Prekas, Marios Kogias, and Edouard Bugnion. Zygos: Achieving low tail latency for microsecond- scale networked tasks. In SOSP, 2017

    2017

  60. [68]

    Alibaba hpn: A data center network for large language model training

    Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan, Binzhang Fu, Xuemei Shi, Fangbo Zhu, Rui Miao, et al. Alibaba hpn: A data center network for large language model training. In SIGCOMM, 2024

    2024

  61. [69]

    Mooncake: Trading more storage for less computation—a{KVCache-centric} architecture for serving{LLM} chatbot

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation—a{KVCache-centric} architecture for serving{LLM} chatbot. In F AST, 2025

    2025

  62. [70]

    Tesla transport protocol over ethernet (ttpoe): A new lossy, exa-scale fabric for the dojo ai supercomputer

    Eric Quinnell. Tesla transport protocol over ethernet (ttpoe): A new lossy, exa-scale fabric for the dojo ai supercomputer. In HCS, 2024

    2024

  63. [71]

    RDMA core userspace libraries and daemons

    RDMA Core. RDMA core userspace libraries and daemons. https://github.com/linux-rdma/ rdma-core, 2024. 15

    2024

  64. [72]

    Scaling up memory disaggregated applications with smart

    Feng Ren, Mingxing Zhang, Kang Chen, Huaxia Xia, Zuoning Chen, and Yongwei Wu. Scaling up memory disaggregated applications with smart. In ASPLOS, 2024

    2024

  65. [73]

    {Ens¯o}: A streaming interface for {NIC-Application} communication

    Hugo Sadok, Nirav Atre, Zhipeng Zhao, Daniel S Berger, James C Hoe, Aurojit Panda, Justine Sherry, and Ren Wang. {Ens¯o}: A streaming interface for {NIC-Application} communication. In OSDI, 2023

    2023

  66. [74]

    Xenic: Smartnic- accelerated distributed transactions

    Henry N Schuh, Weihao Liang, Ming Liu, Jacob Nel- son, and Arvind Krishnamurthy. Xenic: Smartnic- accelerated distributed transactions. In ASPLOS, 2021

    2021

  67. [75]

    A cloud-optimized transport protocol for elastic and scalable hpc

    Leah Shalev, Hani Ayoub, Nafea Bshara, and Erez Sab- bag. A cloud-optimized transport protocol for elastic and scalable hpc. IEEE micro, 2020

    2020

  68. [76]

    IsoStack—Highly efficient network pro- cessing on dedicated cores

    Leah Shalev, Julian Satran, Eran Borovik, and Muli Ben-Yehuda. IsoStack—Highly efficient network pro- cessing on dedicated cores. In ATC, 2010

    2010

  69. [77]

    In NSDI, 2022

    Rajath Shashidhara, Tim Stamler, Antoine Kaufmann, and Simon Peter.{FlexTOE}: Flexible{TCP} offload with{Fine-Grained} parallelism. In NSDI, 2022

    2022

  70. [78]

    Strom: smart remote memory

    David Sidler, Zeke Wang, Monica Chiosa, Amit Kulka- rni, and Gustavo Alonso. Strom: smart remote memory. In EuroSys, 2020

    2020

  71. [79]

    1rma: Re-envisioning remote memory access for multi-tenant datacenters

    Arjun Singhvi, Aditya Akella, Dan Gibson, Thomas F Wenisch, Monica Wong-Chan, Sean Clark, Milo MK Martin, Moray McLaren, Prashant Chandra, Rob Cauble, et al. 1rma: Re-envisioning remote memory access for multi-tenant datacenters. In SIGCOMM, 2020

    2020

  72. [80]

    Demystifying cxl memory with genuine cxl-ready systems and devices

    Yan Sun, Yifan Yuan, Zeduo Yu, Reese Kuper, Chihun Song, Jinghan Huang, Houxiang Ji, Siddharth Agar- wal, Jiaqi Lou, Ipoom Jeong, et al. Demystifying cxl memory with genuine cxl-ready systems and devices. In MICRO, 2023

    2023

  73. [81]

    Scalable rdma transport with efficient connection sharing

    Jian Tang, Xiaoliang Wang, and Huichen Dai. Scalable rdma transport with efficient connection sharing. In INFOCOM. IEEE, 2023

    2023

  74. [82]

    In NSDI, 2018

    Amin Tootoonchian, Aurojit Panda, Chang Lan, Melvin Walls, Katerina Argyraki, Sylvia Ratnasamy, and Scott Shenker.{ResQ}: Enabling{SLOs} in net- work function virtualization. In NSDI, 2018

    2018

  75. [83]

    Lite kernel rdma support for datacenter applications

    Shin-Yeh Tsai and Yiying Zhang. Lite kernel rdma support for datacenter applications. In SOSP, 2017

    2017

  76. [84]

    Understanding the host network

    Midhul Vuppalapati, Saksham Agarwal, Henry Schuh, Baris Kasikci, Arvind Krishnamurthy, and Rachit Agar- wal. Understanding the host network. In SIGCOMM, 2024

    2024

  77. [85]

    Star: Breaking the scalability limit for rdma

    Xizheng Wang, Guo Chen, Xijin Yin, Huichen Dai, Bojie Li, Binzhang Fu, and Kun Tan. Star: Breaking the scalability limit for rdma. In ICNP, 2021

    2021

  78. [86]

    In NSDI, 2023

    Zilong Wang, Layong Luo, Qingsong Ning, Chaoliang Zeng, Wenxue Li, Xinchen Wan, Peng Xie, Tao Feng, Ke Cheng, Xiongfei Geng, et al.{SRNIC}: A scalable architecture for{RDMA}{NICs}. In NSDI, 2023

    2023

  79. [87]

    Characterizing off-path smartnic for accelerating distributed systems

    Xingda Wei, Rongxin Cheng, Yuhan Yang, Rong Chen, and Haibo Chen. Characterizing off-path smartnic for accelerating distributed systems. In OSDI, 2023

    2023

  80. [88]

    Deconstructing{RDMA-enabled} distributed transactions: Hybrid is better! In OSDI, 2018

    Xingda Wei, Zhiyuan Dong, Rong Chen, and Haibo Chen. Deconstructing{RDMA-enabled} distributed transactions: Hybrid is better! In OSDI, 2018

    2018

  81. [89]

    In ATC, 2022

    Xingda Wei, Fangming Lu, Rong Chen, and Haibo Chen.{KRCORE}: A microsecond-scale{RDMA} control plane for elastic computing. In ATC, 2022

    2022

  82. [90]

    Hashing design in modern networks: Challenges and mitigation techniques

    Yunhong Xu, Keqiang He, Rui Wang, Minlan Yu, Nick Duffield, Hassan Wassel, Shidong Zhang, Leon Poutievski, Junlan Zhou, and Amin Vahdat. Hashing design in modern networks: Challenges and mitigation techniques. In ATC, 2022

    2022

  83. [91]

    Don’t forget the i/o when allocating your llc

    Yifan Yuan, Mohammad Alian, Yipeng Wang, Ren Wang, Ilia Kurakin, Charlie Tai, and Nam Sung Kim. Don’t forget the i/o when allocating your llc. In ISCA. IEEE, 2021

    2021

  84. [92]

    Scalable communication endpoints for mpi+ threads applications

    Rohit Zambre, Aparna Chandramowlishwaran, and Pa- van Balaji. Scalable communication endpoints for mpi+ threads applications. In ICPADS. IEEE, 2018

    2018

  85. [93]

    Rpcacc: A high- performance and reconfigurable pcie-attached rpc ac- celerator

    Jie Zhang, Hongjing Huang, Xuzheng Chen, Xiang Li, Ming Liu, and Zeke Wang. Rpcacc: A high- performance and reconfigurable pcie-attached rpc ac- celerator. arXiv preprint arXiv:2411.07632, 2024

    2024 arXiv

  86. [94]

    Smartds: Middle-tier-centric smartnic enabling application-aware message split for disaggregated block storage

    Jie Zhang, Hongjing Huang, Lingjun Zhu, Shu Ma, Dazhong Rong, Yijun Hou, Mo Sun, Chaojie Gu, Peng Cheng, Chao Shi, et al. Smartds: Middle-tier-centric smartnic enabling application-aware message split for disaggregated block storage. In ISCA, 2023

    2023

  87. [95]

    Revisiting congestion control for loss- less ethernet

    Yiran Zhang, Qingkai Meng, Chaolei Hu, and Fengyuan Ren. Revisiting congestion control for loss- less ethernet. In NSDI, 2024

    2024

  88. [96]

    Justitia: Software {Multi-Tenancy} in hardware{Kernel-Bypass} networks

    Yiwen Zhang, Yue Tan, Brent Stephens, and Mosharaf Chowdhury. Justitia: Software {Multi-Tenancy} in hardware{Kernel-Bypass} networks. In NSDI, 2022

    2022

  89. [97]

    {DistServe}: Disaggregating prefill and decoding for 16 goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. {DistServe}: Disaggregating prefill and decoding for 16 goodput-optimized large language model serving. In OSDI, 2024

    2024

  90. [98]

    Deploying user-space {TCP} at cloud scale with{LUNA}

    Lingjun Zhu, Yifan Shen, Erci Xu, Bo Shi, Ting Fu, Shu Ma, Shuguang Chen, Zhongyu Wang, Haonan Wu, Xingyu Liao, et al. Deploying user-space {TCP} at cloud scale with{LUNA}. In ATC, 2023

    2023

  91. [99]

    Congestion control for large-scale rdma de- ployments

    Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Pad- hye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. Congestion control for large-scale rdma de- ployments. SIGCOMM, 2015

    2015

  92. [100]

    One-sided {RDMA-Conscious} ex- tendible hashing for disaggregated memory

    Pengfei Zuo, Jiazhao Sun, Liu Yang, Shuangwu Zhang, and Yu Hua. One-sided {RDMA-Conscious} ex- tendible hashing for disaggregated memory. In ATC, 2021. A Appendix A.1 One-sided Operation Support One-sided operations do not involve any user application logic on the destination,...

    2021

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.