Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that the data-transform steps in AI serving—tokenization, image normalization, KV-cache compression—can be moved onto SmartNIC packet pipelines, cutting CPU utilization by 32–70 percent.

desk verdict Position paper with an original offload framing and three sketches, but the tokenization example has a correctness bug and the headline CPU numbers are unsupported. read the letter →

arxiv 2502.15712 v1 pith:Z3BTQ4UY submitted 2025-01-22 cs.NI cs.AIcs.OS

Mike Wong , Ulysses Butler , Emma Farkash , Praveen Tammana , Anirudh Sivaraman , Ravi Netravali This is my paper · ORCID
classification cs.NIcs.AIcs.OS
keywords SmartNICoffloadingAIinferenceservingdatapreprocessingpacketprocessingpipelinesCPUutilizationreductiontokenizationoffloadcompoundsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This position paper argues that the stack of data-processing work wrapped around AI model calls—tokenization, image normalization, KV-cache compression, prompt rewriting—is a better fit for SmartNIC packet-processing pipelines than for CPUs. SmartNICs are programmable network interface cards whose per-packet pipelines are designed for fast, parallel, rule-based matching; the paper contends that many AI data-processing tasks share exactly those characteristics, being deterministic, bounded-step, and operating on large but simple data structures. If the paper is right, distributed AI serving platforms could offload these tasks onto network hardware already sitting on the data path, cutting median CPU utilization from 86 percent by 32 percent and peak utilization by 70 percent under the paper's TorchServe profiling. A reader should care because CPU contention is a known source of inference latency; turning the network from a delay into a coprocessor would give serving systems a new resource dimension to schedule.

What carries the argument

The mechanism is re-expression of data processing as SmartNIC packet-processing pipelines, where a SmartNIC is a programmable network interface card whose data plane runs fixed-function packet-matching stages. Four techniques carry the argument: (i) localized-window processing—buffer only the neighboring pixels or characters needed for each output, not the full sample; (ii) memoization—replace division, interpolation, and other heavy arithmetic with lookup tables precomputed from constants known a priori; (iii) serialization control—rearrange data layout (tile-major images, overlapping text chunks with de-duplication) so the packet stream aligns with output production; and (iv) two-tier memory—keep hot table entries in fast on-chip BRAM and colder entries in DRAM/HBM. The paper argues these four pieces jointly close the abstraction gap between high-level data-processing specifications and the tight compute, memory, and packet-size constraints of network hardware.

What would settle it

Feed a production tokenizer with a prompt split at 32-byte boundaries and apply the paper's overlap-and-deduplicate correction: if the token sequence differs from the full-context tokenization for any mainstream prompt, the localized-window premise fails for tokenization, and the offload would change model outputs.

Watch

Extended reading notes

Core claim

The central claim is a correspondence: the resource-intensive data-processing functions in modern AI pipelines—transform, filter, and format operations that run before, after, and even inside model inference—have computational characteristics that match the way SmartNICs process packets. Because these functions consist of a bounded number of deterministic steps and touch localized windows of data (neighboring pixels, nearby characters), they can be re-implemented on a packet pipeline without seeing the whole input at once. The paper argues this is concretely achievable by buffering only the localized window, replacing arithmetic with precomputed lookup tables, and changing serialization so the data arrives in a friendly order; it sketches this for image normalization, bilinear interpolation, and tokenization. Its quantitative evidence is a profile of representative multi-model and distributed pipelines on TorchServe showing median and maximum CPU utilization of 86 and 98 percent, and its headline prediction is that offloading data processing to SmartNICs would reduce those figures by 32 and 70 percent respectively. The paper's aim is to establish this offload opportunity as a research direction, not to ship a complete system.

Load-bearing premise

The argument depends on data-processing tasks being deterministic, bounded-step operations on localized windows of data, so that a per-packet SmartNIC implementation can be semantically equivalent to the original full-sample function.

Editorial extensions

If this is right

  • Offloading data-processing tasks to SmartNICs could reduce CPU utilization in AI serving by 32–70 percent, directly attacking the CPU/GPU contention that inflates request latency.
  • Distributed-serving paradigms that move prompts, KV cache, or intermediate tensors across nodes would see the network change from a latency tax into an on-path compute resource.
  • The three example offloads show that bounded buffering and lookup tables keep the pipeline streaming, avoiding the throughput collapse that full-sample buffering would cause.
  • A compiler that maps a data-processing specification to a semantically equivalent SmartNIC implementation would let serving runtimes schedule tasks across CPUs, GPUs, and NICs automatically.
  • If the approach generalizes, the same offload path extends to data cleaning, imputation, feature encoding, and sensor anomaly filtering in real-time monitoring systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 32–70 percent CPU reduction is a projection from profiling, not an end-to-end measurement; a concrete next step the paper does not take is a prototype that offloads one function (e.g., Normalize) to an FPGA SmartNIC and measures actual CPU savings and latency on the same TorchServe harness.
  • The chunk-overlap tokenization design implicitly relies on a bounded maximum token length and locally decidable token boundaries; tokenizers with global merge rules or subword regularization would need a different correctness argument, and testing that boundary is straightforward.
  • If the correspondence holds, the scheduling dimension of AI serving changes: runtimes could co-schedule data-processing stages onto network hardware the way they co-schedule GPUs, making the paper's proposed bottleneck analysis the critical unsolved piece rather than the per-function offload itself.
  • The localization argument also implies a bidirectional research program: data formats could be designed from the start for in-network processing (tile-major, overlap-friendly serialization) rather than re-serializing conventional formats after the fact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper argues that as AI serving pipelines grow more complex and distributed, the network—and specifically SmartNICs—should be viewed as an opportunity rather than solely a source of latency. The authors taxonomize serving tasks, conclude that data-processing operations (e.g., image normalization, tokenization, KV cache compression) are the best candidates for offload because they are deterministic, bounded-step, and operate on localized windows, and discuss four challenges: finite packet size, compute overheads, limited memory, and parallel processing. They propose solutions including data serialization, lookup-table memoization, tiered memory, and pipeline parallelism, and sketch three example offloads: image normalization, bilinear interpolation, and tokenization with an overlap-and-dedup scheme. The paper closes with a roadmap for automatically compiling data-processing specifications to SmartNICs. The central quantitative motivation is a claim that offloading could reduce CPU utilization by 32–70%.

Significance. If the central claims were validated, the paper would make a useful contribution: a clear articulation of why network hardware might help with the non-model parts of AI serving, plus concrete starting points for implementation. The paper is honest about the tension between packet-at-a-time processing and full-sample semantics, and it names specific challenges (BRAM limits, compute mismatches) that later work would need to address. Its strengths are the taxonomy in §2, the challenge/opportunity framing in §3, and the three illustrative offload sketches in §4. However, the quantitative promise is currently unsupported, and the tokenization example—one of the three sketches—is algorithmically incorrect. At present the paper reads as a promising but unvalidated research agenda rather than a demonstrated approach.

major comments (4)
  1. [Section 2, 'Potential benefits'] The central quantitative claim—that offloading data-processing tasks can reduce CPU utilization by 32% and 70%—is asserted without any supporting methodology. The text says the authors profiled 'representative AI pipelines' with TorchServe on a described platform, but it does not list the pipelines, describe the profiler, or explain how the reduction percentages were obtained. Because these numbers are the paper's main motivation, they need to be backed by a reproducible measurement or, at minimum, explicitly presented as illustrative upper bounds rather than measured outcomes.
  2. [Section 4.3] The overlap-and-dedup tokenization scheme is not semantically equivalent to the reference tokenizer. In the paper's own example, the spurious token ' e' generated at the end of the first chunk appears only once and survives into the 'Corrected tokens' list, which also contains extra ' an', ' of', and ' an' compared to the reference. In general, deduplicating tokens that appear twice is unsafe because legitimate repeated tokens in the original text (e.g., 'an an') would be removed. The example therefore fails to demonstrate the claimed equivalence; it must be fixed with a correct algorithm and a proof of equivalence, or removed from the set of motivating offloads.
  3. [Section 4.2] The bilinear interpolation design requires approximately 216 MB of lookup table storage (51M entries × 34 bits), and the proposed BRAM swap-in/out strategy is not evaluated. The paper does not state how many entries can reside in BRAM at once, what the swap traffic would be per image, or whether the off-chip memory latency would negate the benefit of avoiding arithmetic operations. Without such an analysis, the memory-management opportunity of §3.3 remains an unverified hypothesis.
  4. [Sections 2 and 3.2] The class of 'suitable' tasks is defined too loosely for the paper's semantic-equivalence requirement. The claim that KV cache compression is simply 'multiplying their values by a scaling factor' ignores the fact that quantization scales are typically computed from the tensor's min/max, which is a global operation; similarly, imputation (mentioned in §4.4) requires statistics that may not be locally computable in a packet window. The paper needs a precise criterion—e.g., a bound on the state and window size—and must check each motivating example against it, or the offloadability argument does not go through.
minor comments (6)
  1. [Section 2] The phrase 'show to suffer' should be 'shown to suffer'.
  2. [Section 3.3] The phrase 'lookupt tables' should be 'lookup tables'.
  3. [Section 5] The phrase 'within the within resource constraints' contains a duplicated word; remove the second 'within'.
  4. [Section 4.2] The statement that a row for a 1280×720 image 'requires 3840 bytes' should state the pixel format assumption (e.g., 8-bit RGB); the number is not self-evident without this assumption.
  5. [Section 4.3] The bracket notation for overlaps is hard to follow; a table showing chunk boundaries, generated tokens, and the dedup step would improve readability even after the correctness issue is fixed.
  6. [Sections 4.1 and 4.2] The equations are rendered with Unicode artifacts (e.g., '/u1D450' and '/u1D462'); please use a proper math format.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: the paper is a position and research-agenda paper whose CPU reduction figures are asserted profiling estimates, not outputs of its own equations, and whose proposed offloads are forward-looking designs rather than fitted predictions.

full rationale

This paper makes no fitted-parameter prediction and contains no derivation chain whose conclusion is equivalent to its premises. The central claim is that certain AI-pipeline data-processing tasks match SmartNIC packet-processing characteristics; this is argued by taxonomy and example designs, not by a formal model. The 32% and 70% CPU-utilization reductions in Section 2 are presented as profiling observations ('we profiled resource usage... we observe median and maximum CPU utilization values of 86% and 98%... these values could be drastically reduced by 32% and 70%'), but they are asserted estimates rather than results derived from the paper's own equations, so they cannot reduce to the paper's inputs by construction. The Section 4.3 tokenization overlap-and-dedup sketch may be criticized on correctness grounds, because the shown 'Corrected tokens' appear to contain spurious tokens that survive deduplication; however, an incorrect or semantically inequivalent offload is a soundness flaw, not circularity, since the paper does not define equivalence in terms of the offload's own output. The only self-citation, reference [42] (RAGServe), shares an author but is used as an example of compound-AI serving, not as a load-bearing premise or uniqueness theorem; it does not support the paper's main claim. No ansatz is smuggled in via citation, and no known result is renamed. Accordingly, no enumerable circular step meets the quoting-and-reduction bar, and the honest finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central argument rests on the assumption that AI data processing tasks have deterministic, bounded, localized structure that can be expressed in packet-processing pipelines. The profiling numbers, lookup-table sizes, and serialization schemes are all context assumptions rather than measured results.

free parameters (1)
  • CPU utilization reduction from offloading = 32% and 70% (stated)
    Presented as expected reductions in CPU utilization if data processing tasks are offloaded, but no measurement or derivation is provided in the paper.
assumptions (4)
  • domain assumption SmartNIC packet processing can implement data processing functions with semantic equivalence
    This is the core premise of the paper; Section 3 discusses challenges but no implementation demonstrates it.
  • domain assumption AI pipeline data processing tasks consist of bounded deterministic steps with localized data access
    Stated in Section 2 and used to justify offload suitability.
  • domain assumption Lookup tables can replace arithmetic operations within SmartNIC resource constraints
    Proposed in Sections 3.2 and 4; feasibility depends on table sizes and memory hierarchy.
  • domain assumption The profiling setup is representative of emerging AI pipelines
    Profiling in Section 2 uses TorchServe on one hardware configuration; representativeness is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines." pith.science (2026). https://pith.science/paper/Z3BTQ4UY

@misc{pith2026250215712,
  author       = {Pith},
  title        = {Pith review of: GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z3BTQ4UY}},
  note         = {Machine review of arXiv:2502.15712}
}
read the original abstract

The increasing prominence of AI necessitates the deployment of inference platforms for efficient and effective management of AI pipelines and compute resources. As these pipelines grow in complexity, the demand for distributed serving rises and introduces much-dreaded network delays. In this paper, we investigate how the network can instead be a boon to the excessively high resource overheads of AI pipelines. To alleviate these overheads, we discuss how resource-intensive data processing tasks -- a key facet of growing AI pipeline complexity -- are well-matched for the computational characteristics of packet processing pipelines and how they can be offloaded onto SmartNICs. We explore the challenges and opportunities of offloading, and propose a research agenda for integrating network hardware into AI pipelines, unlocking new opportunities for optimization.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SNI-GNN: SmartNIC-Assisted Full-Graph GNN Training with In-Network Embedding Prediction

    cs.LG 2026-08 conditional novelty 6.0 of 10

    In-network linear-trend prediction on SmartNICs reduces communication in multi-server full-graph GNN training by 21 to 45 percent with under 1 percent accuracy loss.

Reference graph

Works this paper leans on

69 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    [n. d.]. Can Scheduling Overhead Dominate LLM In- ference Performance? A Study of CPU Scheduling Overhead on Two Popular LLM Inference Systems. https://mlsys.wuklab.io/posts/scheduling_overhead/, Ret rieved on 2025-01

    work page 2025

  2. [2]

    [n. d.]. Swarm. https://github.com/openai/swarm/tree/ma in/, Re- trieved on 2025-01

    work page 2025

  3. [3]

    [n. d.]. The Evolution of Drone Technology: Embracing AI and Computer Vision in 2024. https://visionplatform.ai/computer-vision-for-drones-and-uav -in-2024, Retrieved on 2024-06

    work page 2024

  4. [4]

    [n. d.]. YouTokenToMe Text Tokenizer. https://github.com/vkcom/youtokentome/, Retrieved on 2025 -01

    work page 2025

  5. [5]

    NVIDIA Data Loading Library (DALI)

    2024. NVIDIA Data Loading Library (DALI). https://github.com/NVIDIA/DALI

    work page 2024

  6. [6]

    Alexander Isenko, Ruben Mayer, Jeffrey Jedele, Hans-Arno Jac obsen . 2022. Where is my training bottleneck? hidden trade-offs in deep learning preprocessing pipelines. In SIGMOD

    work page 2022

  7. [7]

    Jayashree Mohan, Amar Phanishayee, Ashish Raniwala, Vijay Chi- dambaram . 2021. Analyzing and mitigating data stalls in dnn train- ing. In VLDB

    work page 2021

  8. [8]

    Shubham Agarwal, Subrata Mitra, Sarthak Chakraborty, Sr ikr- ishna Karanam, Koyel Mukherjee, and Shiv Saini. 2023. Ap- proximate Caching for Efficiently Serving Diffusion Models. arXiv:2312.04429 [cs.CV] https://arxiv.org/abs/2312.044 29

    arXiv 2023

Show all 69 references
  1. [9]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Mil- lican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Mon- teiro, Jacob Menick, Sebast...

  2. [10]

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nip un Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee

  3. [11]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, J ack- son Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirho- seini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christo- pher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, ...

    2022 arXiv

  4. [12]

    Bret Hull, Vladimir Bychkovsky, Yang Zhang, Kevin Chen, Michel Goraczko, Allen Miu, Eugene Shih, Hari Balakrishnan, Samuel Mad- den. 2006. CarTel: A Distributed Mobile Sensor Computing System. In SenSys

    2006

  5. [13]

    David Cassel. [n. d.]. Are We Ready for AI-Powered Security Cameras? https://thenewstack.io/are-we-ready-for-ai-powered-sec urity-cameras/

  6. [14]

    Ayan Chatterjee and Bestoun S. Ahmed. 2022. IoT anomaly de tec- tion methods and applications: A survey. Journal of King Saud Uni- versity - Computer and Information Sciences 34, 8 (2022), 4547–4563. https://doi.org/10.1016/j.jksuci.2022.01.001

    2022 doi

  7. [15]

    Franklin, Joseph E

    Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: A Low- Latency Online Prediction Serving System. arXiv:1612.03079 [cs.DC] https://arxiv.org/abs/1612.03079

    2017 arXiv

  8. [16]

    Dan Graur, Damien Aymon, Dan Kluser, Tanguy Albrici, Chandram o- han A Thekkath, and Ana Klimovic . 2022. Cachew: Machine learning input data processing as a service. . In USENIX ATC

    2022

  9. [17]

    Murray, Jiří Šimša, Ana Klimovic, and Ihor Indyk

    Derek G. Murray, Jiří Šimša, Ana Klimovic, and Ihor Indyk. 2021. tf.data: A Machine Learning Data Processing Framework. In Proc. VLDB Endow. 14, 12

    2021

  10. [18]

    Yihong Dong, Kangcheng Luo, Xue Jiang, Zhi Jin, and Ge Li. 2024. PACE: Improving Prompt with Actor-Critic Edit- ing for Large Language Model. arXiv:2308.10088 [cs.CL] https://arxiv.org/abs/2308.10088

    2024 arXiv

  11. [19]

    Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang. 2024. Efficiently Serving LLM Reasoning Programs with Certaindex. arXiv:2412.20993 [cs.LG] https://arxiv.org/abs/2412.20993

    2024 arXiv

  12. [20]

    Yi Gao, Xue Li, Xinyu Chen, Xian Liu, Jun Song, Liang Zeng, Shaojie Zhang, Haoyu Chen, Wei Huang, and Zhiqiang Wu. 2024. Efficient Memory Management for Large Language Model Serving with Page- dAttention. ACM Transactions on Computing Systems (TOCS) 42, 3 (2024), 1–25. https://d...

    2024

  13. [21]

    Gerbasi, G

    A. Gerbasi, G. Clementi, F. Corsi, and et al. 2023. Deep- MiCa: Automatic segmentation and classification of breast MIcroCAlcifications from mammograms. Computational Methods and Programs in Biomedicine 235 (2023), 107483. https://doi.org/10.1016/j.cmpb.2023.107483

    2023

  14. [22]

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra

  15. [23]

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like Clockwork: Performance Predictability from the Bot- tom Up. In 14th USENIX Symposium on Operating Systems De- sign and Implementation (OSDI 20) . U...

    2020

  16. [24]

    Gyeong-In Yu,Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models

    2022

  17. [25]

    Hanyu Zhao, Zhi Yang, Yu Cheng, Chao Tian, Shiru Ren, Wencong Xiao, Man Yuan, Langshi Chen, Kaibo Liu, Yang Zhang, Yong Li, and Wei Lin. 2023. Goldminer: Elastic scaling of training data pre- processing pipelines for deep learning. In Proc. ACM Manag. Data, 1(2)

    2023

  18. [26]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun

  19. [27]

    Hutchings, R

    B.L. Hutchings, R. Franklin, and D. Carver. 2002. Assisting networ k in- trusion detection with reconfigurable hardware. In Proceedings. 10th Annual IEEE Symposium on Field-Programmable Custom Comput ing Machines. 111–120. https://doi.org/10.1109/FPGA.2002.1106666

    2002 arXiv

  20. [28]

    Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. 2024. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation. arXiv:2404.12457 [cs.DC] https://arxiv.org/abs/2404.12457

    2024 arXiv

  21. [29]

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hill ier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. The Kinetics Human Action Video Dataset. arXiv:1705.06950 [cs.CV] https://arxiv.or...

    2017 arXiv

  22. [30]

    Anton Korinek. 2023. The Role of Large Lan- guage Models in Automating Economic Research. https://www.nber.org/system/files/working_papers/w30957/w30957.pdf Working Paper

    2023

  23. [31]

    ChonLam Lao, Yanfang Le, Kshiteej Mahajan, Yixi Chen, Wenfei Wu, Aditya Akella, and Michael Swift. 2021. ATP: In-network Aggregation for Multi-tenant Learning. In 18th USENIX Symposium on Networked Systems Design and Im- plementation (NSDI 21) . USENIX Association, 741–761. ht...

    2021

  24. [32]

    Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming-Yu Liu, Kai Li, and Song Han. 2024. Distri- Fusion: Distributed Parallel Inference for High-Resolution Diffusion Models. arXiv:2402.19481 [cs.CV] https://arxiv.org/abs/2 402.19481

    2024 arXiv

  25. [33]

    Gonzalez, and Ion Stoica

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multi- plexing with Model Parallelism for Deep Learning Serving. In 17th USENIX Symposium on Oper...

    2023

  26. [34]

    Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: Efficient Serving of LLM- based Applications with Semantic Variable. arXiv:2405.19888 [cs.L G] https://arxiv.org/abs/2405.19888

    2024 arXiv

  27. [35]

    Haotian Liu, Chunyuan Lin, Qingyang Li, Zhewei Hu, Yong Wang, Cho-Jui Hsieh, and Jason Lee. 2023. LLaV A: Large Language and Vi- sion Assistant. arXiv preprint arXiv:2304.08485 (2023)

    2023 arXiv

  28. [36]

    Shuo Liu, Qiaoling Wang, Junyi Zhang, Wenfei Wu, Qinliang Lin, Yao Liu, Meng Xu, Marco Canini, Ray C. C. Cheung, and Jianfei He

  29. [37]

    Rajalakshmi

    Naga Praveen Babu Mannam, Basa Sidvik, and P. Rajalakshmi

  30. [38]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Sha h, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Split- wise: Efficient generative LLM inference using phase splitting. arXiv:2311.18677 [cs.AR] https://arxiv.org/abs/2311.186 77

    2024 arXiv

  31. [39]

    PyTorch. 2023. TorchServe. https://pytorch.org/ser ve/

    2023

  32. [40]

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yong- wei Wu, Weimin Zheng, and Xinran Xu. 2024. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. arXiv:2407.00079 [cs.DC] https://arxiv.org/abs/2407.000 79

    2024 arXiv

  33. [42]

    Siddhant Ray, Rui Pan, Zhuohan Gu, Kuntai Du, Ganesh Anantha- narayanan, Ravi Netravali, and Junchen Jiang. 2024. RAGServe: Fast Quality-Aware RAG Systems with Configuration Adaptation. arXiv:2412.10543 [cs.LG] https://arxiv.org/abs/2412.105 43

    2024

  34. [43]

    Fredy Reusser. 2024. Tabular Learning: Encoding for En- tity and Context Embeddings. arXiv:2403.19405 [cs.LG] https://arxiv.org/abs/2403.19405

    2024 arXiv

  35. [44]

    L.G. Roberts. 1978. The evolution of packet switching. Proc. IEEE 66, 11 (1978), 1307–1313. https://doi.org/10.1109/PROC.1978 .11141

    1978 doi

  36. [45]

    Romil Bhardwaj, Zhengxu Xia, Ganesh Ananthanarayanan, Junchen Jiang, Yuanchao Shu, Nikolaos Karianakis, Kevin Hsieh, Paramvir Bahl, Ion Stoica. 2022. Ekya: Continuous Learning of Video Analytics Models on Edge Compute Servers. In USENIX NSDI

    2022

  37. [46]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjee v Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. Interna- tional Journal of Computer Vi...

    2015 doi

  38. [47]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, G abriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021 . Learning Transferable Visual Models From Natural Language Super- vision. arXiv prep...

    2021 arXiv

  39. [48]

    Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. 2019. Nexus: A GPU Cluster Engine for Accelerating DNN-Based Video Analysis. In Proceedings of the 27th ACM Symposium on Operating 7 Systems Principles ...

    2019

  40. [49]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeG resley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053 [cs.CL] https://arxiv.org/abs/1909.080 53

    2020 arXiv

  41. [50]

    Karen Simonyan and Andrew Zisserman. 2014. Two-Stream Convolutional Networks for Action Recognition in Videos. arXiv:1406.2199 [cs.CV] https://arxiv.org/abs/1406.2199

    2014 arXiv

  42. [51]

    Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. https://doi.org/10.48550/ARXIV.1409.1556

    2014 doi arXiv

  43. [52]

    Giuseppe Siracusano, Salvator Galea, Davide Sanvito, Moham- mad Malekzadeh, Gianni Antichi, Paolo Costa, Hamed Had- dadi, and Roberto Bifulco. 2022. Re-architecting Traffic Anal- ysis with Neural Network Interface Cards. In 19th USENIX Symposium on Networked Systems Design and I...

    2022

  44. [53]

    Davide Sanvito, Giuseppe Siracusano, and Roberto Bifulco. 2018 . Can the Network be the AI Accelerator?. In Proceedings of the 2018 Morn- ing Workshop on In-Network Computing (Budapest, Hungary) (Net- Compute ’18). Association for Computing Machinery, New York, NY, USA, 20–25....

    2018

  45. [54]

    Tushar Swamy, Annus Zulfiqar, Luigi Nardi, Muhammad Shah- baz, and Kunle Olukotun. 2023. Homunculus: Auto-Generating Efficient Data-Plane ML Pipelines for Datacenter Networks. In Proceedings of the 28th ACM International Conference on Arc hi- tectural Support for Programming Lang...

    2023

  46. [55]

    Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, Yiying Zhang. 2024. Preble: Efficient Distributed Prompt Scheduling for LLM Serving. arXiv:2407.00023 [cs.CV] https://arxiv.org/abs/24 07.00023

    2024 arXiv

  47. [56]

    Viyom Mittal, Shixiong Qi, Ratnadeep Bhattacharya, Xiaosu Lyu, Jun- feng Li, Sameer G Kulkarni, Dan Li, Jinho Hwang, KK Ramakrishnan, and Timothy Wood. 2021. Mu: an efficient, fair and responsive server- less framework for resource constrained edge clouds. In SoCC

    2021

  48. [57]

    Keith Wang, Zhiyu Liu, Xiang Zhao, Haoyu Ma, Kaixi- ang Chen, and Zhiqiang Zhang. 2024. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving. In Proceedings of SIGCOMM 2024 . 1571–1586. https://cs.stanford.edu/~keithw/sigcomm2024/sigcomm24-final157...

    2024

  49. [58]

    Zhaoqi Xiong and Noa Zilberman. 2019. Do Switches Dream of Ma- chine Learning? Toward In-Network Classification. In Proceedings of the 18th ACM Workshop on Hot Topics in Networks (Princeton, NJ, USA) (HotNets ’19). Association for Computing Machinery, New York, NY, USA, 25–33. ...

    2019

  50. [59]

    Tushar Swamy, Alexander Rucker, Muhammad Shahbaz, Isha n Gaur, and Kunle Olukotun. 2022. Taurus: a data plane architecture for per-packet ML. In Proceedings of the 27th ACM International Con- ference on Architectural Support for Programming Language s and Operating Systems (La...

    2022

  51. [60]

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2024. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. arXiv:2405.16444 [cs.LG] https://arxiv.org/abs/2405.16444

    2024 arXiv

  52. [61]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin,Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serv- ing

    2024

  53. [62]

    Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Sto- ica. 2023. SHEPHERD: Serving DNNs in the Wild. In 20th USENIX Symposium on Networked Systems Design and Imple- mentation (NSDI 23) . USENIX Association, Boston, MA, 787–808. https://www.usenix.org/conference/nsdi23/prese...

    2023

  54. [63]

    Zhihao Zhang, Alan Zhu, Lijie Yang, Yihua Xu, Lanting Li, Phitchaya Mangpo Phothilimthana, and Zhihao Jia. 2024. Accelerat- ing Retrieval-Augmented Language Model Serving with Speculation. arXiv:2401.14021 [cs.LG] https://arxiv.org/abs/2401.140 21

    2024 arXiv

  55. [64]

    Youran Zhou and Sunil Aryal. 2024. A Comprehen- sive Review of Handling Missing Data: Exploring Special Missing Mechanisms. arXiv preprint arXiv:2404.04905v1 (2024). https://arxiv.org/abs/2404.04905v1 Contact: echo.zhou@deakin.edu.au, sunil.aryal@deakin.edu.au. 8

    2024 arXiv

  56. [65]

    Hui Yang, Sifu Yue, and Yunzhong He. 2023. Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions. arXiv:2306.02224 [cs.AI] https://arxiv.org/abs/2306.022 24

    2023 arXiv

  57. [2015]

    https://doi.org/10.48550/ARXIV.1512.03385

    Deep Residual Learning for Image Recognition. https://doi.org/10.48550/ARXIV.1512.03385

    doi arXiv

  58. [2021]

    In 2021 International Conference on Smart Generation Computing, Communication and Networking (SMART GENCON)

    Powering Prediction of an Autonomous Campus Shuttle using CFD. In 2021 International Conference on Smart Generation Computing, Communication and Networking (SMART GENCON) . 1–6. https://doi.org/10.1109/SMARTGENCON51891.2021.9645802

    2021

  59. [2022]

    arXiv:2204.14198 [cs.CV] https://arxiv.org/abs/2204.141 98

    Flamingo: a Visual Language Model for Few-Shot Learning. arXiv:2204.14198 [cs.CV] https://arxiv.org/abs/2204.141 98

    arXiv

  60. [2023]

    arXiv:2305.05665 [cs.CV] https://arxiv.org/abs/2305.056 65

    ImageBind: One Embedding Space To Bind Them All. arXiv:2305.05665 [cs.CV] https://arxiv.org/abs/2305.056 65

    arXiv

  61. [2024]

    Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.