Pith. sign in

REVIEW 3 major objections 7 minor 3 cited by

Efficient and Scalable Agentic AI with Heterogeneous Systems

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that agentic AI workloads can be run cheaper on mixed, multi-vendor hardware than on uniform top-of-the-line GPU clusters, by scheduling each step of the agent's computation graph onto the device that fits it best.

desk verdict Well-argued systems proposal, honestly labeled preliminary, but the headline TCO result is not auditable without the underlying performance model and device counts, and the convexity claim needs a fix. read the letter →

arxiv 2507.19635 v1 pith:IUVM4I5V submitted 2025-07-25 cs.LG cs.AIcs.DC

Zain Asgar , Michelle Nguyen , Sachin Katti This is my paper
classification cs.LGcs.AIcs.DC MSC 68M2090C25
keywords agenticAIheterogeneoushardwareprefill-decodedisaggregationtotalcostofownershipMLIRconvexoptimizationLLMinferenceSLA-awarescheduling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that AI agents—workloads that mix speech-to-text, LLM inference, web search, planning, and tool calls—are being served on expensive, uniform GPU clusters, even though the individual steps have very different hardware needs. It proposes a system that represents an agent as a fine-grained dataflow graph, assigns each node a cost, and solves a convex optimization problem to place every task on the cheapest device that still meets an end-to-end latency SLA. The central claim, supported by preliminary simulations, is that disaggregating LLM prefill and decode across vendors—for example, prefill on an NVIDIA H100 and decode on an Intel Gaudi 3—can match or beat the total cost of ownership of a homogeneous cluster of the newest NVIDIA B200 GPUs, while remaining within interactive latency bounds. If true, this would let operators stretch the useful life of older GPU fleets and mix in cheaper accelerators instead of buying the newest generation everywhere.

What carries the argument

The load-bearing piece is the cost-aware optimization over a dataflow graph: the system represents each agent as a directed graph of tasks, estimates the resource demand of each task on each hardware class via profiled or roofline performance numbers, and solves a convex program that minimizes total execution cost subject to latency, throughput, and per-device capacity constraints. A worked prefill/decode example shows how disaggregation can satisfy a 120 ms SLA at lower cost than a homogeneous high-performance assignment, and the simulation uses the same formulation, extended to tensor and pipeline parallelism, to produce the TCO comparisons.

What would settle it

Running the same LLaMA 3-70B/FP8 workload under the stated SLAs on an H100::Gaudi 3 pair and a B200::B200 pair, with actual measured latency, throughput, power draw, and amortized hardware cost, and comparing the resulting cost per million tokens would directly confirm or overturn the paper's headline claim.

Watch

Extended reading notes

Core claim

The paper's central discovery is that agentic AI workloads decompose naturally into granular tasks with sharply different resource profiles, and that a heterogeneous hardware fleet can exploit this asymmetry: prefill is compute-bound and benefits from high TFLOPS, decode is memory-bandwidth-bound, tool calls are network-bound, and data processing runs fine on CPUs. The reported simulation results, generated by a performance model calibrated to measurements, show that for LLaMA 3 configurations the pairing B200::Gaudi 3 (prefill on B200, decode on Gaudi 3) gives the best TCO benefit, and that H100::Gaudi 3 is often comparable to or slightly better than B200::B200 under both interactive and throughput SLAs. The paper reads this as evidence that the latest homogeneous GPU generation is not required to meet agent-serving SLAs, and that incorporating already-amortized older GPUs would push the advantage further.

Load-bearing premise

The claimed TCO comparisons rest on the simulator's device latency and throughput numbers, which are described only as empirical measurements fitted to a roofline model, with no measurements, model parameters, or sensitivity analysis shown in the paper.

Editorial extensions

If this is right

  • Operators can extend the life of deployed H100 infrastructure by pairing it with lower-cost decode accelerators, delaying or avoiding a full upgrade to B200-class hardware.
  • Agent-serving infrastructure can be assembled from mixed-vendor fleets without sacrificing interactive latency SLAs, reducing dependence on a single accelerator vendor.
  • The same optimization framework generalizes beyond LLM inference to mixed graphs of tool calls, retrieval, and CPU-side processing, so the TCO benefits should compound as agents become more heterogeneous.
  • A system built on this design would provide a vendor-neutral compilation and orchestration layer, making multi-vendor hardware usable through a common IR and scheduling interface.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The TCO advantage of H100::Gaudi 3 over B200::B200 is, in the paper, computed only in a performance model; the real-world claim would be substantially stronger if reproduced on physical hardware at production batch sizes.
  • A testable prediction follows: on interactive voice-agent traffic, a mixed cluster should sustain the same SLA at higher utilization and lower cost per request than a homogeneous B200 cluster, which is a directly measurable claim for an operator.
  • The framework treats cost per resource and SLA as static inputs; a natural extension, which the paper does not address, is a controller that re-optimizes placement online as prices, utilization, or traffic mix drift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes a system architecture for executing agentic AI workloads on heterogeneous compute infrastructure. It introduces a graph-based workload representation, an MLIR-based compilation and orchestration stack, and a cost-aware optimization framework that assigns tasks to hardware classes under latency, throughput, and capacity constraints. The authors report preliminary TCO results from a simulated optimizer for disaggregated LLM prefill/decode, claiming that mixed configurations such as H100::Gaudi3 can match or slightly beat a homogeneous B200::B200 deployment on total cost of ownership while meeting interactive latency SLAs. The paper also includes a worked example, hardware cost-efficiency analysis, and a qualitative system design description.

Significance. The problem is timely and practically important: agentic workloads are growing, and infrastructure cost is a major deployment barrier. The paper's decomposition of agent workflows into hardware-sensitive granular tasks, its use of prefill/decode disaggregation across vendors, and its cost-per-resource analysis are useful contributions. The framework is clearly described, and the worked example in Section 3.1.2 nicely illustrates the intended optimization. If the headline TCO finding were backed by disclosed measurements and a reproducible performance model, the result would be significant for infrastructure planning. As submitted, however, the quantitative claim rests on an undisclosed model and is explicitly labeled preliminary, so the paper currently reads as an architecture proposal with an illustrative but not yet validated numerical result.

major comments (3)
  1. [Section 3.1.2, Constraint 3] The text claims the formulation is convex and yields a globally optimal plan, but Constraint 3, sum_i 1/t_i >= R, with t_i linear in the decision variables x_ij, is a superlevel constraint of a convex function and is not generally a convex constraint. This means the global optimality claim is unsupported for any instance in which the throughput SLA is active. Please either remove the global optimality claim, impose conditions under which the constraint set is convex, or reformulate the throughput constraint (for example, by introducing auxiliary variables and a suitable convex relaxation).
  2. [Section 5, Figures 8 and 9] The central TCO ordering, especially the H100::Gaudi3 versus B200::B200 comparison, depends entirely on device latency and throughput values that are described only as "empirical measurements when available" augmented by "theoretical roofline modeling" and "fit to real measurements." No measurements, fitted parameters, calibration procedure, or sensitivity analysis are presented. A first-order feasibility check using the paper's own Table 5 shows the issue is load-bearing: for LLaMA-3-70B-FP16, decoding one token requires reading approximately 140 GB of weights, and a single Gaudi3 at 3.7 TB/s has a memory-bandwidth lower bound of about 38 ms per token, above the 20 ms TBT SLA. The paper does not report whether the result uses FP8, multiple Gaudi3 devices via tensor parallelism, reduced batch sizes, or some other mechanism. Without those details and the underlying model, the headline finding is not auditable.
  3. [Section 5 and Section 5.1] The paper acknowledges that "These findings are preliminary, and comprehensive system validation is currently underway," yet the abstract and Section 5.1 present the H100::Gaudi3 result as a "surprising finding" and a reason to extend the life of deployed infrastructure. Given that the result is produced by an optimizer over an undisclosed performance model, the current text overstates the support for the claim. The paper should either include the full performance model and its validation or clearly reposition the TCO numbers as an illustrative modeling exercise rather than an empirical result.
minor comments (7)
  1. [Section 3.1.1] The bullet explaining the max term says "latency for the slowest task in the graph," but the expression is a max over resources r for a single task; it should say "latency determined by the slowest resource for task i on hardware j."
  2. [Section 5.2, Equations (1) and (2)] The subscripts contain typos: "NP ref illGP U" and "NDeocodeGP U" should be, respectively, the number of prefill GPUs and the number of decode GPUs.
  3. [Section 4.2, final paragraph] There is a typo, "This optmized graph," which should be "This optimized graph."
  4. [Section 5, Figure 8 caption] The "::" notation for prefill::decode device pairs is used in Figure 8 before it is defined in Section 5.1; please define the notation before its first use.
  5. [Table 5] The cost values are described as "averaged across a representative sample of hardware resellers" but no source, range, or sample size is given; providing the underlying data or a citation would improve reproducibility.
  6. [Section 5.3] The text says Gaudi3 was selected for decode in decode-intensive scenarios because of its "lowest marginal cost, as indicated in Figure 4," but Figure 4 shows cost per unit of memory bandwidth, not cost per decoded token; the connection between the two should be made explicit.
  7. [Section 7.3] The reference to "NVIDIA Lepton" appears to be a conflation of NVIDIA and Lepton AI; please verify and correct the name and add an appropriate citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the TCO ranking is an optimizer output, not a fitted prediction; undisclosed performance data is an audit gap, not a circularity.

full rationale

The central quantitative claim (that H100::Gaudi3 can match or slightly beat B200::B200 on TCO) is the output of the convex optimization framework in Section 3.1, evaluated with the device cost and capability inputs in Table 5 and Figure 4. The optimization objective is defined independently of the headline result: it minimizes a cost sum over task-device assignments subject to SLA and capacity constraints. Nothing in the formulation defines the conclusion in terms of the inputs, and no fitted parameter is renamed as a prediction. The Section 5.3 observation that Gaudi3 is selected for decode because it has the lowest marginal memory-bandwidth cost is a direct consequence of the cost model input, which makes the result unsurprising rather than circular; SLA feasibility is a separate non-tautological constraint that the optimizer must verify. There are no load-bearing self-citations, no imported uniqueness theorems, and no known result renamed as a new one. The paper is transparent that results are preliminary and based on a performance model fit to real measurements, and it does disclose that the model incorporates empirical measurements and roofline estimates. The absence of detailed measured TTFT/TBT values, batch sizes, and parallelism choices is a genuine auditability and verification weakness, not a circularity: those missing numbers could invalidate the claim, but they do not show that the claim reduces to its own assumptions by construction. Under the required standard of exhibiting a specific reduction, no circular step is present, so the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical entities are postulated. The proposed MLIR dialect and orchestrator are software designs, not invented entities in the sense of new physics or measurements. The central claim rests instead on the cost and performance assumptions listed above.

free parameters (5)
  • Device cost-per-resource values c_j^(r) = Not disclosed; derived from public hardware listings (Figure 4, Table 5)
    The TCO optimizer's objective uses these unit costs, and the conclusion that Gaudi 3 is cheap for memory bandwidth follows directly from them.
  • Device latency and throughput performance model = Not disclosed; described as empirical measurements augmented by roofline modeling
    The central TCO comparison is an output of this model, and its parameters are not given.
  • SLA thresholds = TTFT 250 ms, TBT 20 ms
    Chosen in Section 5 for interactive and offline scenarios; different thresholds would change which configurations meet SLA.
  • Amortization and utility cost assumptions = 4-year amortization, 8% interest, $0.40/kWh, max TDP
    These convert hardware prices into operating cost and directly determine the TCO comparison.
  • Sequence length scenarios = input=512/output=4096 and input=4096/output=512
    The reported TCO benefits in Figures 8 and 9 are specific to these two workload shapes.
assumptions (5)
  • domain assumption The optimization problem in Section 3.1 is convex and admits a globally optimal solution
    The authors assert convexity, but the throughput constraint involves 1/t_i where t_i is linear in x, which is not generally convex; global optimality is therefore unsupported.
  • domain assumption The unshown performance model accurately captures real H100, B200, Gaudi 3, MI300x, A100, and A40 behavior
    Section 5 says metrics 'incorporate empirical measurements when available and are augmented by theoretical roofline modeling', but no measurements are shown; the TCO result depends on this.
  • domain assumption Unconstrained hardware availability in the simulated workload
    Section 5 simulates a continuous workload with unconstrained hardware availability, which removes fleet-sizing and contention effects that could reduce heterogeneity benefits.
  • domain assumption Static analysis passes on MLIR can extract accurate resource usage vectors theta and latency terms
    Section 4.2 claims MLIR passes for cost estimation feed the convex optimizer, but this is a design statement with no implementation or measured accuracy.
  • domain assumption Operating cost model excludes datacenter, colocation, and NRE costs and assumes each node runs at max TDP
    Section 5.1 states these exclusions, so the TCO comparison is partial.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient and Scalable Agentic AI with Heterogeneous Systems." pith.science (2026). https://pith.science/paper/IUVM4I5V

@misc{pith2026250719635,
  author       = {Pith},
  title        = {Pith review of: Efficient and Scalable Agentic AI with Heterogeneous Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IUVM4I5V}},
  note         = {Machine review of arXiv:2507.19635}
}
read the original abstract

AI agents are emerging as a dominant workload in a wide range of applications, promising to be the vehicle that delivers the promised benefits of AI to enterprises and consumers. Unlike conventional software or static inference, agentic workloads are dynamic and structurally complex. Often these agents are directed graphs of compute and IO operations that span multi-modal data input and conversion), data processing and context gathering (e.g vector DB lookups), multiple LLM inferences, tool calls, etc. To scale AI agent usage, we need efficient and scalable deployment and agent-serving infrastructure. To tackle this challenge, in this paper, we present a system design for dynamic orchestration of AI agent workloads on heterogeneous compute infrastructure spanning CPUs and accelerators, both from different vendors and across different performance tiers within a single vendor. The system delivers several building blocks: a framework for planning and optimizing agentic AI execution graphs using cost models that account for compute, memory, and bandwidth constraints of different HW; a MLIR based representation and compilation system that can decompose AI agent execution graphs into granular operators and generate code for different HW options; and a dynamic orchestration system that can place the granular components across a heterogeneous compute infrastructure and stitch them together while meeting an end-to-end SLA. Our design performs a systems level TCO optimization and preliminary results show that leveraging a heterogeneous infrastructure can deliver significant TCO benefits. A preliminary surprising finding is that for some workloads a heterogeneous combination of older generation GPUs with newer accelerators can deliver similar TCO as the latest generation homogenous GPU infrastructure design, potentially extending the life of deployed infrastructure.

Figures

Figures reproduced from arXiv: 2507.19635 by the authors.

Figure 1
Figure 1. Comparison of agentic architectural patterns, inspired by LangGraph’s taxonomy [ [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Directed graph for a conversational voice agent [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Radar plots comparing system resource demands across various AI workloads. Each subplot visualizes [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Marginal cost-efficiency analysis of contemporary AI accelerators, derived from publicly available hardware [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: High-level orchestration and serving system architecture [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: System design stack from MLIR representation through task planning and compilation to deployment [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Transformation of a LangChain-style agent program into progressively lower-level MLIR representations. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: TCO Benefit for Heterogeneous Configs (input=512, output=4096). Comparison of cost efficiency across different Llama 3 models and device pairings. Dashed line at 1.0 indicates baseline TCO for H100::H100. Bars show top configurations that meet SLA constraints: Latency …
Figure 9
Figure 9. Figure 9: TCO Benefit for Heterogeneous Configs (input=4096, output=512). Comparison of cost efficiency across different Llama 3 models and device pairings. Dashed line at 1.0 indicates baseline TCO for H100::H100. Bars reflect top-performing configurations that satisfy SLA cons…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Agentic Orchestration of HPC Applications in Cloud

    cs.DC 2026-07 conditional novelty 6.5 of 10

    An agentic team using Gemini successfully builds multi-platform containers, deploys and optimizes four HPC applications across 21 AWS instance types in Kubernetes, and produces expert-approved linear scaling studies.

  2. Architectural Implications of Agentic AI Workflows

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Agentic AI workloads fragment execution into bursts of LLM calls and tool runs, and role-aware harvesting and pooling can reclaim idle CPU and GPU capacity.

  3. AgentCgroup: Understanding and Controlling OS Resources of AI Agents

    cs.OS 2026-02 conditional novelty 6.0 of 10

    AI coding-agent workloads show 15.4× memory peak-to-average, tool-call-driven bursts, and 1.8× run-to-run variance, motivating per-tool-call eBPF/cgroup enforcement.

Reference graph

Works this paper leans on

60 extracted references · 53 canonical work pages · cited by 3 Pith papers

  1. [1]

    Survey surfaces rapid adoption of agentic AI

    Mike Vizard. Survey surfaces rapid adoption of agentic AI. https://techstrong.ai/agentic-ai/survey -surfaces-rapid-adoption-of-agentic-ai/ , May 2025. Accessed 2025-06-15

    work page 2025

  2. [2]

    96% of enterprises are expanding use of AI agents

    Cloudera Press Release. 96% of enterprises are expanding use of AI agents. https://www.cloudera.com /about/news-and-blogs/press-releases/2025-04-16-96-percent-of-enterprises-are-expan ding-use-of-ai-agents-according-to-latest-data-from-cloudera.html , April 2025. Accessed 2025-06-15

    work page 2025

  3. [3]

    PwC’s AI agent survey

    PwC. PwC’s AI agent survey. https://www.pwc.com/us/en/tech-effect/ai-analytics/ai-agent-s urvey.html, May 2025. Accessed 2025-06-15

    work page 2025

  4. [4]

    ReAct: Synergizing reasoning and acting in language models, 2023

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models, 2023

    work page 2023

  5. [5]

    Auto-GPT for online decision making: Benchmarks and additional opinions, 2023

    Hui Yang, Sifu Yue, and Yunzhong He. Auto-GPT for online decision making: Benchmarks and additional opinions, 2023

    work page 2023

  6. [6]

    CAMEL: Communicative agents for "mind" exploration of large language model society, 2023

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative agents for "mind" exploration of large language model society, 2023

    work page 2023

  7. [7]

    AI agents under threat: A survey of key security challenges and future pathways, 2024

    Zehang Deng, Yongjian Guo, Changzhou Han, Wanlun Ma, Junwu Xiong, Sheng Wen, and Yang Xiang. AI agents under threat: A survey of key security challenges and future pathways, 2024

    work page 2024

  8. [8]

    Multi-agent collaboration via evolving orchestration, 2025

    Yufan Dang, Chen Qian, Xueheng Luo, Jingru Fan, Zihao Xie, Ruijie Shi, Weize Chen, Cheng Yang, Xiaoyin Che, Ye Tian, Xuantang Xiong, Lei Han, Zhiyuan Liu, and Maosong Sun. Multi-agent collaboration via evolving orchestration, 2025

    work page 2025

Show all 60 references
  1. [9]

    A survey of AI agent protocols, 2025

    Yingxuan Yang, Huacan Chai, Yuanyi Song, Siyuan Qi, Muning Wen, Ning Li, Junwei Liao, Haoyi Hu, Jianghao Lin, Gaowei Chang, Weiwen Liu, Ying Wen, Yong Yu, and Weinan Zhang. A survey of AI agent protocols, 2025

    2025

  2. [10]

    Morley Mao, Beidi Chen, Fan Lai, and Atul Prakash

    Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Z. Morley Mao, Beidi Chen, Fan Lai, and Atul Prakash. Learn to be efficient: Build structured sparsity in large language models, 2024

    2024

  3. [11]

    Evaluating quantized large language models, 2024

    Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models, 2024

    2024

  4. [12]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023

    2023

  5. [13]

    Efficient large-scale language model training on GPU clusters

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on GPU clusters. ...

    2021 arXiv

  6. [14]

    Siegel, Nitya Nadgir, and Arvind Narayanan

    Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI agents that matter, 2024

    2024

  7. [15]

    Russell and P

    S.J. Russell and P. Norvig. Artificial Intelligence: A Modern Approach . Prentice Hall international editions. Prentice Hall, 1995

    1995

  8. [16]

    Multi-agent architectures in LangGraph

    LangGraph Contributors. Multi-agent architectures in LangGraph. https://langchain-ai.github.io/lang graph/concepts/multi_agent/, 2024. Accessed: 2025-06-14. 23 Efficient and Scalable Agentic AI with Heterogeneous Systems A PREPRINT

    2024

  9. [17]

    LLaMA: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language...

    2023

  10. [18]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. Technical report

    2018

  11. [19]

    BERT: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, 2019

    2019

  12. [20]

    NVIDIA Blackwell Datasheet

    NVIDIA. NVIDIA Blackwell Datasheet. Datasheet 3384703, NVIDIA, 2024. Dec 2024

    2024

  13. [21]

    NVLink & NVSwitch: Fastest HPC Data Center Platform

    NVIDIA. NVLink & NVSwitch: Fastest HPC Data Center Platform. https://www.nvidia.com/en-us/data -center/nvlink/, 2025. Accessed: 2025-06-15

    2025

  14. [22]

    NVIDIA InfiniBand (Ethernet & InfiniBand Networking Products)

    NVIDIA. NVIDIA InfiniBand (Ethernet & InfiniBand Networking Products). https://www.nvidia.com/e n-us/networking/products/infiniband/, 2025. Accessed: 2025-06-15

    2025

  15. [23]

    NVIDIA 800V HVDC Architecture Will Power the Next Generation of AI Factories

    NVIDIA. NVIDIA 800V HVDC Architecture Will Power the Next Generation of AI Factories. https: //developer.nvidia.com/blog/nvidia-800-v-hvdc-architecture-will-power-the-next-gener ation-of-ai-factories/ , May 2025. Accessed: 2025-06-15

    2025

  16. [24]

    NVIDIA A40 datasheet

    NVIDIA. NVIDIA A40 datasheet. https://images.nvidia.com/content/Solutions/data-center/a4 0/nvidia-a40-datasheet.pdf. Accessed: 2025-06-14

    2025

  17. [25]

    NVIDIA A100 datasheet

    NVIDIA. NVIDIA A100 datasheet. https://www.nvidia.com/content/dam/en-zz/Solutions/Data-C enter/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf . Accessed: 2025-06-14

    2025

  18. [26]

    NVIDIA H100 datasheet

    NVIDIA. NVIDIA H100 datasheet. https://resources.nvidia.com/en-us-gpu-resources/h100-dat asheet-24306. Accessed: 2025-06-14

    2025

  19. [27]

    NVIDIA B200 datasheet

    NVIDIA. NVIDIA B200 datasheet. https://resources.nvidia.com/en-us-dgx-systems/dgx-b200-d atasheet. Accessed: 2025-06-14

    2025

  20. [28]

    Intel Gaudi 3 AI accelerator white paper

    Intel. Intel Gaudi 3 AI accelerator white paper. https://cdrdv2-public.intel.com/817486/gaudi-3-a i-accelerator-white-paper.pdf . Accessed: 2025-06-14

    2025

  21. [29]

    https://www.amd.com/content/dam/amd/en/documents/instinc t-tech-docs/data-sheets/amd-instinct-mi300x-data-sheet.pdf

    AMD Instinct MI300X accelerator. https://www.amd.com/content/dam/amd/en/documents/instinc t-tech-docs/data-sheets/amd-instinct-mi300x-data-sheet.pdf . Accessed: 2025-07-23

    2025

  22. [30]

    https://resources.nvidia.com/en-us-briefcase-for-datashe ets/proviz-print-rtx6000-1

    NVIDIA RTX 6000 Ada Generation. https://resources.nvidia.com/en-us-briefcase-for-datashe ets/proviz-print-rtx6000-1 . Accessed: 2025-07-23

    2025

  23. [31]

    MLIR: Scaling compiler infrastructure for domain specific computation

    Chris Lattner, Mehdi Amini, Uday Bondhugula, Albert Cohen, Andy Davis, Jacques Pienaar, River Riddle, Tatiana Shpeisman, Nicolas Vasilache, and Oleksandr Zinenko. MLIR: Scaling compiler infrastructure for domain specific computation. In 2021 IEEE/ACM International Symposium on...

    2021

  24. [32]

    LLVM: A Compilation Framework for Lifelong Program Analysis and Transformation

    LLVM Project. LLVM: A Compilation Framework for Lifelong Program Analysis and Transformation. https: //llvm.org, 2023. https://llvm.org

    2023

  25. [33]

    TensorRT

    NVIDIA Corporation. TensorRT. h t t p s : / / d e v e l o p e r . n v i d i a . c o m / t e n s o r r t, 2023. https://developer.nvidia.com/tensorrt

    2023

  26. [34]

    XLA: Optimizing compiler for machine learning

    Google Research. XLA: Optimizing compiler for machine learning. https://www.tensorflow.org/xla ,

  27. [35]

    Apache TVM: An open deep learning compiler stack

    TVM Contributors. Apache TVM: An open deep learning compiler stack. https://tvm.apache.org, 2023. https://tvm.apache.org

    2023

  28. [36]

    IREE: Intermediate representation execution environment

    OpenXLA Contributors. IREE: Intermediate representation execution environment. https://openxla.org/pr ojects/iree, 2023. https://openxla.org/projects/iree

    2023

  29. [37]

    Glow: Graph lowering neural network compiler.https://github.com/pytorch/glow,

    Facebook AI Research. Glow: Graph lowering neural network compiler.https://github.com/pytorch/glow,

  30. [38]

    Roofline: an insightful visual performance model for multicore architectures

    Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM, 52(4):65–76, 2009

    2009

  31. [39]

    https://github.com/pytorch/glow

  32. [40]

    RoCE Initiative

    InfiniBand Trade Association. RoCE Initiative. https://www.roceinitiative.org/about-overview/ ,

  33. [41]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 24 Efficient and Scalable Agentic AI with He...

    2024 arXiv

  34. [42]

    Beyond the buzz: A pragmatic take on inference disaggregation

    Tiyasa Mitra, Ritika Borkar, Nidhi Bhatia, Ramon Matas, Shivam Raj, Dheevatsa Mudigere, Ritchie Zhao, Maximilian Golub, Arpan Dutta, Sailaja Madduri, et al. Beyond the buzz: A pragmatic take on inference disaggregation. arXiv preprint arXiv:2506.05508, 2025

    2025 arXiv

  35. [43]

    Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei...

    2024

  36. [44]

    Rdma over ethernet for distributed training at meta scale

    Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mo- hammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, Shuqiang Zhang, Mikel Jimenez Fernandez, Shashidhar Gandham, and Hongyi Zeng. Rdma over ethernet for distributed trai...

    2024

  37. [45]

    SGLang: Serving LLMs with programmatic prompts and high throughput

    SGLang Contributors. SGLang: Serving LLMs with programmatic prompts and high throughput. https: //github.com/InternLM/sglang, 2024. Accessed: 2025-06-15

    2024

  38. [46]

    Splitwise: Efficient generative llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118–132...

    2024

  39. [47]

    TensorRT-LLM

    NVIDIA. TensorRT-LLM. https://github.com/NVIDIA/TensorRT-LLM, 2023. Accessed: 2025-06-15

    2023

  40. [48]

    llm-d: A kubernetes-native high-performance distributed LLM inference framework

    llm-d community. llm-d: A kubernetes-native high-performance distributed LLM inference framework. https: //llm-d.ai/, 2025. Accessed: 2025-06-15

    2025

  41. [49]

    IREE: Intermediate representation execution environment

    IREE Team. IREE: Intermediate representation execution environment. https://github.com/openxla/iree,

  42. [50]

    Dynamo: A datacenter -scale distributed inference serving framework

    NVIDIA. Dynamo: A datacenter -scale distributed inference serving framework. https://github.com/ai-d ynamo/dynamo, 2025. Accessed: 2025-06-15

    2025

  43. [51]

    Triton: An intermediate language and compiler for tiled neural network computations

    Philippe Tillet, Hyeontaek Johnson, and Christos Kozyrakis. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (AS...

    2022

  44. [52]

    CrewAI: Orchestrate role-playing AI agents with memory and tools

    CrewAI Team. CrewAI: Orchestrate role-playing AI agents with memory and tools. https://github.com/joa omdmoura/crewAI, 2024. Accessed: 2025-06-15

    2024

  45. [53]

    AutoGen: Enabling next-gen LLM applications via multi-agent conversations

    Yizhou Wu, Ziniu Song, Haotian Li, et al. AutoGen: Enabling next-gen LLM applications via multi-agent conversations. https://microsoft.github.io/autogen, 2023. Microsoft Research, Accessed: 2025-06-15

    2023

  46. [54]

    MHLO: MLIR HLO dialect for tensorflow and JAX

    MHLO Contributors. MHLO: MLIR HLO dialect for tensorflow and JAX. https://github.com/openxla/m lir-hlo, 2021. Accessed: 2025-06-15

    2021

  47. [55]

    Distributed scheduling and optimization in datacenters

    Kai Chen, Hong Hu, Yongqiang Chen, and Jianfei Bai. Distributed scheduling and optimization in datacenters. IEEE Communications Magazine, 56(5):84–89, 2018. 25 Efficient and Scalable Agentic AI with Heterogeneous Systems A PREPRINT

    2018

  48. [56]

    Minions: Cost-efficient collaboration between on-device and cloud language models, 2025

    Avanika Narayan, Dan Biderman, Sabri Eyuboglu, Avner May, Scott Linderman, James Zou, and Christopher Re. Minions: Cost-efficient collaboration between on-device and cloud language models, 2025. 26

    2025

  49. [58]

    The tail at scale

    Jeffrey Dean and Luiz André Barroso. The tail at scale. Communications of the ACM, 56(2):74–80, 2013

    2013

  50. [2022]

    Accessed: 2025-06-15

    2025

  51. [2023]

    https://www.tensorflow.org/xla

  52. [2025]

    https://www.roceinitiative.org/about-overview/

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.