Pith. sign in

REVIEW 4 major objections 5 minor 88 references

TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Tensor lifecycle management is a missing abstraction in LLM infrastructure, and a service layer built for it matches specialized systems while enabling composable policies.

desk verdict A serious systems paper with a real design idea: a unified tensor lifecycle layer that is well built and honestly evaluated, but whose core generality claim currently rests on only two inference-engine integrations. read the letter →

arxiv 2608.06007 v1 pith:BQNX2JQR submitted 2026-08-06 cs.DC

Yuhan Zhou , Yuchu Luo , Hao Nie , Wangrunze Lv , Yu Zhou , Yibo Zhu , Daxin Jiang , Chenren Xu This is my paper
classification cs.DC
keywords tensorlifecyclemanagementTensor-as-a-ServiceKVcacheLLMservinginfrastructuremodelweightmaterializationpolicy-mechanismseparationdistributedpoolrequestrouting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern LLM infrastructure increasingly treats tensors — model weights, KV caches, checkpoints — as persistent shared states, yet each workload embeds its own mechanisms for moving and materializing them, creating isolated optimization silos. This paper argues that tensor lifecycle management is a missing abstraction layer and proposes Tensor-as-a-Service (TaaS): a layer that decouples tensor state management from computation logic. The paper builds TensorCast, a distributed runtime that exposes tensors as first-class artifacts with programmable lifecycle primitives, and integrates it with vLLM and SGLang. Through evaluations on weight materialization, weight synchronization, KV cache reuse, and request routing, the paper claims TensorCast matches or beats specialized systems on the tested workloads, and reports that a policy written with TensorCast APIs cuts median TTFT by up to 93.2% under highly concurrent multi-turn agent workloads. If correct, LLM infrastructure can stop reimplementing tensor movement per workload and instead compose lifecycle operations as a reusable service.

What carries the argument

The load-bearing object is the Artifact, a first-class handle that decouples a tensor's identity, ownership, and lifecycle semantics from its physical representation and device placement. Callers compose a finite set of lifecycle primitives — organized as worker Operations (prefetch, pin) and instance Operations (publish, hydrate, transform_into) — into Plans whose steps form a DAG with tracing, deadlines, and idempotency keys; Signal APIs close the loop by exposing worker and instance load state. The runtime handles distributed execution and data movement, using a lightweight global store for low-cardinality tensors such as model weights and shard-home workers chosen by Highest Random Weight hashing with leases and fencing tokens for high-cardinality tensors such as KV pages. The instance adaptor is the mechanism boundary that exports and imports engine-resident tensors, which is what makes the policy-mechanism separation possible.

What would settle it

Integrate TensorCast with a third execution framework, such as Megatron-LM or TensorRT-LLM, and implement a policy that composes checkpoint resharding with weight synchronization; if this requires adding new instance-adaptor interfaces, modifying engine code, or changing the runtime rather than writing a caller-level program, then the claimed decoupling of policy from mechanism fails.

Watch

Extended reading notes

Core claim

The central claim is that tensor lifecycle management — identifying, placing, materializing, transforming, and coordinating tensor states — constitutes a missing layer in LLM infrastructure, and that this layer can be built as a service without sacrificing performance. TensorCast realizes this with four abstractions: Artifacts that give tensors stable identity and ownership independent of physical location, Operations that express lifecycle moves (publish, prefetch, hydrate, transform), Plans that compose operations into idempotent, dependency-ordered workflows, and Signals that expose runtime state for policy decisions. The paper reports that a general mechanism built this way is competitive with specialized systems on model weight loading, weight synchronization, and KV cache sharing, while a programmable request router that composes load balancing with KV migration improves median TTFT by up to 93.2% at 256 concurrent multi-turn agent sessions.

Load-bearing premise

The central assumption is that engine-resident tensor states (KV caches, weights, checkpoints) can be exported, imported, and transformed through a thin instance adaptor without modifying the execution engines themselves, and the evidence covers only vLLM and SGLang on four inference workloads.

Editorial extensions

If this is right

  • New tensor management policies can be implemented as caller programs without modifying vLLM or SGLang internals, making routing, load balancing, and KV placement strategies directly swappable.
  • The same lifecycle primitives cover model weight materialization, weight synchronization, and KV cache sharing, so separate specialized subsystems could be consolidated into one layer.
  • KV reuse performance is comparable to Mooncake under RDMA and better without it, so the general abstraction does not sacrifice the main KV optimization.
  • A router that composes load balancing with KV migration preserves cache hit rates under heavy multi-turn concurrency, cutting median TTFT by up to 93.2%.
  • Checkpoints and weight versions map to versioned artifacts and views, indicating the abstraction extends beyond serving to training-side lifecycle workloads.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the thin-adaptor claim holds, adding a new engine or workload, such as tree-of-thought branching or cross-datacenter KV reuse, would reduce to writing caller policies rather than building new systems.
  • The instance adaptor is the seam to watch: the paper demonstrates only two inference engines, so a third engine (especially a training framework) is the natural test of whether policy-mechanism separation generalizes.
  • The shard-lease and fencing design for high-cardinality tensors generalizes to any data-centric service that must index millions of small mutable objects without a metadata bottleneck.
  • One testable extension is applying the same router policy to training-serving synchronization, where weights and KV caches change on different timescales, to see whether composed lifecycle policies remain effective.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TensorCast, a distributed tensor management layer that operationalizes the ``Tensor-as-a-Service'' (TaaS) abstraction for LLM infrastructure. TensorCast decouples tensor lifecycle management from execution engines by introducing four programming abstractions (Artifact, Operation, Plan, Signal), a caller-worker runtime with a global store, and instance adaptors that export/import engine-resident tensors. The authors integrate TensorCast with vLLM and SGLang and evaluate four representative workloads: model weight materialization, weight synchronization, KV cache management, and programmable request routing. The headline result is a 93.2% median TTFT reduction under a custom rebalancing policy for agentic multi-turn workloads, with competitive or better performance than specialized systems such as Mooncake and InstantTensor.

Significance. If the results hold, the paper offers a valuable architectural contribution: it demonstrates that a single uniform tensor lifecycle layer can match the performance of specialized tensor management systems on individual tasks while enabling cross-component policies that are difficult to express in siloed stacks. The open-sourced implementation and the honest disclosure of limitations (notably the deferred training-framework integration) are strengths. The reported improvements are substantial, but the evidence for the central ``thin adaptor'' claim is narrower than the abstraction suggests, and the evaluation methodology needs to be tightened to support the quantitative conclusions.

major comments (4)
  1. [§6.1, Figures 5 and 6] The ``TensorCast warm'' configuration pre-materializes weight slices via the prefetch API before the measured interval begins, so the reported weight load time (e.g., 0.345 s) and end-to-end ready time (e.g., 0.367 s for Qwen3-30B-A3B) exclude the cost of that prefetch. The baselines (Default, InstantTensor) do not have an equivalent warm start; thus the speedups of 228.6x and 40.7x over Default and InstantTensor are not apples-to-apples comparisons. The paper should either include the prefetch cost in the measured interval or demonstrate that this pre-materialization overlaps with instance launch in a way baselines could also exploit (e.g., by adding a warm-start baseline for the competing systems).
  2. [§3.1, §4.1, §7] The central architectural claim is that a thin instance adaptor can export, import, and transform engine-resident tensor states, so policies can be decoupled from execution engines. The evidence is narrower: the vLLM integration in §6.1 replaces the weight loading module rather than exporting an existing engine state, and the SGLang weight synchronization in §6.2 replaces the weight updater. The only genuine export/import boundary exercised is the KV-cache path in §6.3, which reuses SGLang's existing HiCache/backend interface. Section 7 explicitly defers Megatron-LM and DeepSpeed integration. If each engine requires a bespoke adaptor that reaches into proprietary allocators or checkpoint formats, the claimed decoupling reduces to an interface contract for a small set of modular inference engines. Please either add at least one third-engine integration or a training checkpoint/optimizer-state integration, or explicitly scope the abstraction claim to modular inference engines and state the conditions under which the adaptor remains thin.
  3. [§6.1–§6.4] All figures report single-point measurements without error bars, confidence intervals, or a stated number of repetitions. The paper draws strong quantitative conclusions (e.g., 228.6x launch speedup in §6.1, 8x TTFT reduction in §6.3, and 93.2% TTFT reduction in §6.4). Without variance information, the reader cannot judge whether these improvements are stable or within measurement noise. Please add repeated trials (at least 3–5 per configuration) and report mean/median with dispersion, or justify why single runs are representative.
  4. [§6.4, Algorithm 1] The rebalancing policy that delivers the headline 93.2% median TTFT reduction has several hyperparameters (alpha, theta_abs, theta_rel, H, lambda, and the tick interval), and the workloads rely on LogNormal presets defined in Table 3. The paper does not report how these hyperparameters were selected or whether the reported reductions are robust to their variation. Since this result is central to the programmability claim, please provide the tuning procedure and a sensitivity analysis (e.g., varying each hyperparameter over a small range).
minor comments (5)
  1. [Figure 8 caption] The caption contains a typo: ``redcution'' should be ``reduction''.
  2. [Reference [62]] The reference title is misspelled: ``InstentTensor'' should be ``InstantTensor''.
  3. [§6.3, Figures 9b and 9d] The y-axis of these figures shows negative TTFT reductions, meaning the KV reuse is slower than the first-instance baseline. The text should explain these negative values (e.g., retrieval overhead exceeding reuse benefit) rather than focusing only on positive cases.
  4. [§6.3, setup] The KV-cache workload resends identical prompts to all instances, yielding a 100% cache hit rate. This stress-tests retrieval but does not exercise eviction, fragmentation, or partial hits, so the claim of ``comparable performance with specialized KV systems'' under realistic workloads is not fully demonstrated. A short discussion of this limitation would help.
  5. [§6.4, baselines] The paper should briefly describe how the three baseline routers are implemented (especially the ``load-aware + Mooncake'' combination) so readers can assess whether the comparison is fair to Mooncake, which is a KV cache backend rather than a router.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: TensorCast's claims are supported by external baselines, with scope limits disclosed rather than disguised.

full rationale

I examined the full text for circular reductions. TensorCast's efficiency claims in Sections 6.1-6.3 are measured against external baselines: vLLM's default loader, InstantTensor, and Mooncake. The programmability claim in Section 6.4 compares a TensorCast-composed rebalancing router against load-aware, load-aware-plus-Mooncake, and cache-aware baselines. No parameter is fitted to the target metric and then reported as a prediction; the rebalancing policy's hyperparameters are specified in Algorithm 1, but no result is derived from them by construction. The 'missing layer' argument is a design thesis, not a theorem, and the paper expressly disclaims universality: Section 7 leaves Megatron-LM and DeepSpeed integration to future work and states, 'We do not claim that a general abstraction is inherently faster than every specialized implementation.' The only works affiliated with the authors' ecosystem are not load-bearing: Mooncake is used as an independent baseline, and the SGLang HiCache interface is an external integration point. No self-citation carries a correctness or uniqueness claim. Thus no step reduces to its inputs by definition or by fitted value; the central empirical claims are self-contained against outside systems, and the abstraction claim is honestly scoped.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central claim rests primarily on domain assumptions about workload commonality, adaptor feasibility, and representative measurement, plus several undisclosed policy constants that influence the headline TTFT number. No physical entities are introduced; the new artifacts are software abstractions with open-source implementations.

free parameters (3)
  • Rebalancing policy hyperparameters (alpha, theta_abs, theta_rel, H, lambda, tick interval) = not reported
    Algorithm 1 in Appendix A uses these constants to decide when and which requests to migrate. The 93.2% TTFT claim in Section 6.4 depends on them, but values and sensitivity are not given.
  • LogNormal workload presets (mu, sigma) = fast (2.1, 0.6), medium (3.0, 0.8), slow (4.1, 1.0)
    Table 3 defines inter-turn delays for the agent workload. These hand-chosen presets shape queue pressure and directly influence measured TTFT curves.
  • HRW shard top-k = k = 3
    Section 4.2 selects k=3 for shard lease acquisition to reduce contention; no sensitivity analysis is reported for this design constant.
assumptions (4)
  • domain assumption The five lifecycle primitives (identify/own, place/move, materialize, transform, compose/coordinate) form a common basis across LLM tensor management workloads.
    Section 2.2 and Table 1 assert this from three example workloads; no systematic survey or formal derivation is provided.
  • standard math Lease-based shard ownership with fencing tokens and HRW hashing provides correct consistency without becoming a bottleneck.
    Section 4.2 relies on standard distributed systems techniques, but no correctness proof or failure-injection evaluation is presented.
  • domain assumption A thin instance adaptor can expose engine-resident tensors with negligible overhead and without changing engine semantics.
    Sections 3.1 and 4.1 assume this boundary; only vLLM and SGLang integrations are demonstrated, while training engines are deferred in Section 7.
  • domain assumption Userspace mTCP with multipath transmission outperforms kernel TCP in the tested non-RDMA settings.
    Section 6.3 attributes TensorCast's non-RDMA advantage to mTCP, but no independent network-level benchmark isolates this mechanism.
invented entities (2)
  • Artifact independent evidence
    purpose: First-class logical handle for tensor identity, ownership, and lifecycle, decoupled from physical placement.
    Implemented in the open-source TensorCast runtime and exercised in all four evaluation workloads; it is a software abstraction, not a physical entity.
  • Tensor-as-a-Service (TaaS) layer independent evidence
    purpose: Conceptual abstraction layer separating tensor lifecycle management from computation logic in LLM infrastructure.
    Realized by TensorCast and evaluated against specialized baselines; the conceptual layer itself is a framing device rather than an independently falsifiable entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure." pith.science (2026). https://pith.science/paper/BQNX2JQR

@misc{pith2026260806007,
  author       = {Pith},
  title        = {Pith review of: TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BQNX2JQR}},
  note         = {Machine review of arXiv:2608.06007}
}
read the original abstract

Modern LLM infrastructure increasingly manages tensors not only as computation data, but also as persistent states shared across distributed components. Existing systems optimize individual tensor management tasks, such as model weight loading, KV cache management, and checkpoint synchronization, by deeply integrating task-specific mechanisms with execution engines, networks, or storage backends. However, this specialization creates isolated silos that hinder the reuse and composition of tensor management strategies across evolving LLM workloads. In this paper, we identify tensor lifecycle management as a missing abstraction layer in LLM infrastructure and propose Tensor-as-a-Service (TaaS), which decouples tensor state management from computation logic. We design and build TensorCast, a distributed tensor management layer that provides first-class tensor abstractions, programmable lifecycle primitives, and a runtime that separates tensor management policies from execution mechanisms. This enables developers to write tensor management programs using TensorCast APIs while transparently leveraging distributed execution and data movement. We integrate TensorCast with vLLM and SGLang and evaluate it across diverse tensor lifecycle workloads, including model weight materialization, weight synchronization, KV cache management, and programmable request routing. Our results show that TensorCast achieves competitive performance with specialized tensor management systems while enabling new cross-component optimization policies. A programmable policy implemented with TensorCast improves median TTFT by up to 93.2% under highly concurrent multi-turn agent workloads.

Figures

Figures reproduced from arXiv: 2608.06007 by the authors.

Figure 1
Figure 1. Existing LLM infrastructure embeds tensor [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Decoupling tensor lifecycle management from computation. A unified Tensor-as-a-Service (TaaS) layer allows developers to compose tensor man￾agement task programs across workloads. are distributed across elastic serving instances, KV caches are reused across nodes, and updated model weights are synchronized between training and inference pipelines. A large body of research has optimized individual tensor management t… view at source ↗
Figure 3
Figure 3. TensorCast’s programming model. the TaaS layer should provide programmable lifecycle primi￾tives rather than hard-code optimization strategies. Different from compute-centric frameworks such as Spark [26] and Ray [25], where users dispatch computation tasks, TaaS users orchestrate tensor states by composing lifecycle operations and defining workload-specific policies. This programmabil￾ity enables system extensibili… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: TensorCast system architecture. All com￾ponents belong to the same TensorCast cluster ex￾cept for the Req. Router App process. Steps 1 – 8 illus￾trate the workflow of executing the KV migration task shown in List. 1 (line 12–23). See detailed explanation in §4.5. Tenso…
Figure 5
Figure 5. Figure 5: Instance launch time for Qwen3-30B-A3B model. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Instance launch time for Qwen3-235B-A22B model. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Model weight synchronization time. Setup. We use a single node with 64 CPUs, 500 GB DRAM, and 8 × H800 GPUs to launch a vLLM instance (TP=8). This setup emulates a MaaS auto-scaling scenario, where new instances are launched as quickly as possible. We load model weight…
Figure 8
Figure 8. Figure 8: Qwen3-32B TTFT redcution ratio with TensorCast and Mooncake KV storage backend. 2 4 Node Number 0 10 20 TTFT Reduction (%) Mooncake TensorCast (a) 16k prompt w/ RDMA. 2 4 Node Number 150 100 50 0 TTFT Reduction (%) Mooncake TensorCast (b) 16k prompt w/o RDMA. 2 4 Node …
Figure 9
Figure 9. Figure 9: Qwen3-235B-A22B TTFT reduction ratio with [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Request TTFT against different router policies with three levels of agentic coding workloads. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Request cache hit rate against different router policies with three levels of agentic coding workloads. [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

88 extracted references · 54 canonical work pages

  1. [1]

    https://www.kimi.com/blog/kimi- k3

    Kimi k3: Open frontier intelligence. https://www.kimi.com/blog/kimi- k3. (Accessed on 07/23/2026)

    2026

  2. [2]

    ServerlessLLM: Low-Latency serverless inference for large language models

    Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. ServerlessLLM: Low-Latency serverless inference for large language models. InUSENIX OSDI, 2024

    2024

  3. [3]

    Deep- flow: Serverless large language model serving at scale.arXiv e-prints, pages arXiv–2501, 2025

    Junhao Hu, Jiang Xu, Zhixia Liu, Yulong He, Yuetao Chen, Hao Xu, Jiang Liu, Baoquan Zhang, Shining Wan, Gengyuan Dan, et al. Deep- flow: Serverless large language model serving at scale.arXiv e-prints, pages arXiv–2501, 2025

    2025

  4. [4]

    Faaswap: Slo-aware, gpu-efficient serverless inference via model swap- ping.arXiv preprint arXiv:2306.03622, 2023

    Minchen Yu, Ao Wang, Dong Chen, Haoxuan Yu, Xiaonan Luo, Zhuo- hao Li, Wei Wang, Ruichuan Chen, Dapeng Nie, and Haoran Yang. Faaswap: Slo-aware, gpu-efficient serverless inference via model swap- ping.arXiv preprint arXiv:2306.03622, 2023

    arXiv 2023

  5. [5]

    Minchen Yu, Rui Yang, Chaobo Jia, Zhaoyuan Su, Sheng Yao, Tingfeng Lan, Yuchen Yang, Yue Cheng, Wei Wang, Ao Wang, et al.{\lambda} scale: Enabling fast scaling for serverless large language model infer- ence.arXiv preprint arXiv:2502.09922, 2025

    arXiv 2025

  6. [6]

    BlitzScale: Fast and Live Large Model Autoscaling with O (1) Host Caching

    Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. BlitzScale: Fast and Live Large Model Autoscaling with O (1) Host Caching. InUSENIX OSDI, 2025

    2025

  7. [7]

    Hydraserve: Minimizing cold start latency for serverless llm serving in public clouds.arXiv preprint arXiv:2502.15524, 2025

    Chiheng Lou, Sheng Qi, Chao Jin, Dapeng Nie, Haoran Yang, Yu Ding, Xuanzhe Liu, and Xin Jin. Hydraserve: Minimizing cold start latency for serverless llm serving in public clouds.arXiv preprint arXiv:2502.15524, 2025

    arXiv 2025

  8. [8]

    https://lmsys.org/blog/2025-12-10-rfork/

    Let Tensors Fly — Accelerating Large Model Weight Loading with R-Fork. https://lmsys.org/blog/2025-12-10-rfork/. (Accessed on 01/28/2026)

    2025

Show all 88 references
  1. [9]

    Mooncake: Trad- ing more storage for less computation—a KVCache-centric architecture for serving LLM chatbot

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trad- ing more storage for less computation—a KVCache-centric architecture for serving LLM chatbot. InUSENIX FAST, 2025

    2025

  2. [10]

    Dualmap: Enabling both cache affinity and load bal- ancing for distributed LLM serving

    Anonymous. Dualmap: Enabling both cache affinity and load bal- ancing for distributed LLM serving. InSubmitted to The Fourteenth International Conference on Learning Representations, 2025. under re- view

    2025

  3. [11]

    Lmcache: An efficient kv cache layer for enterprise-scale llm inference

    Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaot- ing Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, et al. Lmcache: An efficient kv cache layer for enterprise-scale llm inference. arXiv preprint arXiv:2510.09665, 2025

  4. [12]

    https://lmsys.org/blog/2025-09-10-sglang-hicache/

    SGLang HiCache. https://lmsys.org/blog/2025-09-10-sglang-hicache/. (Accessed on 01/19/2026)

    2025

  5. [13]

    Stateful large language model serving with pensieve

    Lingfan Yu, Jinkun Lin, and Jinyang Li. Stateful large language model serving with pensieve. InACM EuroSys, 2025

    2025

  6. [14]

    Cacheblend: Fast large language model serving for rag with cached knowledge fusion

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In ACM EuroSys, 2025

    2025

  7. [15]

    Cost-Efficient large language model serving for multi-turn conversations with Cache- dAttention

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Cost-Efficient large language model serving for multi-turn conversations with Cache- dAttention. InUSENIX ATC, 2024

    2024

  8. [16]

    {ByteCheckpoint}: A unified checkpointing system for large foundation model development

    Borui Wan, Mingji Han, Yiyao Sheng, Yanghua Peng, Haibin Lin, Mo- fan Zhang, Zhichao Lai, Menghan Yu, Junda Zhang, Zuquan Song, et al. {ByteCheckpoint}: A unified checkpointing system for large foundation model development. InUSENIX NSDI, 2025

    2025

  9. [17]

    https://github.com/MoonshotAI/chec kpoint-engine

    Moonshot checkpoint engine. https://github.com/MoonshotAI/chec kpoint-engine. (Accessed on 01/19/2026)

    2026

  10. [18]

    https://vllm.ai/

    vLLM: The High-Throughput and Memory-Efficient inference and serving engine for LLMs. https://vllm.ai/. (Accessed on 01/19/2026). 15

    2026

  11. [19]

    Sglang: Efficient execution of structured language model programs.Advances in neural information processing systems, 37:62557–62583, 2024

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs.Advances in neural information processing systems, 37:...

    2024

  12. [20]

    https://nvidia.github.io/TensorRT-LLM/

    Tensorrt-llm. https://nvidia.github.io/TensorRT-LLM/. (Accessed on 01/19/2026)

    2026

  13. [21]

    NVIDIA DGX SuperPOD: Next Generation Scalable Infras- tructure for AI Leadership. https://docs.nvidia.com/dgx- superpod/reference-architecture/scalable-infrastructure- h200/latest/_downloads/bbd08041e98eb913619944ead1f92373/RA- 11336-001-DSPH200-ReferenceArch.pdf#page=8.10. (A...

  14. [22]

    https://redis.io/

    Redis – Your App is about to get faster. https://redis.io/. (Accessed on 01/22/2026)

    2026

  15. [23]

    Vineyard: Optimizing data sharing in data- intensive analytics.Proc

    Wenyuan Yu, Tao He, Lei Wang, Ke Meng, Ye Cao, Diwen Zhu, Sanhong Li, and Jingren Zhou. Vineyard: Optimizing data sharing in data- intensive analytics.Proc. ACM Manag. Data, 1(2), jun 2023

    2023

  16. [24]

    https://github.com/ray-project/plasma

    Ray Plasma. https://github.com/ray-project/plasma. (Accessed on 01/22/2026)

    2026

  17. [25]

    Ray: A distributed framework for emerging ai applications

    Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging ai applications. InUSENIX OSDI 18, 2018

    2018

  18. [26]

    Resilient distributed datasets: A Fault-Tolerant abstraction for In-Memory cluster computing

    Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, Michael J Franklin, Scott Shenker, and Ion Stoica. Resilient distributed datasets: A Fault-Tolerant abstraction for In-Memory cluster computing. InUSENIX NSDI, 2012

    2012

  19. [27]

    Serverless computing: Design, implementation, and performance

    Garrett McGrath and Paul R Brenner. Serverless computing: Design, implementation, and performance. In2017 IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW), pages 405–410. IEEE, 2017

    2017

  20. [28]

    Infless: a native serverless system for low-latency, high-throughput inference

    Yanan Yang, Laiping Zhao, Yiming Li, Huanyu Zhang, Jie Li, Mingyang Zhao, Xingzhen Chen, and Keqiu Li. Infless: a native serverless system for low-latency, high-throughput inference. InACM ASPLOS, 2022

    2022

  21. [29]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  22. [30]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  23. [31]

    Flexkv: Flexible index offloading for memory-disaggregated key-value store.arXiv preprint arXiv:2512.16148, 2025

    Zhisheng Hu, Jiacheng Shen, and Ming-Chang Yang. Flexkv: Flexible index offloading for memory-disaggregated key-value store.arXiv preprint arXiv:2512.16148, 2025

    2025

  24. [32]

    In USENIX OSDI, 2024

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang.{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In USENIX OSDI, 2024

    2024

  25. [33]

    Splitwise: Efficient genera- tive llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient genera- tive llm inference using phase splitting. In2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118–

  26. [34]

    Inference without interference: Disaggregate llm inference for mixed downstream workloads.arXiv preprint arXiv:2401.11181, 2024

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. Inference without interference: Disaggregate llm inference for mixed downstream workloads.arXiv preprint arXiv:2401.11181, 2024

    2024 arXiv

  27. [35]

    D \’ej\avu: Kv-cache streaming for fast, fault- tolerant generative llm serving.arXiv preprint arXiv:2403.01876, 2024

    Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. D \’ej\avu: Kv-cache streaming for fast, fault- tolerant generative llm serving.arXiv preprint arXiv:2403.01876, 2024

    2024 arXiv

  28. [36]

    Prompt cache: Modular attention reuse for low-latency inference.Proceedings of Machine Learning and Systems, 6:325–338, 2024

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khan- delwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference.Proceedings of Machine Learning and Systems, 6:325–338, 2024

    2024

  29. [37]

    Cachegen: Kv cache compression and streaming for fast large language model serving

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anantha- narayanan, et al. Cachegen: Kv cache compression and streaming for fast large language model serving. InACM SIGCOMM, 2024

    2024

  30. [38]

    Ragcache: Efficient knowledge caching for retrieval-augmented generation.ACM Transactions on Computer Sys- tems, 44(1):1–27, 2025

    Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Shufan Liu, Xu- anzhe Liu, and Xin Jin. Ragcache: Efficient knowledge caching for retrieval-augmented generation.ACM Transactions on Computer Sys- tems, 44(1):1–27, 2025

    2025

  31. [39]

    Attentionstore: Cost-effective attention reuse across multi-turn conversations in large language model serving.arXiv preprint arXiv:2403.19708, 52:20–38, 2024

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Attentionstore: Cost-effective attention reuse across multi-turn conversations in large language model serving.arXiv preprint arXiv:2403.19708, 52:20–38, 2024

    2024 arXiv

  32. [40]

    Dist checkpointing package

    Megatron Team. Dist checkpointing package. https: //docs.nvidia.com/megatron-core/developer-guide/latest/api- guide/dist_checkpointing.html. (Accessed on 01/26/2026)

    2026

  33. [41]

    Getting started with Distributed Check- point (DCP)

    PyTorch Team. Getting started with Distributed Check- point (DCP). https://docs.pytorch.org/tutorials/recipes/distributed_checkpo int_recipe.html. (Accessed on 01/26/2026)

    2026

  34. [42]

    Universal checkpointing: Efficient and flexible checkpointing for large scale distributed training

    Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko, Masahiro Tanaka, Stas Bekman, Olatunji Ruwase, and Minjia Zhang. Universal checkpointing: Efficient and flexible checkpointing for large scale distributed training. arXiv preprint arXiv:2406.18820, 2024

    2024 arXiv

  35. [43]

    Simple is better: Multiplication may be all you need for llm request scheduling

    Dingyan Zhang, Jinbo Han, Kaixi Zhang, Xingda Wei, Sijie Shen, Chen- guang Fang, Wenyuan Yu, Jingren Zhou, and Rong Chen. Simple is better: Multiplication may be all you need for llm request scheduling. InUSENIX OSDI, 2026

    2026

  36. [44]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InACM SOSP, 2023

    2023

  37. [45]

    https://docs.vllm.ai/en/stable/desig n/prefix_caching/

    vllm – Automatic Prefix Caching. https://docs.vllm.ai/en/stable/desig n/prefix_caching/. (Accessed on 01/19/2026)

    2026

  38. [46]

    https://lmsys.org/blog/2024-01-17-sglang/

    Fast and Expressive LLM Inference with RadixAttention and SGLang. https://lmsys.org/blog/2024-01-17-sglang/. (Accessed on 01/29/2026)

    2024

  39. [47]

    Pie: A pro- grammable serving system for emerging llm applications

    In Gim, Zhiyao Ma, Seung-seob Lee, and Lin Zhong. Pie: A pro- grammable serving system for emerging llm applications. InACM SOSP, 2025

    2025

  40. [48]

    Chain-of-thought prompt- ing elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompt- ing elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  41. [49]

    Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023

    2023

  42. [50]

    Training language models to follow instruc- tions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wain- wright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instruc- tions with human feedback.Advances in neural information processing systems, 35:2...

    2022

  43. [51]

    Seer: Online context learning for fast synchronous llm reinforcement learning.arXiv preprint arXiv:2511.14617, 2025

    Ruoyu Qin, Weiran He, Weixiao Huang, Yangkun Zhang, Yikai Zhao, Bo Pang, Xinran Xu, Yingdi Shan, Yongwei Wu, and Mingxing Zhang. Seer: Online context learning for fast synchronous llm reinforcement learning.arXiv preprint arXiv:2511.14617, 2025

    2025 arXiv

  44. [52]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint 16 arXiv:2501.12599, 2025

    2025 arXiv

  45. [53]

    Totrl: Unlock llm tree-of-thoughts reasoning potential through puzzles solving.arXiv preprint arXiv:2505.12717, 2025

    Haoyuan Wu, Xueyi Chen, Rui Ming, Jilong Gao, Shoubo Hu, Zhuolun He, and Bei Yu. Totrl: Unlock llm tree-of-thoughts reasoning potential through puzzles solving.arXiv preprint arXiv:2505.12717, 2025

    2025

  46. [54]

    Using name-based map- pings to increase hit rates.IEEE/ACM Transactions on networking, 6(1):1–14, 2002

    David G Thaler and Chinya V Ravishankar. Using name-based map- pings to increase hit rates.IEEE/ACM Transactions on networking, 6(1):1–14, 2002

    2002

  47. [55]

    Paxos made simple.ACM SIGACT News (Distributed Computing Column) 32, 4 (Whole Number 121, December 2001), pages 51–58, 2001

    Leslie Lamport. Paxos made simple.ACM SIGACT News (Distributed Computing Column) 32, 4 (Whole Number 121, December 2001), pages 51–58, 2001

    2001

  48. [56]

    In search of an understandable consensus algorithm

    Diego Ongaro and John Ousterhout. In search of an understandable consensus algorithm. InUSENIX ATC, pages 305–319, 2014

    2014

  49. [57]

    Chain replication for sup- porting high throughput and availability

    Robbert Van Renesse and Fred B Schneider. Chain replication for sup- porting high throughput and availability. InUSENIX OSDI, volume 4, 2004

    2004

  50. [58]

    Object storage on craq: High- throughput chain replication for read-mostly workloads

    Jeff Terrace and Michael J Freedman. Object storage on craq: High- throughput chain replication for read-mostly workloads. InUSENIX ATC, 2009

    2009

  51. [59]

    https://duckdb.org/

    DuckDB – Run analytics where your data lives. https://duckdb.org/. (Accessed on 06/24/2026)

    2026

  52. [60]

    mtcp: a highly scalable user-level tcp stack for multicore systems

    EunYoung Jeong, Shinae Wood, Muhammad Jamshed, Haewon Jeong, Sunghwan Ihm, Dongsu Han, and KyoungSoo Park. mtcp: a highly scalable user-level tcp stack for multicore systems. InUSENIX NSDI, 2014

    2014

  53. [61]

    https://github.com/juicedata/juicefs

    JuiceFS. https://github.com/juicedata/juicefs. (Accessed on 06/24/2026)

    2026

  54. [62]

    https://github.com/scitix/InstantTensor

    InstentTensor. https://github.com/scitix/InstantTensor. (Accessed on 06/24/2026)

    2026

  55. [63]

    Long- bench: A bilingual, multitask benchmark for long context understand- ing

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Long- bench: A bilingual, multitask benchmark for long context understand- ing. InProceedings of the 62nd annual meeting of the association for comp...

    2024

  56. [64]

    Prefill-as-a-service: Kvcache of next-generation models could go cross-datacenter.arXiv preprint arXiv:2604.15039, 2026

    Ruoyu Qin, Weiran He, Yaoyu Wang, Zheming Li, Xinran Xu, Yongwei Wu, Weimin Zheng, and Mingxing Zhang. Prefill-as-a-service: Kvcache of next-generation models could go cross-datacenter.arXiv preprint arXiv:2604.15039, 2026

    2026 arXiv

  57. [65]

    https://docs.sglang.io/docs/advanced_feature s/sgl_model_gateway

    Sglang model gateway. https://docs.sglang.io/docs/advanced_feature s/sgl_model_gateway. (Accessed on 07/12/2026)

    2026

  58. [66]

    The power of two choices in randomized load balancing.IEEE transactions on parallel and distributed systems, 12(10):1094–1104, 2002

    Michael Mitzenmacher. The power of two choices in randomized load balancing.IEEE transactions on parallel and distributed systems, 12(10):1094–1104, 2002

    2002

  59. [67]

    https://huggingf ace.co/datasets/SWE-Gym/OpenHands-Sampled-Trajectories

    Swe-gym/openhands-sampled-trajectories dataset. https://huggingf ace.co/datasets/SWE-Gym/OpenHands-Sampled-Trajectories. (Ac- cessed on 07/12/2026)

    2026

  60. [68]

    Statistical analysis of a telephone call center: A queueing-science perspective.Journal of the American statistical association, 100(469):36–50, 2005

    Lawrence Brown, Noah Gans, Avishai Mandelbaum, Anat Sakov, Haipeng Shen, Sergey Zeltyn, and Linda Zhao. Statistical analysis of a telephone call center: A queueing-science perspective.Journal of the American statistical association, 100(469):36–50, 2005

    2005

  61. [69]

    A poissonian explanation for heavy tails in e-mail communi- cation.Proceedings of the National Academy of Sciences, 105(47):18153– 18158, 2008

    R Dean Malmgren, Daniel B Stouffer, Adilson E Motter, and Luís AN Amaral. A poissonian explanation for heavy tails in e-mail communi- cation.Proceedings of the National Academy of Sciences, 105(47):18153– 18158, 2008

    2008

  62. [70]

    Megatron-lm: Training multi- billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi- billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  63. [71]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 35...

    2020

  64. [72]

    Alpa: Automating inter-and{Intra-Operator} par- allelism for distributed deep learning

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. Alpa: Automating inter-and{Intra-Operator} par- allelism for distributed deep learning. In16th USENIX Symposium on Operating Systems De...

    2022

  65. [73]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Min- jia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. InSC22: In- ternationa...

    2022

  66. [74]

    Loongserve: Efficiently serving long-context large lan- guage models with elastic sequence parallelism

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serving long-context large lan- guage models with elastic sequence parallelism. InACM SOSP, 2024

    2024

  67. [75]

    Fast dis- tributed inference serving for large language models.arXiv preprint arXiv:2305.05920, 2023

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast dis- tributed inference serving for large language models.arXiv preprint arXiv:2305.05920, 2023

    2023 arXiv

  68. [76]

    Orca: A distributed serving system for {Transformer-Based}generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based}generative models. InUSENIX OSDI, 2022

    2022

  69. [77]

    Taming{Throughput-Latency} tradeoff in{LLM} inference with {Sarathi-Serve}

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ram- jee. Taming{Throughput-Latency} tradeoff in{LLM} inference with {Sarathi-Serve}. InUSENIX OSDI, 2024

    2024

  70. [78]

    {InfiniGen}: Efficient generative inference of large language mod- els with dynamic{KV}cache management

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. {InfiniGen}: Efficient generative inference of large language mod- els with dynamic{KV}cache management. InUSENIX OSDI, 2024

    2024

  71. [79]

    Jenga: Effective memory management for serving llm with heterogeneity

    Chen Zhang, Kuntai Du, Shu Liu, Woosuk Kwon, Xiangxi Mo, Yufeng Wang, Xiaoxuan Liu, Kaichao You, Zhuohan Li, Mingsheng Long, et al. Jenga: Effective memory management for serving llm with heterogeneity. InACM SOSP, 2025

    2025

  72. [80]

    Bitnet: Scaling 1-bit transformers for large language models.arXiv preprint arXiv:2310.11453, 2023

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models.arXiv preprint arXiv:2310.11453, 2023

    2023 arXiv

  73. [81]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of machine learning and systems, 6:87–100, 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei- Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of machine learning and systems, 6:87–100, 2024

    2024

  74. [82]

    Flashattention: Fast and memory-efficient exact attention with io- awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io- awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    2022

  75. [83]

    Flashinfer: Efficient and customizable attention engine for llm inference serving.arXiv preprint arXiv:2501.01005, 2025

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. Flashinfer: Efficient and customizable attention engine for llm inference serving.arXiv preprint arXiv:2501.01005, 2025

    2025 arXiv

  76. [84]

    https://gith ub.com/langchain-ai/langchain

    LangChain: Build context-aware reasoning ap- plications. https://gith ub.com/langchain-ai/langchain. (Accessed on 02/15/2026)

    2026

  77. [85]

    https://www.langflow.org/

    Langflow: Low-code AI builder for agentic and RAG applications. https://www.langflow.org/. (Accessed on 02/15/2026)

    2026

  78. [86]

    Auto- gen: Enabling next-gen llm applications via multi-agent conversation framework.arXiv preprint arXiv:2308.08155, 3(4), 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Auto- gen: Enabling next-gen llm applications via multi-agent conversation framework.arXiv preprint arXiv:2308.08155, 3(4), 2023

    2023 arXiv

  79. [87]

    Dspy: Compiling declarative 17 language model calls into state-of-the-art pipelines

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. Dspy: Compiling declarative 17 language model calls into state-of-the-art pipelines. InThe Twelfth International ...

    2024

  80. [88]

    A System for Microserving of LLMs.arXiv preprint arXiv:2412.12488, 2024

    Hongyi Jin, Ruihang Lai, Charlie F Ruan, Yingcheng Wang, Todd C Mowry, Xupeng Miao, Zhihao Jia, and Tianqi Chen. A System for Microserving of LLMs.arXiv preprint arXiv:2412.12488, 2024. 18 Algorithm 1:Router Request Rebalancing Policy. 1For Each Rebalance Tick: 2foreachinstanc...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.