Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Under a fixed budget and fluctuating GPU availability, a mixed GPU fleet chosen by a joint MILP planner serves LLM requests up to 41% faster than tuned homogeneous clusters.

desk verdict Solid MILP-based planning for heterogeneous GPU fleets, but the headline 20–41% gains may be computed from the optimizer's own profile rather than measured end-to-end, so the gains are conditional. read the letter →

arxiv 2502.00722 v2 pith:UL66UQ3U submitted 2025-02-02 cs.DC

Youhe Jiang , Fangcheng Fu , Xiaozhe Yao , Guoliang He , Xupeng Miao , Ana Klimovic , Bin Cui , Binhang Yuan
show 1 more author
Eiko Yoneki
This is my paper · ORCID
classification cs.DC
keywords LLMservingheterogeneousGPUscostefficiencymixed-integerlinearprogrammingGPUschedulingworkloadheterogeneitycloudavailabilitymulti-model
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Under a fixed hourly budget and fluctuating cloud GPU availability, the paper argues that a deliberately mixed fleet of GPUs serves LLM requests more cheaply than any tuned fleet of one GPU type. It supports this with benchmarks on two model sizes, nine workload shapes, and six GPU classes, and with a mixed-integer linear program that chooses which GPUs to rent, how to configure each replica's parallelism, and which replicas should handle which request types. Across three real-world traces and several budgets, the method reports up to 41% higher throughput and up to 54% lower percentile latency than homogeneous baselines optimized with the same scheduler, with average gains around 20%. A reader should care because GPU rental is both costly and supply-constrained; if heterogeneity can be exploited as claimed, the same money serves more users or the same users at lower latency.

What carries the argument

The load-bearing object is a mixed-integer linear program over a precomputed configuration catalog. Each configuration $c$ is a model replica described by which GPU types it uses, its data/tensor/pipeline parallelism degrees, its hourly cost, and a profiled throughput $h_{c,w}$ for each workload type $w$; the decision variables count how many copies $y_c$ of each configuration to activate and what fraction $x_{c,w}$ of each workload to route to it. The solver minimizes the makespan $T = \max_c \sum_w x_{c,w} f_w / (y_c h_{c,w})$ subject to assignment, budget, and availability constraints, so the same plan picks the GPU fleet, the deployment layouts, and the request routing at once.

What would settle it

Deploy the solver's plan for, say, Trace 1 at a $30/h budget on the same cloud availability, feed it live mixed traffic with burst arrivals, and compare the measured end-to-end throughput and latency with the values predicted from the profiling table $h_{c,w}$. If per-configuration throughput under mixed workload batching deviates by much more than the 4–7% errors reported in Table 6, or if the measured end-to-end gain over the best homogeneous baseline falls well below the claimed 20%, the static-profile premise is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that cost-efficient LLM serving is a joint selection problem: the GPU composition, the per-replica deployment configuration, and the assignment of heterogeneous workloads must be optimized together, because different GPU types match different phases of inference. Prefill is compute-bound and decoding is memory-bound, so data-center GPUs earn their price on long-prompt compute-heavy work while workstation and consumer GPUs are more cost-effective on memory-heavy or small-model workloads. On the paper's benchmarks the joint optimization reduces makespan step by step: choosing the composition under a fixed budget gives up to a 20% speedup in a toy case, choosing parallelism gives another 14%, and workload-aware routing adds about 8%. The full system formulates the combined decision as a mixed-integer linear program that minimizes the makespan subject to a price budget and real-time GPU availability, extends the same formulation to multiple models, and couples it with binary search and knapsack-style feasibility checks for scalability.

Load-bearing premise

The entire plan and its reported gains rest on the one-time profile that assigns each configuration a fixed throughput per workload type measured in isolation; if mixed traffic, batching, or replica interference makes real throughput deviate from that table, the planned gains may not materialize in deployment.

Editorial extensions

If this is right

  • For a fixed budget, a heterogeneous plan can serve substantially more requests than the best homogeneous cluster, with reported gains up to 41% in throughput and 54% in latency percentiles.
  • Scheduling can absorb real-time availability: with four sampled availabilities, the method still beats homogeneous baselines, meaning the optimizer adapts when preferred GPUs are out of stock.
  • Replanning after workload shifts or GPU drops recovers throughput: in the paper's tests, a workload surge without replanning loses 13% and a four-GPU drop loses 29%, while replanning regains 25% and 10% respectively.
  • The same MILP covers multi-model serving; for a mix of 8B and 70B models it reports up to 35% higher throughput than homogeneous baselines.
  • The gap between heterogeneous and homogeneous plans narrows as budget grows (roughly 30% at $5/h to 15% at $60/h), so heterogeneity matters most when resources are scarce.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the static profiling table drifts under mixed online traffic, the planner could be re-run periodically (the paper itself sketches replanning); a live-system study that feeds measured throughput back into $h_{c,w}$ would test whether the reported gains survive production dynamics.
  • The hardware-level story suggests an unexploited lever: because prefill and decoding have opposite bottlenecks, phase-disaggregated serving over the same heterogeneous pool might further separate compute-optimized and memory-optimized GPUs, pushing cost-efficiency beyond what a per-replica configuration catalog captures.
  • The 4–7% profiling error in the paper's estimator table is small relative to the reported 20% average gain, so even a conservative planner should retain most of the benefit; quantifying how gain decays as estimation error grows would give operators a tuning knob.
  • The method treats workloads as fixed categories; an online extension that clusters live request-length distributions before each planning window would let the same MILP track non-stationary traffic without manual trace labeling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies cost-efficient LLM serving over heterogeneous GPU clouds. It contributes a benchmarking study of six GPU types, two Llama models, and nine workload types across deployment configurations, and uses the resulting observations to motivate a mixed-integer linear programming (MILP) scheduler that jointly optimizes GPU composition, deployment configuration, and workload assignment under a price budget and real-time GPU availability constraints. The authors report that their approach improves end-to-end throughput by up to 41% (average 20%) and reduces latency by up to 54% (average 20%) compared with tuned homogeneous baselines, and they compare against HexGen and Helix, provide an ablation study, a binary-search acceleration, and a multi-model extension.

Significance. If the reported gains are validated by real deployment, the paper makes a practically useful point: under budget and availability constraints, a deliberately chosen heterogeneous GPU pool can be more cost-efficient than the best homogeneous cluster for the same money. The benchmarking data, the decomposition into GPU composition, deployment configuration, and workload assignment, and the explicit treatment of budget and availability constraints are valuable contributions. The ablation study showing that each of the three factors contributes is also a strength. However, the central validation is currently incomplete, and the MILP formulation as written contains a nonlinearity, so the significance can only be assessed after these issues are resolved.

major comments (3)
  1. [§5.2, Appendix L, Table 6] The main end-to-end claims (up to 41% throughput gain and up to 54% latency reduction) are not yet established because the paper does not state whether the results in Figures 5-7 and 10 were obtained by actually deploying the chosen plans on rented GPUs with vLLM or by computing them from the same profiled throughput table h_{c,w} used by the MILP. Section 5.1 says GPUs are rented from Vast.ai and that all experiments use vLLM, but no measurement protocol for the end-to-end experiments is given. Table 6 validates the estimator on only one workload shape (long-input/short-output) with 4-7% throughput errors and no latency validation. Please state explicitly how the end-to-end numbers were produced, and provide deployment-measured throughput and latency for at least the headline configurations and baselines.
  2. [§4.3, Eq. (3)] Constraint (3), sum_w (x_{c,w} f_w) / (y_c h_{c,w}) <= T, is not a linear constraint: y_c is an integer decision variable and appears in the denominator. A reformulation that multiplies through by y_c gives y_c h_{c,w} T on the right-hand side, introducing the bilinear product y_c * T. As written, the program is therefore not a mixed-integer linear program. The authors should either provide a valid linearization (e.g., with auxiliary variables and big-M constraints) or clarify how the nonlinear terms are handled by the solver. This is load-bearing because the entire scheduling algorithm and all downstream comparisons rely on this formulation.
  3. [§4.3, Appendix L] The model assumes that each configuration c has a static per-replica throughput h_{c,w} per workload type, measured in isolation, and that the time to process a workload fraction is x f / (y h), which implies linear scaling in the number of replicas and no interference between workloads, no batching dynamics, no prefill-decode interference, no replica-level queueing, and no KV-cache fragmentation. Under mixed online traffic these effects can cause actual throughput and latency to deviate nonlinearly from the profile. Since the only validation in Table 6 covers a single workload shape and reports no latency errors, the claimed 20-41% throughput and 20-54% latency advantages may not transfer to real deployment. Please add a sensitivity analysis over profile errors and deployment-measured validation on mixed workloads, including latency percentiles.
minor comments (5)
  1. [§5.1] There is a typo: 'vairous percentile latencies' should be 'various percentile latencies'.
  2. [Appendix F] The feasibility-check condition in Appendix F, sum_w x_{c,w} / h_{c,w} <= \hat{T}, omits the request counts f_w and the replica counts y_c that appear in Eq. (3); please align the notation with the MILP formulation.
  3. [Figure 6 and Appendix M] The text in Appendix M claims the method achieves the lowest P99 latency, but Figure 6 reports P10-P100 latency curves and Section 5.1 defines P90 latency; please clarify whether P99 was measured or is a typo.
  4. [Table 6] The rows labeled 'H100 (4,2) (cross machine)' and 'L40 (4,2) (cross machine)' appear to use tensor parallelism across machines, which contradicts the heuristic in Appendix D that TP is only used within a single machine; please clarify how these configurations are generated and why they are included.
  5. [References] The reference to 'JIANG et al.' uses inconsistent capitalization and is missing the first-name initials; please normalize the citation style.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains come from measured vLLM deployments, and the profiled throughput constants serve as optimizer inputs, not as the source of the end-to-end numbers.

full rationale

The paper's derivation chain is self-contained. The MILP in Section 4.3 takes per-configuration throughput h_{c,w} from one-time profiling as input and minimizes makespan; this is a standard optimization use of measured constants. The end-to-end results in Section 5.2 are presented as experiments run with vLLM on rented GPUs from Vast.ai, with homogeneous baselines also tuned by the same scheduling algorithm, so the claimed improvements are not computed from the optimizer's own objective by construction. The only validation table for the profiled estimator, Table 6, compares estimated throughput against real measured throughput for representative configurations and reports 4-7% errors, providing independent grounding rather than circularity. No load-bearing step reduces to a fitted parameter renamed as a prediction, and the self-citations to HexGen appear only as a comparison baseline, not as justification for the paper's central premise. The text does not state that the end-to-end throughput or latency numbers are derived from the h_{c,w} table, so no specific reduction can be quoted. The paper may face correctness risks about profile transferability, but that is a measurement-validity concern, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's contribution rests on a profiled performance table, an offline makespan model, and market price inputs. The profiled h_{c,w} values are the largest de facto free parameters, since the paper does not publish them and uses them both for optimization and for reporting results.

free parameters (3)
  • Per-configuration throughput h_{c,w} = Not published; profiled per GPU type, parallelism strategy, and workload type (Appendix L)
    The MILP uses these numbers as constants for both planning and, apparently, evaluating the plan; the reported gains depend on their accuracy.
  • Workload long/short thresholds = 512 input tokens, 128 output tokens
    Used to categorize traces into four request classes in Figure 1 and later to define nine workload types by average token lengths; the thresholds are chosen by hand and shape all benchmark results.
  • Binary search tolerance tau = 1 second (Appendix F)
    Controls how precisely the approximate makespan is returned; chosen by hand.
assumptions (5)
  • domain assumption Hourly GPU rental prices from Table 1 are fixed and known in advance.
    Entered in Section 3 and the MILP budget constraint (5); actual cloud prices fluctuate, which the paper acknowledges only informally for availability, not price.
  • domain assumption Each configuration c has a static throughput h_{c,w} independent of load, batch composition, and other replicas.
    Entered in Section 4.3 (equation 3); this is the central modeling assumption that the MILP and the reported results rely on.
  • domain assumption The offline makespan objective over fixed request counts f_w is a valid proxy for online serving of dynamic traces.
    The paper states in Appendix M that online replanning is orthogonal; the main evaluation uses static workload fractions.
  • domain assumption Prefill is compute-bound and decoding is memory-bound, so profiled single-layer latencies compose linearly into full-model throughput.
    Entered in Appendix L, based on prior work (DistServe, Splitwise); Table 6 checks only one workload type.
  • standard math MILP branch-and-bound solves the formulated program correctly.
    Standard solver correctness, not questioned.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs." pith.science (2026). https://pith.science/paper/UL66UQ3U

@misc{pith2026250200722,
  author       = {Pith},
  title        = {Pith review of: Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UL66UQ3U}},
  note         = {Machine review of arXiv:2502.00722}
}
read the original abstract

Recent advancements in Large Language Models (LLMs) have led to increasingly diverse requests, accompanied with varying resource (compute and memory) demands to serve them. However, this in turn degrades the cost-efficiency of LLM serving as common practices primarily rely on homogeneous GPU resources. In response to this problem, this work conducts a thorough study about serving LLMs over heterogeneous GPU resources on cloud platforms. The rationale is that different GPU types exhibit distinct compute and memory characteristics, aligning well with the divergent resource demands of diverse requests. Particularly, through comprehensive benchmarking, we discover that the cost-efficiency of LLM serving can be substantially optimized by meticulously determining GPU composition, deployment configurations, and workload assignments. Subsequently, we design a scheduling algorithm via mixed-integer linear programming, aiming at deducing the most cost-efficient serving plan under the constraints of price budget and real-time GPU availability. Remarkably, our approach effectively outperforms homogeneous and heterogeneous baselines under a wide array of scenarios, covering diverse workload traces, varying GPU availablilities, and multi-model serving. This casts new light on more accessible and efficient LLM serving over heterogeneous cloud resources.

Figures

Figures reproduced from arXiv: 2502.00722 by the authors.

Figure 1
Figure 1. The real-world workload traces from the Swiss [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The number of different types of GPUs on Vast.ai [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Benchmarked results for Llama3-70B model with different GPU types on different workload types. The left [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Throughput results for Llama3-70B model with different deployment configurations on different workloads. The [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: End-to-end throughput results on Llama3-70B [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: End-to-end latency results on Llama3-70B model [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Ours vs. HexGen. The first and second bars in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Ablation study of Llama3-70B on traces 1 and 2. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 10
Figure 10. Figure 10: End-to-end experiments on multiple model types [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Benchmarked results for Llama3-8B model with different GPU types on different workload types. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Throughput and latency results for Llama3-70B model with different deployment configurations on different [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Throughput and latency results for Llama3-70B model with different deployment configurations on different [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Illustration of a simple example. Algorithm 1 Binary Search on T Input: T, T {initial bounds} Input: τ {tolerance} Output: Approximate minimal feasible makespan while (T − T) > τ do Tˆ ← T +T 2 if FEASIBILITYCHECK(Tˆ) is true then T ← Tˆ {If feasible, try smaller Tˆ} …
Figure 15
Figure 15. Figure 15: End-to-end experiments on Llama3-8B model with different setups. 5 10 15 20 25 30 35 40 45 50 55 60 Price Budgets ($/h) 0 150 300 450 600 750 Makespan (s) Ours (Avail 1) RTX 4090 (Homo) RTX 6000 (Homo) H100 (Homo) [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: System performance v.s. price budget. demonstrate the efficiency, effectiveness, and scalability of our scheduling algorithm. G. Other Optimizations for Speeding up MILP For large numbers of GPUs and model types, it might take hours for the MILP solver to provide a re…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters

    cs.DC 2025-07 conditional novelty 6.0 of 10

    Zorse integrates interleaved pipeline parallelism, ZeRO-2 data parallelism, and CPU offloading to accelerate LLM training on heterogeneous GPU clusters by up to 4x.

Reference graph

Works this paper leans on

53 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    S., Ramjee, R., and Tumanov, A

    Agrawal, A., Kedia, N., Mohan, J., Panwar, A., Kwatra, N., Gulavani, B. S., Ramjee, R., and Tumanov, A. Vidur: A large-scale simulation framework for llm inference. Proceedings of Machine Learning and Systems, 6: 0 351--366, 2024 a

    work page 2024

  2. [2]

    Taming \ Throughput-Latency \ tradeoff in \ LLM \ inference with \ Sarathi-Serve \

    Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B., Tumanov, A., and Ramjee, R. Taming \ Throughput-Latency \ tradeoff in \ LLM \ inference with \ Sarathi-Serve \ . In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp.\ 117--134, 2024 b

    work page 2024

  3. [3]

    The claude 3 model family: Opus, sonnet, haiku, 2024

    Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf

    2024

  4. [4]

    Azure public dataset, 2024

    Azure. Azure public dataset, 2024. URL https://github.com/Azure/AzurePublicDataset

    work page 2024

  5. [5]

    Petals: Collaborative inference and fine-tuning of large models

    Borzunov, A., Baranchuk, D., Dettmers, T., Riabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., and Raffel, C. Petals: Collaborative inference and fine-tuning of large models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp.\ 558--568, 2023

    work page 2023

  6. [6]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  7. [7]

    The world’s most widely adopted ai developer tool, 2024

    GitHub. The world’s most widely adopted ai developer tool, 2024. URL https://github.com/features/copilot

    work page 2024

  8. [8]

    M 'elange: Cost efficient large language model serving by exploiting gpu heterogeneity

    Griggs, T., Liu, X., Yu, J., Kim, D., Chiang, W.-L., Cheung, A., and Stoica, I. M 'elange: Cost efficient large language model serving by exploiting gpu heterogeneity. arXiv preprint arXiv:2404.14527, 2024

    arXiv 2024

Show all 53 references
  1. [9]

    and Yoneki, E

    He, G. and Yoneki, E. Cuasmrl: Optimizing gpu sass schedules via deep reinforcement learning. In Proceedings of the 23rd ACM/IEEE International Symposium on Code Generation and Optimization, CGO '25, pp.\ 493–506, New York, NY, USA, 2025. Association for Computing Machinery. I...

    2025

  2. [10]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations

  3. [11]

    Inference without interference: Disaggregate llm inference for mixed downstream workloads

    Hu, C., Huang, H., Xu, L., Chen, X., Xu, J., Chen, S., Feng, H., Wang, C., Wang, S., Bao, Y., et al. Inference without interference: Disaggregate llm inference for mixed downstream workloads. arXiv preprint arXiv:2401.11181, 2024

    2024 arXiv

  4. [12]

    V., Wu, Y., et al

    Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

    2019

  5. [13]

    and Lee, S

    Jeon, J. and Lee, S. Large language models in education: A focus on the complementary relationship between human teachers and chatgpt. Education and Information Technologies, 28 0 (12): 0 15873--15892, 2023

    2023

  6. [14]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  7. [15]

    Hexgen: Generative inference of large language model over heterogeneous environment

    JIANG, Y., Yan, R., Yao, X., Zhou, Y., Chen, B., and Yuan, B. Hexgen: Generative inference of large language model over heterogeneous environment. In Forty-first International Conference on Machine Learning

  8. [16]

    Thunderserve: High-performance and cost-efficient llm serving in cloud environments

    Jiang, Y., Fu, F., Yao, X., Wang, T., Cui, B., Klimovic, A., and Yoneki, E. Thunderserve: High-performance and cost-efficient llm serving in cloud environments. arXiv preprint arXiv:2502.09334, 2025 a

    2025

  9. [17]

    Hexgen-2: Disaggregated generative inference of llms in heterogeneous environment

    Jiang, Y., Yan, R., and Yuan, B. Hexgen-2: Disaggregated generative inference of llms in heterogeneous environment. arXiv preprint arXiv:2502.07903, 2025 b

    2025 arXiv

  10. [18]

    H., Gonzalez, J., Zhang, H., and Stoica, I

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp.\ 611--626, 2023

    2023

  11. [19]

    Hetu v2: A general and scalable deep learning system with hierarchical and heterogeneous single program multiple data annotations

    Li, H., Fu, F., Ge, H., Lin, S., Wang, X., Niu, J., Miao, X., and Cui, B. Hetu v2: A general and scalable deep learning system with hierarchical and heterogeneous single program multiple data annotations. arXiv preprint arXiv:2504.20490, 2025

    2025 arXiv

  12. [20]

    E., et al

    Li, Z., Zheng, L., Zhong, Y., Liu, V., Sheng, Y., Jin, X., Huang, Y., Chen, Z., Zhang, H., Gonzalez, J. E., et al. \ AlpaServe \ : Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation...

    2023

  13. [21]

    Deepseek-v3 technical report

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024 a

    2024 arXiv

  14. [22]

    Understanding llms: A comprehensive overview from training to inference

    Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. Understanding llms: A comprehensive overview from training to inference. arXiv preprint arXiv:2401.02038, 2024 b

    2024 arXiv

  15. [23]

    Deja vu: Contextual sparsity for efficient llms at inference time

    Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., et al. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning, pp.\ 22137--22176. PMLR, 2023

    2023

  16. [24]

    Helix: Distributed serving of large language models via max-flow on heterogeneous gpus

    Mei, Y., Zhuang, Y., Miao, X., Yang, J., Jia, Z., and Vinayak, R. Helix: Distributed serving of large language models via max-flow on heterogeneous gpus. arXiv preprint arXiv:2406.01566, 2024

    2024 arXiv

  17. [25]

    Spotserve: Serving generative large language models on preemptible instances

    Miao, X., Shi, C., Duan, J., Xi, X., Lin, D., Cui, B., and Jia, Z. Spotserve: Serving generative large language models on preemptible instances. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vo...

    2024

  18. [26]

    Exegpt: Constraint-aware resource scheduling for llm inference

    Oh, H., Kim, K., Kim, J., Kim, S., Lee, J., Chang, D.-s., and Seo, J. Exegpt: Constraint-aware resource scheduling for llm inference. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pp....

    2024

  19. [27]

    Openai gpt-4o, 2024

    OpenAI. Openai gpt-4o, 2024. URL https://platform.openai.com/docs/models/gpt-4o

    2024

  20. [28]

    Splitwise: Efficient generative llm inference using phase splitting

    Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, \'I ., Maleki, S., and Bianchini, R. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp.\ 118--132. IEEE, 2024

    2024

  21. [29]

    Queue management for slo-oriented large language model serving

    Patke, A., Reddy, D., Jha, S., Qiu, H., Pinto, C., Narayanaswami, C., Kalbarczyk, Z., and Iyer, R. Queue management for slo-oriented large language model serving. In Proceedings of the 2024 ACM Symposium on Cloud Computing, pp.\ 18--35, 2024

    2024

  22. [30]

    E., PourNejatian, N., Costa, A

    Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. A study of generative large language model for medical research and healthcare. NPJ digital medicine, 6 0 (1): 0 210, 2023

    2023

  23. [31]

    Hexgen-text2sql: Optimizing llm inference request scheduling for agentic text-to-sql workflow

    Peng, Y., Jiang, Y., Wang, C., and Yuan, B. Hexgen-text2sql: Optimizing llm inference request scheduling for agentic text-to-sql workflow. arXiv preprint arXiv:2505.05286, 2025

    2025

  24. [32]

    Conserve: Harvesting gpus for low-latency and high-throughput large language model serving

    Qiao, Y., Anzai, S., Yu, S., Ma, H., Wang, Y., Kim, M., and Xu, H. Conserve: Harvesting gpus for low-latency and high-throughput large language model serving. arXiv preprint arXiv:2410.01228, 2024

    2024 arXiv

  25. [33]

    Mooncake: Kimi's kvcache-centric architecture for llm serving

    Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: Kimi's kvcache-centric architecture for llm serving. arXiv preprint arXiv:2407.00079, 2024

    2024 arXiv

  26. [34]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  27. [35]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  28. [36]

    Dynamollm: Designing llm inference clusters for performance and energy efficiency

    Stojkovic, J., Zhang, C., Goiri, \'I ., Torrellas, J., and Choukse, E. Dynamollm: Designing llm inference clusters for performance and energy efficiency. arXiv preprint arXiv:2408.00741, 2024

    2024

  29. [37]

    Llumnix: Dynamic scheduling for large language model serving

    Sun, B., Huang, Z., Zhao, H., Xiao, W., Zhang, X., Li, Y., and Lin, W. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp.\ 173--191, 2024

    2024

  30. [39]

    C., and Chu, X

    Wang, Y., Chen, Y., Li, Z., Tang, Z., Guo, R., Wang, X., Wang, Q., Zhou, A. C., and Chu, X. Towards efficient and reliable llm serving: A real-world workload study. arXiv preprint arXiv:2401.17644, 2024 b

    2024 arXiv

  31. [40]

    Fast distributed inference serving for large language models

    Wu, B., Zhong, Y., Zhang, Z., Huang, G., Liu, X., and Jin, X. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920, 2023

    2023 arXiv

  32. [41]

    Flashflex: Accommodating large language model training over heterogeneous environment

    Yan, R., Jiang, Y., Tao, W., Nie, X., Cui, B., and Yuan, B. Flashflex: Accommodating large language model training over heterogeneous environment. arXiv preprint arXiv:2409.01143, 2024

    2024 arXiv

  33. [42]

    S., Kim, G.-W., Kim, S., and Chun, B.-G

    Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for \ Transformer-Based \ generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp.\ 521--538, 2022

    2022

  34. [43]

    Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization

    Zhang, J., Huang, H., Zhang, P., Wei, J., Zhu, J., and Chen, J. Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. In International Conference on Machine Learning (ICML), 2025 a

    2025

  35. [44]

    Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training

    Zhang, J., Wei, J., Zhang, P., Xu, X., Huang, H., Wang, H., Jiang, K., Zhu, J., and Chen, J. Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training. arXiv preprint arXiv:2505.11594, 2025 b

    2025

  36. [45]

    Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration

    Zhang, J., Wei, J., Zhang, P., Zhu, J., and Chen, J. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations (ICLR), 2025 c

    2025

  37. [46]

    Spargeattn: Accurate sparse attention accelerating any model inference

    Zhang, J., Xiang, C., Huang, H., Wei, J., Xi, H., Zhu, J., and Chen, J. Spargeattn: Accurate sparse attention accelerating any model inference. In International Conference on Machine Learning (ICML), 2025 d

    2025

  38. [47]

    Sageattention2++: A more efficient implementation of sageattention2

    Zhang, J., Xu, X., Wei, J., Huang, H., Zhang, P., Xiang, C., Zhu, J., and Chen, J. Sageattention2++: A more efficient implementation of sageattention2. arXiv preprint arXiv:2505.21136, 2025 e

    2025 arXiv

  39. [48]

    Llm-pq: Serving llm on heterogeneous clusters with phase-aware partition and adaptive quantization

    Zhao, J., Wan, B., Peng, Y., Lin, H., and Wu, C. Llm-pq: Serving llm on heterogeneous clusters with phase-aware partition and adaptive quantization. the 29th ACM SIGPLAN Annual Sympo-sium Principles and Practice of Parallel Programming (PPoPP’24)(02/03/2024-06/03/2024, Edinbur...

    2024

  40. [49]

    Wildchat: 1m chatgpt interaction logs in the wild

    Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: 1m chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations

  41. [50]

    Blendserve: Optimizing offline inference for auto-regressive large models with resource-aware batching

    Zhao, Y., Yang, S., Zhu, K., Zheng, L., Kasikci, B., Zhou, Y., Xing, J., and Stoica, I. Blendserve: Optimizing offline inference for auto-regressive large models with resource-aware batching. arXiv preprint arXiv:2411.16102, 2024 b

    2024 arXiv

  42. [51]

    Lmsys-chat-1m: A large-scale real-world llm conversation dataset

    Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E., et al. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. In The Twelfth International Conference on Learning Representations

  43. [52]

    \ DistServe \ : Disaggregating prefill and decoding for goodput-optimized large language model serving

    Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. \ DistServe \ : Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp.\ 193--210, 2024

    2024

  44. [53]

    \ PetS \ : A unified framework for \ Parameter-Efficient \ transformers serving

    Zhou, Z., Wei, X., Zhang, J., and Sun, G. \ PetS \ : A unified framework for \ Parameter-Efficient \ transformers serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pp.\ 489--504, 2022

    2022

  45. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.