Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One GPU can give LLM serving the latency isolation of disaggregation and the utilization of a monolithic engine — by splitting its compute cores between prefill and decode on the fly.

desk verdict A well-scoped systems paper with plausible large gains, but the cost model's calibration procedure is under-specified and the evaluation lacks independent validation, so the results need verification. read the letter →

arxiv 2507.06608 v5 pith:BP5OP5KM submitted 2025-07-09 cs.DC cs.LG

Xiaoxiang Shi , Colin Cai , Junjia Du , Zhihao Jia This is my paper
classification cs.DCcs.LG
keywords LLMservingprefill-decodedisaggregationintra-GPUresourcepartitioningdynamicSMallocationanalyticalcostmodeltime-to-first-tokentime-between-tokenschunkedprefillinterference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the two phases of LLM inference — compute-bound prefill and memory-bound decode — can be disaggregated inside a single GPU by dynamically partitioning the GPU's streaming multiprocessors (its kernel-executing compute units), without the hardware cost of engine-level disaggregation. It reports that both stages show diminishing returns beyond a saturation point, so each phase needs only the compute it actually uses, and that memory-bandwidth contention, not compute, is the main residual interference. Nexus predicts per-operator latency with an analytic cost model, picks SM partitions with a fast greedy search, and runs phase-specific schedulers. On Qwen and LLaMA models, the paper reports up to 2.2x higher throughput, 20x lower time-to-first-token, and 2.5x lower time-between-tokens than vLLM, while matching or exceeding two-GPU disaggregated serving.

What carries the argument

The load-bearing mechanism is the runtime SM partition chosen by a greedy search over an analytic cost model. The model sums per-operator latencies, each the max of a compute term and a memory term; compute latency follows a two-regime saturation-decay curve, $T^{\mathrm{compute}}_o(c_o,r) = c_o/(rC)$ below a saturation ratio $R_{\mathrm{sat}}$ and $c_o/(R_{\mathrm{sat}}C)\cdot(1+\lambda(r-R_{\mathrm{sat}}))$ above it, with $R_{\mathrm{sat}}$ and $\lambda$ calibrated by offline profiling per model and workload. Decode memory latency is modeled by estimating the probability that decode overlaps prefill attention and then dividing decode's memory bytes by the bandwidth share it gets under assumed full saturation. The controller switches between decode-prioritized and prefill-prioritized objectives based on live KV-cache usage, the greedy search converges in 2–4 cost-model evaluations, and a hysteresis buffer filters out SM-ratio changes smaller than $\delta$.

What would settle it

Run the Long Data Collections workload on one L20 GPU with Qwen2.5-3B, log the SM ratio the controller chooses each iteration, and compare the cost model's predicted prefill and decode latencies with the measured latencies under concurrent execution; a systematic gap — especially decode TBT rising faster than predicted as prefill KV length grows — would show the offline-calibrated model fails to transfer to runtime conditions.

Watch

Extended reading notes

Core claim

Nexus's central claim is that intra-engine prefill-decode disaggregation, backed by a proactive cost model, achieves what monolithic and disaggregated designs each achieve only partially: monolithic chunked prefilling mixes compute-heavy prefill and memory-bound decode in one batch, causing fine-grained phase interference that inflates time-between-tokens, while engine-level disaggregation removes interference but wastes hardware and adds coordination overhead. Nexus instead runs prefill and decode as separate concurrent streams on one GPU, with a runtime controller that splits streaming multiprocessors between them. The split is chosen by a lightweight cost model that encodes two measured effects — per-operator latency saturating as SM ratio grows, and decode latency rising with prefill memory traffic — and by a greedy search that solves a dual-objective problem: prioritize prefill when KV-cache usage is low and decode when it is high, keeping the other phase within a slowdown budget. With phase-specific scheduling (shortest-prompt-first for prefill, FCFS for decode) and a hysteresis buffer that suppresses flapping, the paper claims Nexus matches the latency isolation of disaggregated systems and the utilization of monolithic systems on a single commodity GPU.

Load-bearing premise

The cost model's saturation and decay parameters are extracted from offline profiling for each model and workload configuration, and the memory-contention equations assume full bandwidth saturation; if those calibrated numbers do not match the latencies actually observed under concurrent prefill-decode execution, the chosen SM partitions will be off and the claimed gains will shrink.

Editorial extensions

If this is right

  • If the cost model holds under concurrent execution, one GPU can deliver disaggregation-level time-to-first-token and time-between-tokens without a second engine, cutting hardware cost for latency-sensitive LLM serving.
  • Chunked prefill's interference is not inherent: separating phases into concurrent streams and giving decode enough SMs removes the fine-grained TBT inflation measured in mixed batches.
  • Phase-specific scheduling becomes practical once phases are decoupled: shortest-prompt-first prefill cuts TTFT substantially, while the decode scheduler can remain simple FCFS.
  • Dynamic SM reallocation can respond to workload shifts on sub-second timescales, since the greedy search needs only a few closed-form cost-model evaluations.
  • The same mechanism works across multiple GPUs: on two GPUs, the paper reports 2.2x throughput over vLLM and 2x over SGLang with better TTFT and TBT for a 14B model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The offline-calibrated cost model suggests a natural extension: online calibration, in which measured iteration latencies continuously update $R_{\mathrm{sat}}$ and $\lambda$, would make the proactive controller adaptive without per-workload profiling.
  • The diminishing-returns result is more general than prefill/decode: any GPU co-tenant pair where one phase is compute-bound and the other memory-bound could use the same saturation-aware SM split.
  • Because the contention model assumes full bandwidth saturation, partition choices under low memory pressure may be conservative; a traffic-based contention model could shift more SMs to decode when actual bandwidth use is low.
  • The paper evaluates on three workloads; a stress test with rapidly switching prompt-length distributions would reveal whether the controller's KV-cache-triggered mode switching responds quickly enough under adversarial mixes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Nexus, a vLLM-based LLM serving engine that splits a single GPU's SMs between prefill and decode phases, executes the two phases concurrently on separate CUDA streams, and repartitions SMs dynamically using a two-regime saturation cost model (Eq. 7) with a memory-bandwidth contention term (Eqs. 8–9), a greedy partition search (Algorithm 1), and phase-specific SPF/FCFS schedulers. The paper claims up to 2.2x higher throughput, 20x lower TTFT, and 2.5x lower TBT than vLLM, up to 2x improvements over SGLang, and competitive or better performance than a two-GPU disaggregated vLLM. Evaluation covers three workloads, Qwen2.5-3B and LLaMA3.1-8B on one GPU and Qwen2.5-14B on two GPUs, plus ablations, offline makespan, and a latency breakdown.

Significance. If the cost-model validation gap is closed, this is a meaningful contribution: it offers a plausible path to prefill-decode isolation without extra GPUs, which is relevant to cost-sensitive LLM serving. The implementation is substantial (roughly 6K lines plus a CUDA extension), and the evaluation is broad in terms of workloads, model sizes, and single/dual-GPU configurations. The ablation study is well designed and supports the qualitative claim that both dynamic SM switching and SPF scheduling contribute to the end-to-end results. The paper also reports useful secondary evidence in the form of offline makespan and overhead breakdown. However, the central 'proactive' claim rests on a cost model whose calibration procedure is under-specified and not validated against concurrent execution, which is a load-bearing weakness.

major comments (4)
  1. [§4.1.1, Eq. (7), §5] The paper states that R_sat and λ are extracted 'per operator' from 'end-to-end measurements of the full stage (prefill or decode) under varying SM allocations,' but a single stage-level latency curve T_stage(r) = Σ_i T_i(r) cannot uniquely determine the per-operator parameters (R_sat,i, λ_i). Figure 5 shows that per-kernel breakdowns were measured, so a per-operator fit is plausible, but the text never describes the fitting procedure, including whether per-kernel timings were collected and what objective/constraints were used. Since Algorithm 1 and the 'proactive' adaptation claims depend on this model, the paper should either specify the per-operator fitting method or explicitly downgrade the model to stage-level, and should include a predicted-versus-measured validation for concurrent prefill/decode execution.
  2. [§5 vs. §7] Section 5 says λ is profiled 'for each model and workload configuration,' while the Related Work section asserts that Nexus 'uses a one-time profiling pass... avoiding offline workload tracing.' These statements contradict each other. If parameters are refit per workload distribution and the same three workload distributions from Table 1 are then used in Section 6, the reported gains may reflect calibration on the evaluation traces rather than proactive adaptation to unseen conditions. Please clarify precisely which quantities are workload-dependent in the profile and demonstrate generalization, for example by profiling on one workload and evaluating on a held-out workload, or by showing that the fitted parameters are stable across workload distributions.
  3. [§4.1.1, Eqs. (8)–(9)] The memory-contention model assumes full bandwidth saturation during each overlap window and that bandwidth splits between prefill and decode in proportion to their memory traffic. Section 3.3 demonstrates qualitatively that prefill KV length affects decode latency, but no experiment validates the quantitative predictions of Eq. (9) under concurrent execution. Because the greedy controller adjusts SM ratios based on this model, the absence of a predicted-versus-measured comparison makes the causal attribution of the end-to-end gains to the cost model unsupported. Please add a validation experiment that varies prefill load, decode batch size, and SM split, and report the model's prediction error.
  4. [§6.1, Figures 9–10] The evaluation reports no error bars, run counts, or confidence intervals. The headline numbers (up to 20x lower TTFT, 2.2x throughput, 2.5x lower TBT) are point estimates; without knowing the number of runs and the run-to-run variance, it is difficult to judge whether the improvements are robust. Please report the number of independent runs and either standard deviations or confidence intervals for the key metrics, at least for the largest claimed gains.
minor comments (6)
  1. [§6.1] FastServe is a reimplementation; please report how parity with the original system was validated (for example, matching published throughput or latency numbers) or soften the conclusions drawn from this baseline.
  2. [Figure 9 caption] The third row of Figure 9 says 'Llama-3-1.8B', but Section 6.1 and the text use 'LLaMA3.1-8B'; the figure should be corrected.
  3. [§4.2] The phrase 'Green Contexts provide logical SM isolation' should refer to 'CUDA Green Context' for consistency with Section 5; the plural form is confusing.
  4. [§6.5 and elsewhere] There are several typos that should be fixed, including 'bottneleck' (§6.5), 'Particularily' and 'employee' (§4.3), 'FIgure' (§6.2.2), 'repsectively' (§6.3), and 'acheives' (§6.3).
  5. [§5] The sentence 'The maximum batch size and chunk size for prefill of Nexus are same as those of vLLM..' contains a doubled period and should be rephrased.
  6. [§3.2–3.3] The motivation experiments use NVIDIA MPS to control SM partitioning, while the implemented system uses CUDA Green Context; the paper should state whether the two mechanisms have equivalent partitioning granularity and runtime overhead, since the diminishing-returns evidence is collected with MPS.

Circularity Check

1 steps flagged · score 6.0 of 10

Cost-model latency 'predictions' are fitted to the same model/workload configurations used in evaluation, partially circular.

  1. fitted input called prediction [Section 4.1.1 (Eq. 7); Section 5 Implementation]
    "“We extract Rsat and λ per operator from end-to-end measurements of the full stage (prefill or decode) under varying SM allocations.” “The decaying λ for each operator in cost model (§4.1) is obtained by profiling prefill and decode offline, and is done for each model and workload configuration.”"

    Eqs. 5–7 define the cost model's predicted stage latency as a sum of per-operator saturation curves, whose free parameters Rsat and λ are extracted from end-to-end measurements of the same full stage under varying SM allocations. Therefore the 'prediction' for any SM split is an interpolation of the measured stage-latency curve, not an independent estimate. Section 5 states calibration is done per model and per workload configuration, and Section 6 evaluates on exactly those configurations. The greedy SM search (Algorithm 1) optimizes against this fitted curve, so the controller's partition choices are tailored to the evaluation distribution.

full rationale

Nexus's headline throughput/latency results are measured end-to-end against external baselines (vLLM, SGLang, FastServe, vLLM-P/D), so those comparisons are not circular. The central design claim, however, is that a lightweight analytical cost model 'predicts latency under any SM split' and enables proactive adaptation. Section 4.1.1 reveals that the model's Rsat/λ parameters are extracted from end-to-end measurements of the full prefill/decode stage under varying SM allocations, and Section 5 says this profiling is done for each model and workload configuration—the same configurations used in the Section 6 evaluation. Thus the 'prediction' used by Algorithm 1's greedy search is a fit to the measured latency-vs-SM curve for that workload, not an out-of-sample prediction. The paper also never validates predicted latencies against measured concurrent prefill/decode latencies, so the cost model's accuracy is unverified. Related Work's claim of a 'one-time profiling pass' that 'avoids offline workload tracing' is in tension with the per-workload calibration stated in Section 5. No load-bearing self-citation or imported uniqueness theorem appears; citations to the authors' prior work are contextual. Overall, one load-bearing prediction reduces to calibration data, but because the final performance numbers are real measurements against external systems, this is partial circularity rather than a fully forced derivation.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a cost model with profiled parameters, hardware isolation assumptions, and a bandwidth contention model, none of which are validated against ground-truth runtime measurements. No new physical entities are introduced.

free parameters (7)
  • per-operator saturation threshold Rsat = not reported (profiled offline)
    Eq. 7 models sub-saturation vs post-saturation latency; Rsat is extracted per operator from end-to-end stage measurements under varying SM allocations (Section 5).
  • per-operator decay coefficient lambda = not reported (profiled offline)
    Eq. 7 post-saturation decay; obtained by profiling prefill and decode offline for each model and workload configuration (Section 5).
  • SPF anti-starvation gamma = 15 (default)
    Eq. 10 score = l_i - gamma*(t - a_i); gamma set to 15 by default (Section 5).
  • prefill slack alpha = 1.3
    Slowdown budget for prefill in decode-prioritized mode; Section 5 states alpha=1.3.
  • decode slack beta = 1.1
    Slowdown budget for decode in prefill-prioritized mode; Section 5 states beta=1.1.
  • KV_switch threshold = 70% of available KV cache
    Switch between prefill-prioritized and decode-prioritized modes based on live KV cache usage; Section 5.
  • hysteresis buffer delta = not reported
    Algorithm 1 line 9 suppresses SM ratio changes smaller than delta; delta value is not specified.
assumptions (5)
  • ad hoc to paper GPU execution can be modeled by per-operator saturation-decay latency curves that are stable across runtime conditions.
    Eq. 7 assumes a two-regime curve; parameters are fit offline and assumed to hold during online operation (Sections 4.1.1, 5).
  • domain assumption CUDA Green Contexts provide logical SM isolation with no interference beyond memory bandwidth.
    The design relies on Green Context to partition SMs (Section 5), and the cost model attributes cross-phase interference solely to memory bandwidth.
  • ad hoc to paper Memory bandwidth is fully saturated during overlap windows and splits between phases in proportion to their memory traffic.
    Eq. 8-9 compute effective decode bandwidth assuming full saturation and traffic-proportional sharing; this is a conservative modeling assumption stated in Section 4.1.1.
  • ad hoc to paper Offline-profiled cost model parameters generalize to the dynamic workloads used in evaluation.
    The evaluation workloads match the model and workload configurations used for profiling (Sections 5, 6.1), so generalization outside the calibration distribution is not demonstrated.
  • domain assumption Workload arrival is Poisson and the three benchmark datasets are representative.
    Section 6.1 generates request arrivals with a Poisson process, following prior work; representativeness is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving." pith.science (2026). https://pith.science/paper/BP5OP5KM

@misc{pith2026250706608,
  author       = {Pith},
  title        = {Pith review of: Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BP5OP5KM}},
  note         = {Machine review of arXiv:2507.06608}
}
read the original abstract

Monolithic serving with chunked prefill improves GPU utilization by batching prefill and decode together, but suffers from fine-grained phase interference. Engine-level prefill-decode (PD) disaggregation avoids interference but incurs higher hardware and coordination overhead. Prior intra-GPU disaggregation approaches multiplex prefill and decode within a single GPU, using SLO-based tuning guided by heuristics from offline profiling or reactive feedback loops. However, these methods respond reactively to performance issues rather than anticipating them, limiting adaptability under dynamic workloads. We ask: can we achieve proactive intra-GPU disaggregation that adapts effectively to dynamic workloads? The key challenge lies in managing the conflicting resource demands of prefill and decode under varying conditions. We first show that GPU resources exhibit diminishing returns -- beyond a saturation point, more allocation yields minimal latency benefit. Second, we observe that memory bandwidth contention becomes a critical bottleneck. These insights motivate a design that dynamically partitions GPU resources across prefill and decode phases, while jointly considering compute capacity, memory footprint, and bandwidth contention. Evaluated on diverse LLMs and workloads, our system Nexus achieves up to 2.2x higher throughput, 20x lower TTFT, and 2.5x lower TBT than vLLM; outperforms SGLang by up to 2x; and matches or exceeds disaggregated vLLM.

Figures

Figures reproduced from arXiv: 2507.06608 by the authors.

Figure 1
Figure 1. Design evolution of LLM inference systems. Com￾parison between monolithic, disaggregated, and intra-engine disag￾gregated designs. Ap is the prefill phase of request A; Bd, Cd, and Dd are the decode phases of requests B, C, and D. settings, even small delays matter a lot: humans perceive la￾tencies above one second as disruptive [67], and sub-second improvements has been shown to substantially boost en￾gagement [16]… view at source ↗
Figure 2
Figure 2. Inference process of transformer-based LLMs. Red boxes indicate compute-bound operations (KQV Linear, Prefill At￾tention, Attention Linear, and FFN Layer), while the orange box (Attention) represents a memory-bound operation. Auxiliary com￾ponents such as LayerNorm are omitted for clarity. Together, these mechanisms allow Nexus to match the high utilization of monolithic designs while achieving the isolation benefit… view at source ↗
Figure 3
Figure 3. Simplified GPU execution model. Modern GPUs share a global kernel queue, with SMs (streaming multiprocessors) dynamically fetching kernels to execute. Concurrently executing kernels compete for shared memory bandwidth. (𝑛 = 1), resulting in a matrix-vector multiplication (GEMV) with cost𝑂(𝐿𝑑). While the FLOP count is low, this operation is memory-bound [68] due to repeated access to the model weights and the growing… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Diminishing returns in prefill and decode with increasing SM allocation. (a) End-to-end latency for prefill and decode flattens well before full SM usage. (b) Prefill kernels (e.g., FFN, KQV, attention linear) show varied sensitivity to SM scaling, with FFN benefiting …
Figure 6
Figure 6. Figure 6: Memory contention’s impact and variability. (a) Decode latency increases as prefill KV length grows due to shared memory bandwidth pressure; (b) Prefill KV length fluctuates signif￾icantly over time, making contention difficult to predict statically. decode is memory-b…
Figure 8
Figure 8. Figure 8: Mechanisms for SM partition switching. Compari￾son between synchronous, asynchronous, and our asynchronous with hysteresis approach. Pitfall 2: Naive asynchronous switching. Letting streams switch independently avoids global stalls, but creates new problems: SM oversub…
Figure 9
Figure 9. Figure 9: End-to-end results on Single GPU. All systems use a single L20 GPU, except vLLM-P/D which uses two. This figure compares three workloads: Long Data Collection and Arxiv use Qwen-2.5–3B(first two rows), and Mixed uses Llama-3–1.8B(third row). The first and second column…
Figure 11
Figure 11. Figure 11: Offline Inference. Run on a single L20 under Long Data Collections and Mixed Workloads with 3B and 8B models repsectively. X means timeout. All systems use a single L20 GPU, except vLLM-P/D with two. In sum, Nexus offers the best latency–throughput tradeoff among all …
Figure 13
Figure 13. Figure 13: Ablation Study. Run with Mixed Workload on Llama3.1-8B using a single L20 GPU. PF-DF-Wo-SC is the intra￾engine PD disaggregation that uses FCFS for both prefill and decode scheduling, without dynamical GPU SM changing. PF-DF-W-SC is the intra-engine PD disaggregation …

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  2. DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing

    cs.LG 2025-11 conditional novelty 4.0 of 10

    DuetServe dynamically splits a GPU's compute cores between prefill and decode only when a latency model predicts trouble, improving serving throughput by up to 1.3x at similar or better token latency.

Reference graph

Works this paper leans on

73 extracted references · 38 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gulavani, Alexey Tumanov, and Ramachandran Ramjee

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, Ada Gavrilovska and D...

    work page 2024

  2. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, 13 Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfen...

    arXiv 2023

  3. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    work page 2020

  4. [5]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  5. [6]

    ccdv. 2025. arxiv-summarization.https://huggingface.co/datasets/ ccdv/arxiv-summarization

    work page 2025

  6. [7]

    Yihua Cheng, Kuntai Du, Jiayi Yao, and Junchen Jiang. 2024. Do Large Language Models Need a Content Delivery Network?arXiv preprint arXiv:2409.13761(2024)

    arXiv 2024

  7. [8]

    Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. 2022. Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Sharing. InProceedings of the 2022 USENIX Annual Technical Conference, USENIX ATC 2022, Carlsbad, CA, USA, July 11-13, 2022, Jiri Schindler and Noa Zilberman (Eds.). USENIX Assoc...

    work page 2022

  8. [9]

    Weihao Cui, Yukang Chen, Han Zhao, Ziyi Xu, Quan Chen, Xusheng Chen, Yangjie Zhou, Shixuan Sun, and Minyi Guo

    work page

Show all 73 references
  1. [10]

    Cursor. 2025. Cursor.https://www.cursor.com/

    2025

  2. [11]

    DeepSeek. 2025. DeepSeek.https://www.deepseek.com/

    2025

  3. [12]

    Jiangsu Du, Jinhui Wei, Jiazhi Jiang, Shenggan Cheng, Dan Huang, Zhiguang Chen, and Yutong Lu. 2024. Liger: Interleaving Intra- and Inter-Operator Parallelism for Distributed Large Model Inference. In Proceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Pract...

    2024

  4. [13]

    Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. 2024. MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, J...

    2024

  5. [14]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Ka- dian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston...

    2024 doi arXiv

  6. [15]

    Jingqi Feng, Yukai Huang, Rui Zhang, Sicheng Liang, Ming Yan, and Jie Wu. 2025. WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling. InProceedings of the 52nd Annual International Symposium on Computer Architecture. 1283–1295

    2025

  7. [16]

    2023.Amazon Found Every 100ms of Latency Cost them 1% in Sales.https://www.gigaspaces.com/blog/amazon-found-every- 100ms-of-latency-cost-them-1-in-salesAccessed: 2025-05-28

    GigaSpaces. 2023.Amazon Found Every 100ms of Latency Cost them 1% in Sales.https://www.gigaspaces.com/blog/amazon-found-every- 100ms-of-latency-cost-them-1-in-salesAccessed: 2025-05-28

    2023

  8. [17]

    Ruihao Gong, Shihao Bai, Siyu Wu, Yunqian Fan, Zaijun Wang, Xiuhong Li, Hailong Yang, and Xianglong Liu. 2025. Past-Future Scheduler for LLM Serving under SLA Guarantees. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages ...

    2025

  9. [18]

    Google. 2025. gemini.https://gemini.google.com/

    2025

  10. [19]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforce- ment learning.arXiv preprint arXiv:2501.12948(2025)

    2025 arXiv

  11. [20]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek- Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196(2024)

    2024 arXiv

  12. [21]

    Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. 2022. Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences. In16th USENIX Symposium on Operating Systems De- sign and Implementation, OSDI 2022, Carlsbad, CA, USA, July 11-13, 2022, Marcos K. Aguilera ...

    2022

  13. [22]

    Ke Hong, Lufang Chen, Zhong Wang, Xiuhong Li, Qiuli Mao, Jian- ping Ma, Chao Xiong, Guanyu Wu, Buhe Han, Guohao Dai, Yun Liang, and Yu Wang. 2025. semi-PD: Towards Efficient LLM Serv- ing via Phase-Wise Disaggregated Computation and Unified Storage. arXiv:2504.19867 [cs.CL]htt...

    2025 arXiv

  14. [23]

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024. Inference without Interfer- ence: Disaggregate LLM Inference for Mixed Downstream Work- loads.CoRRabs/2401.11181 (2024...

    2024 doi arXiv

  15. [24]

    Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar

    Aditya K. Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. 2025. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Progr...

    2025

  16. [25]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  17. [26]

    Zejia Lin, Hongxin Xu, Guanyi Chen, Xianwei Zhang, and Yutong Lu

  18. [27]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al . 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    2024 arXiv

  19. [28]

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anantha- narayanan, et al. 2024. Cachegen: Kv cache compression and stream- ing for fast large language model serving. InProceedings of the ACM SIGCOMM 2024 Con...

    2024

  20. [29]

    arXiv:2504.19516 [cs.DC]https:// arxiv.org/abs/2504.19516

    Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration. arXiv:2504.19516 [cs.DC]https:// arxiv.org/abs/2504.19516

  21. [30]

    Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving large language models over heterogeneous gpus and network via max-flow. InProceedings of the 30th ACM International Conference on Architectural Support for Programming ...

    2025

  22. [31]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xi- aoxiang Shi, et al. 2024. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. InPro- ceedi...

    2024

  23. [32]

    Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, et al. 2025. Deepcoder: A fully open-source 14b coder at o3-mini level. Notion Blog(2025)

    2025

  24. [33]

    Nvidia. 2024. FasterTransformer.https://github.com/NVIDIA/ FasterTransformer

    2024

  25. [34]

    Nvidia. 2025. B200.https://resources.nvidia.com/en-us-blackwell- architecture

    2025

  26. [35]

    Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. 2024. Spotserve: Serving generative large language models on preemptible instances. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages ...

    2024

  27. [36]

    Nvidia. 2025. H100.https://resources.nvidia.com/en-us-hopper- architecture/nvidia-h100-tensor-c

    2025

  28. [37]

    Nvidia. 2025. MIG.https://www.nvidia.com/en-sg/technologies/multi- instance-gpu/

    2025

  29. [38]

    Nvidia. 2025. CUDA Driver API: Green COntexts.https: //docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__GREEN_ _CONTEXTS.html

    2025

  30. [39]

    Hyungjun Oh, Kihong Kim, Jaemin Kim, Sungkyun Kim, Junyeol Lee, Du-Seong Chang, and Jiwon Seo. 2024. ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Op...

    2024

  31. [40]

    OpenAI. 2023. GPT-4 Technical Report.CoRRabs/2303.08774 (2023). doi:10.48550/ARXIV.2303.08774arXiv:2303.08774

    2023 doi arXiv

  32. [41]

    Nvidia. 2025. MPS.https://docs.nvidia.com/deploy/mps/contents. html

    2025

  33. [42]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In51st ACM/IEEE Annual International Symposium on Computer Architecture, ISCA 2024, Buenos Ai...

    2024

  34. [43]

    Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravis- hankar K. Iyer. 2024. Queue Management for SLO-Oriented Large Language Model Serving. InProceedings of the 2024 ACM Symposium on Cloud Computing, SoCC ...

    2024

  35. [44]

    OpenAI. 2025. ChatGPT.https://chatgpt.com

    2025

  36. [45]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Moon- cake: Trading More Storage for Less Computation — A KVCache- centric Architecture for Serving LLM Chatbot. In23rd USENIX Confer- ence on File and Storag...

    2025

  37. [46]

    ShareGPT. 2025. ShareGPT.https://huggingface.co/datasets/ anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ ShareGPT_V3_unfiltered_cleaned_split.json

    2025

  38. [47]

    Gonzalez, and Ion Stoica

    Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, and Ion Stoica. 2024. Fairness in Serving Large Language Models. In18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2...

    2024

  39. [48]

    perplexity. 2025. perplexity.https://www.perplexity.ai/

    2025

  40. [49]

    Foteini Strati, Xianzhe Ma, and Ana Klimovic. 2024. Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications. InProceedings of the Nineteenth European Conference on Computer Sys- tems, EuroSys 2024, Athens, Greece, April 22-25, 2024. ACM, 1075–1092. doi:10.1145/...

    2024

  41. [50]

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic Scheduling for Large Language Model Serving. In18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024, A...

    2024

  42. [51]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al

  43. [52]

    Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. 2024. Preble: Efficient Distributed Prompt Scheduling for LLM Serving.CoRRabs/2407.00023 (2024). doi:10.48550/ARXIV. 2407.00023arXiv:2407.00023

    2024 doi arXiv

  44. [53]

    TogetherComputer. 2025. Long-Data-Collections.https://huggingface. co/datasets/togethercomputer/Long-Data-Collections

    2025

  45. [54]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. InAdvances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Pro- cessing Systems...

    2017

  46. [55]

    vLLM team. 2025. vLLM-v1.https://blog.vllm.ai/2025/01/27/v1-alpha- release.html

    2025

  47. [56]

    5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599(2025)

    Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599(2025)

    2025 arXiv

  48. [57]

    SGLang team. 2025. SGLang-v0.4.https://lmsys.org/blog/2024-12-04- sglang-v0-4/

    2025

  49. [58]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    2025 arXiv

  50. [59]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  51. [60]

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2024. CacheBlend: Fast Large Language Model Serving with Cached Knowledge Fusion. arXiv preprint arXiv:2405.16444(2024)

    2024 arXiv

  52. [61]

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. 2023. Fast Distributed Inference Serving for Large Language Models.CoRRabs/2305.05920 (2023). doi:10.48550/ARXIV. 2305.05920arXiv:2305.05920

    2023 doi arXiv

  53. [62]

    Jiaming Xu, Jiayi Pan, Yongkang Zhou, Siming Chen, Jinhao Li, Yaoxiu Lian, Junyi Wu, and Guohao Dai. 2025. Specee: Accelerating large language model inference with speculative early exiting. InProceedings of the 52nd Annual International Symposium on Computer Architecture. 467–481

    2025

  54. [63]

    Lingfan Yu, Jinkun Lin, and Jinyang Li. 2025. Stateful large language model serving with pensieve. InProceedings of the Twentieth European Conference on Computer Systems. 144–158

    2025

  55. [64]

    Haochen Yuan, Yuanqing Wang, Wenhao Xie, Yu Cheng, Ziming Miao, Lingxiao Ma, Jilong Xue, and Zhi Yang. 2025. NeuStream: Bridging Deep Learning Serving and Stream Processing. InProceedings of the Twentieth European Conference on Computer Systems. 671–685

    2025

  56. [65]

    Shulai Zhang, Quan Chen, Weihao Cui, Han Zhao, Chunyu Xue, Zhen Zheng, Wei Lin, and Minyi Guo. 2025. Improving GPU Sharing Per- formance through Adaptive Bubbleless Spatial-Temporal Sharing. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025,...

    2025

  57. [66]

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. 2025. FlashInfer: Efficient and Cus- tomizable Attention Engine for LLM Inference Serving.arXiv preprint arXiv:2501.01005(...

    2025 arXiv

  58. [67]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In16th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2022, Carlsbad, CA, USA, July 11-13, 20...

    2022

  59. [68]

    Kan Zhu, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Yufei Gao, Qinyu Xu, Tian Tang, Zihao Ye, Keisuke Kamahori, Chien- Yu Lin, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci

  60. [71]

    Gonzalez, Clark W

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. 2023. Efficiently Program- ming Large Language Models using SGLang.CoRRabs/2312.07104 (2023). doi:...

    2023 doi arXiv

  61. [72]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In18th USENIX Symposium on Operating Systems Design and Implementation, O...

    2024

  62. [74]

    doi:10.48550/ARXIV.2408

    NanoFlow: Towards Optimal Large Language Model Serving Throughput.CoRRabs/2408.12757 (2024). doi:10.48550/ARXIV.2408. 12757arXiv:2408.12757 16

    2024 doi arXiv

  63. [2023]

    InProceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, Jason Flinn, Margo I

    Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonatha...

    2023

  64. [2024]

    doi:10.1145/3698038.3698523

    ACM, 18–35. doi:10.1145/3698038.3698523

  65. [2025]

    CoRRabs/2504.14489 (2025)

    Optimizing SLO-oriented LLM Serving with PD-Multiplexing. CoRRabs/2504.14489 (2025). doi:10.48550/ARXIV.2504.14489 arXiv:2504.14489

    2025 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.