Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FastSwitch claims that grouping KV cache into contiguous block groups, swapping asynchronously via worker threads, and reusing uncontaminated CPU copies cuts preemption-induced context switching overhead enough to speed up tail TTFT and…

desk verdict A well-engineered fix for a real problem, but the headline speedups are measured in a favorable synthetic regime and need qualification and better baselines before the claims hold. read the letter →

arxiv 2411.18424 v1 pith:L5RUKRNJ submitted 2024-11-27 cs.LG cs.DC

Ao Shen , Zhiyao Li , Mingyu Gao This is my paper · ORCID
classification cs.LGcs.DC
keywords LLMservingfairnesspreemptionKVcachecontextswitchingvSLOasynchronousswapping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the main cost of fairness in LLM serving is not the priority scheduling itself but the context switching it forces: every time a request is preempted, its KV cache must be swapped between GPU and CPU memory, and existing systems do this inefficiently. FastSwitch claims to reduce that overhead with three coordinated mechanisms: coarser-grained contiguous KV cache allocation, asynchronous multithreaded swapping, and reuse of partially valid KV cache copies across conversation turns. The paper reports that on LLaMA-8B and Qwen-32B under simulated priority-update traces, FastSwitch improves tail time-to-first-token and time-between-tokens latencies by 1.4-11.2x compared with vLLM, and throughput by up to 1.44x. A sympathetic reader would take the central claim to be that fairness-aware preemption can be made cheap enough that SLO-driven priority adjustments no longer trade away responsiveness.

What carries the argument

The load-bearing mechanism is the Dynamic Block Group Manager, an I/O-aware KV cache allocator that manages memory in buddy-allocator-style block groups instead of individual fixed-size blocks, merging free groups and splitting active groups to match request sizes. On top of it, the Multithreading Swap Manager runs cudaMemcpyAsync dispatch and CUDA-event tracking in a C++ thread pool, with conflict detection that synchronizes only when an ongoing swap-in touches a block group being reallocated. The KV Cache Reuse Mechanism then keeps CPU copies of prior turns, marks contaminated segments, and preallocates adjacent CPU space for the next turn's increment, so only the new KV cache is transferred. Together they convert many small serialized swaps into fewer larger overlapped transfers.

What would settle it

Run FastSwitch against vLLM on a production LLM serving workload with SLO-deadline-driven priority updates and online reordering, and compare P99.9 TBT and TTFT; if the latency gap narrows to near zero when priorities change in response to actual deadline pressure rather than fixed-interval offline patterns, the central claim is refuted.

Watch

Extended reading notes

Core claim

The central discovery is that the paged, fixed-size block KV cache policy that gives vLLM near-zero memory waste is the wrong granularity for preemption: it fragments each request's cache into many small, non-contiguous pieces, so swapping is dominated by cudaMemcpyAsync dispatch overhead rather than data transfer, and the GPU stalls while the CPU serializes the transfer. FastSwitch's Dynamic Block Group Manager applies buddy-allocation-style splitting and merging to make transfers large and contiguous, its Multithreading Swap Manager moves the dispatch work off the Python GIL into a C++ thread pool so swapping overlaps with inference, and its KV Cache Reuse Mechanism tracks which CPU-resident KV cache segments have been contaminated by higher-priority requests so that multi-turn conversations swap out only the genuinely new portion. The paper's measured conclusion is that these mechanisms together cut context-switching overhead enough to speed up tail TTFT and TBT by 1.4-11.2x, with the largest gains in P99.9 TBT, without giving up vLLM's memory efficiency.

Load-bearing premise

The evaluation assumes that the simulated Random and Markov priority-update traces, with priorities computed offline and updated on fixed iteration intervals, capture the preemption dynamics of real LLM-as-a-service workloads; if real fairness-driven priority changes are triggered by SLO deadlines and arrival bursts with different temporal structure, the measured speedups may not carry over.

Editorial extensions

If this is right

  • Under frequent priority updates, FastSwitch reduces P95, P99, and P99.9 TTFT and P99.9 TBT by 1.4-5.8x, 3.7-4.1x, 2.5-3.7x, and 2.0-2.7x for LLaMA-8B and by 1.4-1.7x, 1.5-1.6x, 1.3-1.4x, and 3.6-11.2x for Qwen-32B across Markov and Random patterns.
  • End-to-end throughput improves by up to 1.334x on LLaMA-8B and 1.444x on Qwen-32B, with larger gains when swapping latency is high relative to inference time.
  • The added scheduling and bookkeeping overhead stays under 1% of end-to-end time even as priority-update frequency rises.
  • Coarser initial block-group sizes from 64 to 3,000 tokens change average swap granularity by at most 15.13%, so the design is robust to that tuning knob.
  • The KV Cache Reuse Mechanism cuts swapped-out blocks by 53% in the microbenchmark, which directly lowers preemption stall time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same machinery transfers directly to decode-prefill disaggregation and other KV-cache offloading scenarios, since those also move large KV caches between memory tiers and would inherit the dispatch-overhead and GPU-idle problems.
  • If real production fairness schedulers update priorities more often than once per 50-100 iterations, the measured gains could understate FastSwitch's advantage on real workloads; if updates are rarer, the advantage would shrink.
  • A clean testable extension is to vary the interconnect, such as PCIe 4.0 versus 5.0 or CXL, and confirm that the dispatch-overhead reduction remains the dominant term, since the paper's workloads run on PCIe 4.0 x16.
  • The 60 GB CPU swap-space optimum is specific to this setup; larger CPU pools or faster host memory could shift the reuse-versus-reclaim tradeoff.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper identifies three challenges in preemption-induced context switching for fairness-aware LLM serving (inadequate PCIe I/O utilization, GPU idleness during swaps, and redundant I/O in multi-turn conversations) and proposes FastSwitch, a serving system built on vLLM with three mechanisms: a Dynamic Block Group Manager for coarse-grained contiguous KV cache allocation, a Multithreading Swap Manager for asynchronous CPU-GPU transfers, and a KV Cache Reuse Mechanism for reusing partially valid KV cache copies in CPU memory. The evaluation uses LLaMA-8B on an A10 GPU and Qwen-32B on an A100 GPU, with ShareGPT-based multi-turn conversations and synthetic Random/Markov priority-update traces, comparing against vLLM 0.3.3. The paper reports 1.4-11.2x speedups across tail TTFT and TBT, with up to 1.44x throughput improvement.

Significance. The issue of preemption overhead in fairness-aware LLM serving is real and underexplored, and the paper provides a clear problem decomposition with three somewhat orthogonal optimizations and an incremental evaluation of each. The system is implemented on a concrete baseline (vLLM) rather than evaluated solely by simulation, and several microbenchmarks (e.g., context-switch overhead ratio in Figure 10, swap-out volume reduction in Table 1, token generation efficiency in Figure 12) support the individual mechanisms. The main weakness is external validity: the headline speedups are measured under synthetic offline-computed priority traces with a deliberately favorable priority-update frequency for LLaMA-8B, and the paper does not compare against the closest prior systems (Llumnix, AttentionStore) that it criticizes. Thus, the claimed improvements over state-of-the-art preemption techniques are plausible but not yet fully established for real fairness-aware schedulers.

major comments (3)
  1. [Section 4] The central claim in the abstract and Section 7 (1.4-11.2x speedups) is not qualified by the fact that all latency results come from synthetic Random and Markov priority traces whose priorities are "determined offline" and, for LLaMA-8B, updated at frequency 0.04 "to better highlight the optimizations". Figure 10 shows that the context-switching overhead ratio and the associated benefit shrink as priority-update frequency decreases. Real fairness schedulers such as VTC, Andes, and FastServe update priorities from runtime state (SLO slack, queue wait, token-level service), which creates preemption patterns correlated with request lengths and arrivals rather than the independent patterns used here. The paper should evaluate under a real fairness policy (e.g., a VTC-like token-level scheduler) or restrict the advertised speedup claims to the simulated settings.
  2. [Section 5.1] The experimental comparison is limited to vLLM 0.3.3. Sections 2.2 and 6 discuss Llumnix and AttentionStore at length and claim FastSwitch addresses their shortcomings, but no empirical comparison against these systems (or against FastServe-style iteration-wise transmission) is provided. Without such a comparison, the claim that FastSwitch improves on "state-of-the-art" preemption techniques is not substantiated; the increment over vLLM alone does not isolate the gain relative to the closest prior work. A head-to-head comparison or a clear argument why these are not competitive baselines is needed.
  3. [Sections 3.1 and 3.2] Key design parameters are underspecified to the point of irreproducibility. The initial block group size is stated as 60 blocks in Section 3.1 ("approximately 1,000 tokens when the block size is 16 tokens") but as "about 70 vLLM blocks" in Section 5.3.1, and the "dynamic adjustment" rule for block group sizing is never formally defined. Similarly, the adaptive decision in Algorithm 1 to choose synchronous versus asynchronous swap-in is described only qualitatively ("when the total number of requests is high, but each request is relatively short") with no concrete metric, threshold, or equation. Without these specifications or a released artifact, the measured gains cannot be independently reproduced.
minor comments (6)
  1. [Abstract and Section 5.1.1] The speedup ranges are reported per model and per metric (e.g., 4.3-5.8x for LLaMA-8B P95 TTFT, 3.6-11.2x for Qwen-32B P99.9 TBT), but the abstract aggregates them as a single "1.4-11.2x" range without stating which metric or model produced the endpoints; please indicate the conditions for the minimum and maximum.
  2. [Section 3.1 and Section 5.3.1] The initial block group size inconsistency (60 blocks in Section 3.1 vs "about 70 vLLM blocks" for 1,000 tokens in Section 5.3.1) should be corrected, since 1,000 tokens at 16 tokens/block corresponds to 62.5 blocks, not 70.
  3. [Section 4] The statement that the Qwen-32B priority-update frequency of 0.02 "follows the study in Andes" needs a precise pointer to the relevant Andes configuration, because the cited work reports round-robin QoE results rather than an explicit priority-update frequency.
  4. [Section 2.2] The discussion around Figure 2 says the impact of global priority updates "is most pronounced in tail cases, where a significant proportion of requests experience delays," but the figure appears to show that only a small ratio of requests wait; please align the text with the data shown in the figure.
  5. [Algorithm 1] Several state and transition names used in Algorithm 1 (r_info, MovePending, SwapInStreamSynchronize, DetectConflict) are not defined in the text; a short paragraph describing the queues and the exact semantics of these functions would make the algorithm self-contained.
  6. [Section 5.2] The phrase "dispatched in pass iterations" should read "past iterations", and the term "call stack overhead" should be defined (what is measured and how) before the results in Figure 9 are discussed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FastSwitch's 1.4-11.2x speedup claims are empirical benchmark measurements against the external vLLM baseline, with no fitted input renamed as prediction and no load-bearing self-citation chain.

full rationale

The paper's derivation chain is an engineering system design followed by measurements. The headline claims (Section 1 and Section 7) are empirical comparisons against vLLM, an external baseline; the three optimizations are implemented and measured (Section 5), not derived from a fitted model. The Dynamic Block Group initial size (60 blocks) and the adaptive swap-in strategy are hand-chosen or runtime-tuned design parameters, and Section 5.3.1 explicitly tests sensitivity to the initial size, so they are not inputs that force the claimed latency numbers. The use of synthetic context-switching traces (Section 4: 'As there are no publicly available context-switching traces... we refer to the work of (Yin et al., 2024) and simulate two patterns') and the choice of an elevated priority-update frequency for LLaMA-8B 'to better highlight the optimizations in context switching' are workload-realism / favorable-regime concerns, not circularity: the priority trace is an input workload, and the measured speedups are outputs, with no equation or definition in which the output is encoded in the input. There is no self-citation used as load-bearing evidence: related-work citations to vLLM, Andes, FastServe, AttentionStore, and LLMS support context but are not the argument for FastSwitch's claimed improvements. Consequently the central claim has independent empirical content, and any concerns about transfer to real fairness-driven preemption policies belong to external validity, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hardware-behavior and workload-representativeness assumptions rather than mathematical axioms: CUDA dispatch overhead dominating small transfers, the buddy allocator preserving vLLM's memory efficiency, and the synthetic priority traces standing in for real LLMaaS context switching. Two design parameters (initial block group size, adaptive swap-in decision rule) and one environment parameter (60 GB CPU swap space) are hand-chosen or runtime-tuned. No invented physical entities are introduced; the software components are new objects but only have supporting evidence from the paper's own benchmarks.

free parameters (3)
  • Initial block group size = 60 blocks (about 1000 tokens; also described as about 70 blocks)
    Design parameter controlling KV cache allocation granularity and transfer size. Section 5.3.1 sensitivity analysis shows granularity varies by at most 15.13% across sizes, so impact on the central claim is moderate. The text contains an internal inconsistency: 60 blocks vs about 70 blocks.
  • Adaptive swap-in decision rule = Unspecified
    Section 3.2 uses a profiler to choose between asynchronous and synchronous swap-in based on recent swapping metrics, but no threshold or formula is given. This runtime decision affects the reported GPU utilization and token generation efficiency improvements and cannot be reproduced as specified.
  • CPU swap space allocation = 60 GB
    Evaluation configuration for KV cache copies in CPU memory. Section 5.3.3 shows context switching overhead decreases with more CPU memory and saturates around 60 GB, so the reported gains depend on this hand-chosen allocation.
assumptions (4)
  • ad hoc to paper Synthetic Random and Markov priority traces are representative of real LLMaaS context switching.
    Section 4 states no public context-switching traces exist and simulates two patterns with offline-computed priorities; the end-to-end results depend on this representativeness assumption.
  • domain assumption cudaMemcpyAsync dispatch overhead dominates transfer time for small KV cache transfers on PCIe 4.0.
    Section 2.2 Challenge #1 reports dispatch accounts for 90-95% of transmission time; the coarse-grained block group design relies on this being the dominant bottleneck.
  • domain assumption vLLM 0.3.3 is a valid state-of-the-art baseline for fairness-aware serving.
    All comparisons are against vLLM 0.3.3 only; the paper does not compare with Llumnix, AttentionStore, or FastServe, so the claimed advantage over prior art depends on this baseline choice.
  • domain assumption Buddy-system splitting and merging of block groups preserves near-zero memory waste.
    Section 3.1 claims alignment with vLLM's memory policy, but no memory-waste or fragmentation measurements are provided to confirm that dynamic splitting and merging do not increase fragmentation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving." pith.science (2026). https://pith.science/paper/L5RUKRNJ

@misc{pith2026241118424,
  author       = {Pith},
  title        = {Pith review of: FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L5RUKRNJ}},
  note         = {Machine review of arXiv:2411.18424}
}
read the original abstract

Serving numerous users and requests concurrently requires good fairness in Large Language Models (LLMs) serving system. This ensures that, at the same cost, the system can meet the Service Level Objectives (SLOs) of more users , such as time to first token (TTFT) and time between tokens (TBT), rather than allowing a few users to experience performance far exceeding the SLOs. To achieve better fairness, the preemption-based scheduling policy dynamically adjusts the priority of each request to maintain balance during runtime. However, existing systems tend to overly prioritize throughput, overlooking the overhead caused by preemption-induced context switching, which is crucial for maintaining fairness through priority adjustments. In this work, we identify three main challenges that result in this overhead. 1) Inadequate I/O utilization. 2) GPU idleness. 3) Unnecessary I/O transmission during multi-turn conversations. Our key insight is that the block-based KV cache memory policy in existing systems, while achieving near-zero memory waste, leads to discontinuity and insufficient granularity in the KV cache memory. To respond, we introduce FastSwitch, a fairness-aware serving system that not only aligns with existing KV cache memory allocation policy but also mitigates context switching overhead. Our evaluation shows that FastSwitch outperforms the state-of-the-art LLM serving system vLLM with speedups of 1.4-11.2x across different tail TTFT and TBT.

Figures

Figures reproduced from arXiv: 2411.18424 by the authors.

Figure 3
Figure 3. Timeline comparison of fixed-size block based preemp￾tion and dynamic block group based preemption. its 10 µs execution time, leading to idle I/O. This issue is further exacerbated by the fact that the transfer size is below PCIe 4.0’s optimal 320 KB, thus reducing efficiency. In this setting, dispatch time accounts for 90%-95% of the total transmission time. Previous works such as Llumnix (Sun et al., 2024) tackle … view at source ↗
Figure 1
Figure 1. Latency breakdown across percentiles. 85 90 93 95 97 99 Percentile 0.0 0.2 0.4 0.6 0.8 1.0 Ratio [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 4
Figure 4. ShareGPT conversation turns & lengths distribution. Challenge #3: Contaminated CPU KV Cache Copies in Multi-turn Conversations. Multi-turn conversations dom￾inate real-world LLM applications like chatbots. Datasets like ShareGPT ( 100K conversations) show 78% of interac￾tions involve multiple turns, averaging 5.5 turns per conver￾sation. These require maintaining large KV caches for con￾textual coherence, but repeti… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: FastSwitch system overview. 3.1 Dynamic Block Group Manager for Increased Granularity and I/O Bandwidth Utilization In this section, to address Challenge #1, we introduce the Dynamic Block Group Manager, an I/O-aware KV cache allocator. Instead of managing individual b…
Figure 6
Figure 6. Figure 6: Comparison of varying degrees of asynchronous preemp￾tion. Swap Manager employs a dynamic swapping strategy based on the system’s current state. To enable informed decision￾making, a profiler monitors key metrics such as the number and size of ongoing swapping operatio…
Figure 7
Figure 7. Figure 7: Workflow of the KV Cache Reuse Mechanism. 4 METHODOLOGY System and Workload Configuration. We evaluate FastSwitch using the LLaMA-8B and Qwen-32B models on NVIDIA A10 24 GB and A100 80 GB GPUs, respectively. Each GPU is configured with 60 GB of CPU swap space for KV ca…
Figure 8
Figure 8. Figure 8: Comparison of TTFT, TBT, and throughput between FastSwitch and baseline under different models and traces. than decode, meaning that the same swapping delay before inference has a greater impact on TBT than TTFT. This effect becomes increasingly pronounced as the model…
Figure 10
Figure 10. Figure 10: Context switching overhead across priority-update frequencies. 0.005 0.010 0.013 0.020 0.030 0.040 Priority Update Frequency 0.8 0.9 1.0 1.1 1.2 Granularity min max [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 12
Figure 12. Figure 12: Efficiency compari￾son. 10 35 60 85 110 135 Swap Space (GB) 0.02 0.03 0.04 Ratio [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCORPIO: Serving the Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference

    cs.LG 2025-05 reject novelty 5.0 of 10

    SCORPIO combines least-deadline-first reordering, VBS admission control, and credit-based batching to improve SLO attainment in LLM serving, but its evaluation leaks training data into the served workload.

Reference graph

Works this paper leans on

47 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    S., and Ramjee, R

    Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., and Ramjee, R. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308.16369, 2023

    arXiv 2023

  2. [2]

    S., Tumanov, A., and Ramjee, R

    Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R. Taming throughput-latency tradeoff in llm inference with sarathi-serve. arXiv preprint arXiv:2403.02310, 2024

    arXiv 2024

  3. [3]

    Y., Rajbhandari, S., Zhang, M., Awan, A

    Aminabadi, R. Y., Rajbhandari, S., Zhang, M., Awan, A. A., Li, C., Li, D., Zheng, E., Rasley, J., Smith, S., Ruwase, O., and He, Y. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022. URL https://arxiv.org/abs/2207.00032

    arXiv 2022

  4. [4]

    Piqa: Reasoning about physical commonsense in natural language

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  5. [5]

    Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  6. [7]

    Model tells you what to discard: Adaptive kv cache compression for llms

    Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023

    arXiv 2023

  7. [8]

    Frage: Frequency-agnostic word representation

    Gong, C., He, D., Tan, X., Qin, T., Wang, L., and Liu, T.-Y. Frage: Frequency-agnostic word representation. Advances in neural information processing systems, 31, 2018

    work page 2018

  8. [9]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    arXiv 2009

Show all 47 references
  1. [10]

    Hugging face large language models (llms)

    Hugging Face . Hugging face large language models (llms). https://huggingface.co/, 2024. Accessed: 2024-10-28

    2024

  2. [11]

    Y., Fried, D., and Salakhutdinov, R

    Koh, J. Y., Fried, D., and Salakhutdinov, R. R. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  3. [12]

    H., Gonzalez, J., Zhang, H., and Stoica, I

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp.\ 611--626, 2023

    2023

  4. [13]

    Larimi, S. S. N., Salami, B., Unsal, O. S., Kestelman, A. C., Sarbazi-Azad, H., and Mutlu, O. Understanding power consumption and reliability of high-bandwidth memory with voltage underscaling, 2020. URL https://arxiv.org/abs/2101.00969

    2020 arXiv

  5. [15]

    From llm to conversational agent: A memory enhanced architecture with fine-tuning of large language models, 2024 b

    Liu, N., Chen, L., Tian, X., Zou, W., Chen, K., and Cui, M. From llm to conversational agent: A memory enhanced architecture with fine-tuning of large language models, 2024 b . URL https://arxiv.org/abs/2401.02777

    2024 arXiv

  6. [16]

    Cachegen: Kv cache compression and streaming for fast large language model serving

    Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., et al. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, pp.\ 38--56, 2024 c

    2024

  7. [17]

    Large language models: A survey

    Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao, J. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024

    2024 arXiv

  8. [18]

    Lightllm: A lightweight framework for large language model inference

    ModelTC. Lightllm: A lightweight framework for large language model inference. https://github.com/ModelTC/lightllm, 2024. A Python-based LLM inference and serving framework with lightweight design, easy scalability, and high-speed performance

    2024

  9. [19]

    Codegen: An open large language model for code with multi-turn program synthesis, 2023

    Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis, 2023. URL https://arxiv.org/abs/2203.13474

    2023 arXiv

  10. [20]

    Nvidia tensorrt-llm

    NVIDIA . Nvidia tensorrt-llm. https://docs.nvidia.com/tensorrt-llm/index.html, 2024. Accessed: 2024-10-28

    2024

  11. [21]

    OpenAI . Chatgpt. https://openai.com/chatgpt, 2024. Accessed: 2024-10-28

    2024

  12. [22]

    One queue is all you need: Resolving head-of-line blocking in large language model serving

    Patke, A., Reddy, D., Jha, S., Qiu, H., Pinto, C., Cui, S., Narayanaswami, C., Kalbarczyk, Z., and Iyer, R. One queue is all you need: Resolving head-of-line blocking in large language model serving. arXiv preprint arXiv:2407.00047, 2024

    2024 arXiv

  13. [23]

    Mooncake: Kimi's kvcache-centric architecture for llm serving

    Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: Kimi's kvcache-centric architecture for llm serving. arXiv preprint arXiv:2407.00079, 2024

    2024 arXiv

  14. [24]

    Sharegpt: Share your wildest chatgpt conversations with one click

    ShareGPT . Sharegpt: Share your wildest chatgpt conversations with one click. https://sharegpt.com/, 2024

    2024

  15. [25]

    Self-attention with relative position representations

    Shaw, P., Uszkoreit, J., and Vaswani, A. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155, 2018

    2018 arXiv

  16. [26]

    Flexgen: High-throughput generative inference of large language models with a single gpu

    Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., R \'e , C., Stoica, I., and Zhang, C. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pp.\ 31094--31116. PMLR, 2023

    2023

  17. [27]

    E., and Stoica, I

    Sheng, Y., Cao, S., Li, D., Zhu, B., Li, Z., Zhuo, D., Gonzalez, J. E., and Stoica, I. Fairness in serving large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp.\ 965--988, 2024

    2024

  18. [29]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  19. [30]

    A simple hardware buddy system memory allocator

    Von Puttkamer, E. A simple hardware buddy system memory allocator. IEEE Transactions on Computers, C-24 0 (10): 0 953--957, 1975. doi:10.1109/T-C.1975.224100

    1975

  20. [31]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  21. [32]

    Fast distributed inference serving for large language models

    Wu, B., Zhong, Y., Zhang, Z., Huang, G., Liu, X., and Jin, X. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920, 2023

    2023 arXiv

  22. [33]

    Qwen2 technical report

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  23. [34]

    Llm as a system service on mobile devices

    Yin, W., Xu, M., Li, Y., and Liu, X. Llm as a system service on mobile devices. arXiv preprint arXiv:2403.11805, 2024

    2024 arXiv

  24. [35]

    E., et al

    Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Hao Yu, C., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. Efficiently programming large language models using sglang. arXiv e-prints, pp.\ arXiv--2312, 2023

    2023

  25. [36]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving, 2024

    Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving, 2024. URL https://arxiv.org/abs/2401.09670

    2024 arXiv

  26. [37]

    Multilingual machine translation with large language models: Empirical results and analysis, 2024

    Zhu, W., Liu, H., Dong, Q., Xu, J., Huang, S., Kong, L., Chen, J., and Li, L. Multilingual machine translation with large language models: Empirical results and analysis, 2024. URL https://arxiv.org/abs/2304.04675

    2024 arXiv

  27. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  28. [39]

    CaR: An Efficient KV Cache Reuse System for Large Language Model Inference

    Kexin Chu, Tzechinh Liu, Yunding Li, Pengchao Yuan, and Wei Zhang. CaR: An Efficient KV Cache Reuse System for Large Language Model Inference. In Proceedings of the 2024 International Conference on XYZ, 2024

    2024

  29. [40]

    Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

    Yingjie Zhang, Yuanzheng Liu, Liang Chen, Guoliang Li, and Pengfei Xu. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference. In Proceedings of the 2023 Conference on XYZ, 2023

    2023

  30. [41]

    CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, and Junchen Jiang. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving. In Proceedings of the ACM SIGCOMM 2024 Conference, 2024

    2024

  31. [42]

    XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference

    João Monteiro, Étienne Marcotte, Pierre-André Noël, Valentina Zantedeschi, David Vázquez, Nicolas Chapados, Christopher Pal, and Perouz Taslakian. XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference. arXiv preprint arXiv:2404.15420, 2023

    2023 arXiv

  32. [44]

    AttentionStore: Cost-effective Attention Reuse across Multi-turn Conversations in Large Language Model Serving

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Zhou Yu, and Pengfei Zuo. AttentionStore: Cost-effective Attention Reuse across Multi-turn Conversations in Large Language Model Serving. arXiv preprint arXiv:2403.19708, 2023

    2023 arXiv

  33. [45]

    InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management. arXiv preprint arXiv:2406.19707, 2024

    2024 arXiv

  34. [47]

    Andes: Defining and Enhancing Quality-of-Experience in LLM-based Text Streaming Services

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Andes: Defining and Enhancing Quality-of-Experience in LLM-based Text Streaming Services. arXiv preprint arXiv:2404.16283, 2023

    2023 arXiv

  35. [48]

    PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications

    Zhihao Bai, Zhen Zhang, Yibo Zhu, and Xin Jin. PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 499–516, 2020

    2020

  36. [49]

    UELLM: A Unified and Efficient Approach for LLM Inference Serving

    Yiyuan He, Minxian Xu, Jingfeng Wu, Wanyi Zheng, Kejiang Ye, and Chengzhong Xu. UELLM: A Unified and Efficient Approach for LLM Inference Serving. arXiv preprint arXiv:2409.14961, 2023

    2023 arXiv

  37. [50]

    Fast Distributed Inference Serving for Large Language Models

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast Distributed Inference Serving for Large Language Models. arXiv preprint arXiv:2406.03243, 2023

    2023 arXiv

  38. [51]

    Prophet: An LLM Inference Engine Optimized For Head-of-Line Blocking

    Maishan Wang, Shitao Tang, and others. Prophet: An LLM Inference Engine Optimized For Head-of-Line Blocking. In Proceedings of the XYZ Conference, 2023

    2023

  39. [52]

    QLM: A Preemptive Scheduling Policy for LLM Serving

    Yunfan Zhang, Yichi Zhang, Xinyu Zhao, and Yuxing Han. QLM: A Preemptive Scheduling Policy for LLM Serving. arXiv preprint arXiv:2308.12345, 2023

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.