Pith. sign in

REVIEW 6 major objections 6 minor 87 references

Memory Offloading for Large Language Model Inference with Latency SLO Guarantees

T0 review · 6 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Select-N shows LLM memory offloading can meet latency SLOs by turning the memory/latency tradeoff into a single tunable interval, using 2.37x more host memory than FlexGen and raising throughput 1.85x.

desk verdict A real engineering advance on SLO-aware offloading with an honest evaluation, but the central determinism assumption and the under-derived interval formula need referee attention before the guarantee claims can be trusted. read the letter →

arxiv 2502.08182 v1 pith:DWL4V6KA submitted 2025-02-12 cs.DC

Chenxiang Ma , Zhisheng Ye , Hanyu Zhao , Zehua Yang , Tianhao Fu , Jiaxun Han , Jie Zhang , Yingwei Luo
show 4 more authors
Xiaolin Wang Zhenlin Wang Yong Li Diyu Zhou
This is my paper · ORCID
classification cs.DC
keywords memoryoffloadingLLMinferencelatencySLOintervalprefilldecodingdisaggregationPCIebandwidthcontentionhostutilization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that memory offloading for LLM serving can satisfy strict latency SLOs while still using host memory aggressively, if the offloading decision is made dynamically rather than statically. Its central move is to exploit the deterministic per-layer computation time of decoder-only LLMs, which makes it possible to predict how much transfer latency can be hidden by compute. Select-N introduces an offloading interval: the number of consecutive layers kept in GPU memory before one layer is offloaded to the host. By choosing the smallest interval that still meets the required TTFT and TPOT targets, Select-N claims to avoid both the SLO violations of DeepSpeed and the host-memory underuse of FlexGen, with experiments reporting consistent SLO compliance, 2.37x more host memory usage, and 1.85x higher throughput than FlexGen.

What carries the argument

The central object is the offloading interval, defined as the number of consecutive layers kept in GPU memory before one layer's state is offloaded to host memory. The load-bearing mechanism is that offloaded state is prefetched when compute reaches the first layer of the interval, so the transfer time of one offloaded layer is overlapped with the computation of several layers, not just one as in DeepSpeed and FlexGen. The paper argues this works because each decoder layer's compute time is deterministic, depending only on batch size and sequence length, allowing an offline performance record to predict which interval will meet a given SLO. The per-bus coordinator then uses the minimum and maximum valid intervals for each GPU, plus an estimate of PCIe bandwidth consumed per interval, to pick intervals that keep all GPUs within their SLOs while maximizing aggregate host memory use.

What would settle it

A direct test is to run a long serving workload under sustained GPU load and measure per-layer compute time distributions; if any layer's compute time drifts by more than the SLO slack used by Select-N, and SLO violations appear despite correct interval selection, the determinism premise is falsified. Alternatively, instrument a production server to record actual per-layer times across different thermal and clock states and show they vary enough to break the offline decision.

Watch

Extended reading notes

Core claim

Select-N's central claim is that the tension between meeting latency SLOs and maximizing host memory usage reduces to picking one integer: the offloading interval. For an interval of i, the state of every i-th decoder layer is offloaded to host memory, and the transfer is prefetched when computation begins on the first layer of that interval, so the load of one layer is hidden by the compute of i layers. Because each decoder layer has the same structure, the same operations, and the same input size within a phase, the paper asserts that per-layer compute time is deterministic for a given model, batch size, and sequence length. This determinism lets an offline analyzer build a performance record mapping SLO, batch size, and sequence length to the optimal interval, while a per-bus coordinator adjusts intervals at each inference iteration when GPUs share PCIe bandwidth. The paper reports that Select-N consistently meets TTFT and TPOT SLOs across batch sizes and contention scenarios, uses 2.37x more host memory than FlexGen under the same SLOs, and improves serving throughput over FlexGen by up to 1.85x, while also enabling models whose memory demands exceed GPU capacity.

Load-bearing premise

The load-bearing premise is that each decoder layer's compute time is deterministic and depends only on batch size and sequence length, so an offline performance record remains accurate during online serving; if GPU clocks, thermal state, or other contention perturb per-layer compute time beyond the SLO slack, the chosen interval no longer guarantees the SLO.

Editorial extensions

If this is right

  • If Select-N's claims hold, LLM serving can meet TTFT and TPOT SLOs while offloading far more state to host memory, cutting GPU memory cost without sacrificing responsiveness.
  • The same offloading interval mechanism can support models larger than GPU memory, since the interval can be reduced until enough layers are offloaded to fit the model.
  • Separating prefill and decoding becomes more effective, because the analyzer can assign different optimal intervals to the compute-intensive prefill phase and the memory-intensive decoding phase.
  • Under PCIe bandwidth contention, dynamic per-iteration interval adjustment should keep all GPUs sharing a bus within their SLOs, whereas static offloading schemes like FlexGen can violate SLOs at smaller batch sizes.
  • Because larger memory savings translate into more GPU blocks for tokens, Select-N supports larger batch sizes, longer input sequences, and longer output sequences than a fixed-portion offloader at the same SLO.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the deterministic per-layer compute time premise holds across a wider range of models and hardware, the offline-record approach could be reused as a cheap admission-control signal: a request whose required interval is below the GPU's feasible minimum can be rejected before entering the queue.
  • The paper does not address slow drift in compute time from GPU clock throttling, thermal state, or co-tenant interference; a natural testable extension is to add a lightweight online re-measurement that adjusts the interval when observed layer times drift beyond the SLO slack.
  • The interval abstraction could combine with KV-cache offloading policies, since Select-N currently bounds host memory usage by layer-state transfers while leaving KV-cache management to the underlying page-based allocator; unifying both under one memory budget is a plausible next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper presents Select-N, a memory-offloading system for decoder-only LLM inference that introduces an 'offloading interval' knob to trade host-memory usage against latency SLOs. The design assumes per-layer compute time is deterministic, uses an offline analyzer to build a performance record mapping SLOs, batch sizes, and sequence lengths to the smallest SLO-satisfying interval, and adds a per-bus coordinator to adjust intervals under PCIe bandwidth contention. Evaluations on A10 GPUs with OPT, Qwen2-beta, and LLaMA models claim that Select-N consistently meets TTFT and TPOT SLOs, uses 2.37x more host memory than a modified SLO-aware FlexGen, and improves throughput by up to 1.85x.

Significance. If the deterministic-compute premise holds, the offloading-interval abstraction is a simple and practical way to expose the SLO/memory trade-off, and Select-N is a plausible improvement over DeepSpeed and FlexGen for latency-bounded serving. The paper has clear strengths: it provides a concrete two-stage design, an implementation on vLLM, a comparison against a reasonable SLO-aware FlexGen variant, explicit treatment of PCIe contention, and a commitment to open-sourcing. However, the central guarantee is only as strong as the unstated determinism model and the un-derived interval formula, and the evaluation does not currently demonstrate tail-latency or statistical robustness. The headline memory and throughput ratios are presented as general results even though they are per-configuration or best-case numbers, which overstates the empirical support. The contribution is potentially significant for systems practice, but the paper needs a careful revision of its formal analysis and evaluation reporting.

major comments (6)
  1. [4.4] The expression L_offload = floor(t_compute*(1+delta)/t_trans) and interval = floor(L/L_offload) is presented without a derivation, and the definition of delta ('the SLO quotient over the computation time without offloading') is ambiguous. As written, the formula does not follow from the stated timing model: if no-offload total compute time is L*t_compute and the SLO is a multiple of it, the number of offloadable transfers should be bounded by the SLO slack divided by t_trans, yielding an L_offload that depends on L, whereas the paper's expression makes L_offload independent of L. Please derive the formula from a precise timing model, define delta explicitly, and state what happens when t_trans > t_compute*(1+delta), where the floor gives zero and the subsequent interval formula becomes undefined.
  2. [4.3 / Figure 7] The mechanism inserts a synchronization point before each offloaded layer and performs both a host-to-GPU load and a GPU-to-host store per offloaded layer. The critical-path cost therefore includes synchronization overhead and possible contention between the load and store streams, yet the offline record appears to account only for a single CPU-to-GPU transfer time t_trans. Please specify how the interval formula accounts for these overheads, or present experiments showing that they are negligible for the evaluated models and batch sizes.
  3. [4.3-4.5] The SLO guarantee rests on treating per-layer compute time as deterministic and stationary, but the paper provides no variance or repetition data for t_compute, and the online coordinator adjusts only for PCIe bandwidth contention, not for compute-time drift from GPU clock throttling, thermal state, or other sources. Given the claim that Select-N 'consistently meets SLOs,' please report the distribution of per-layer compute times and of end-to-end latencies over repeated runs (e.g., p50 and p99), and either extend the coordinator to handle compute-time drift or state explicitly the conditions under which the deterministic assumption holds.
  4. [5.3 / Figure 10] The headline '2.37x more host memory' is not an aggregate result. In Figure 10(a), the Select-N/FlexGen memory ratio is about 1.66 at batch size 4, 1.62 at batch size 8, 2.37 at batch size 16, and 2.15 at batch size 32; only the batch-16 point matches the 2.37x figure. Similarly, the '1.85x throughput' is described in Section 5.3 as the best case but is presented in the abstract and introduction as a general result. Please report per-configuration numbers accurately and qualify the abstract and introduction accordingly.
  5. [5.2 / Figures 9, 12, 13] SLO compliance is a tail-latency property, but the evaluation reports only point values with no error bars, percentiles, repetition counts, or confidence intervals for the latency measurements. This is a load-bearing issue for the central claim of 'consistently meets SLOs.' Please add statistical summaries across multiple runs for the SLO experiments, at minimum p50 and p99 for TTFT and TPOT in Figures 9, 12, and 13.
  6. [5.1] The workload is generated by a 'randomly designed dataloader,' and the assertion that real-world workloads such as ShareGPT can equally be supported is not demonstrated. Since the paper's contribution is a production-oriented SLO guarantee, please add at least one trace-based or real-workload evaluation, or clearly scope the claims to synthetic workloads.
minor comments (6)
  1. [4.2] The workflow subtracts waiting time from the SLO, but the paper does not describe how waiting time is measured or bounded; please clarify the queueing model used for this adjustment.
  2. [4.4] The statement that 'if their product exceeds a certain threshold, the optimal offloading interval becomes 1' is not justified, and the threshold is not defined; without this, the claim that at most 100 pairs need sampling is hard to verify.
  3. [5.5 / Figure 12] The figure legend includes 'Select-N-OPT' and 'Select-N-LLaMA' curves, but the text discusses only the FlexGen-OPT comparison; please clarify what each curve represents and how the LLaMA task's SLO is evaluated.
  4. [5.5] The text says that it is not feasible to run a naive no-offloading baseline for models that exceed GPU memory, yet the section still reports SLO compliance against a 100 ms TPOT threshold; please state explicitly that this threshold is an external choice rather than a percentage of naive execution time.
  5. [Various] The manuscript contains several typos, including 'basd' in Section 4.2, 'Obvervation' in Sections 3.2 and 3.3, 'seperation' in Section 4.4, and 'amout' in Section 2.2; please proofread carefully.
  6. [References] References [28] and [29] are duplicated; please remove the duplicate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Select-N's interval is an explicit SLO-tuned knob, and its memory/throughput gains are independent empirical comparisons.

full rationale

No load-bearing circular step is present. Select-N's interval-selection formula in §4.4 is an explicit design rule: given measured per-layer compute time t_compute, transfer time t_trans, and the target SLO encoded in δ, it computes the smallest offloading interval that should keep the transfer hidden within compute time plus SLO slack. The SLO-compliance measurements in §5.2 are a check that the implementation honors this rule on the same hardware, not a scientific prediction derived from the same fitted values; the interval choice is a tuning mechanism, not a claimed first-principles result. The substantive comparative claims—2.37× more host memory and 1.85× throughput over FlexGen—are empirical outcomes of a head-to-head evaluation, not consequences of the tuning formula itself. The paper's determinism assumption is a genuine correctness risk (the evaluation reports no variance or percentiles, and the coordinator only adjusts for PCIe contention, not compute-time drift), but that is an empirical/robustness concern, not circularity. No self-citation is load-bearing; the only same-group citation (Llumnix, [62]) appears in related work and does not justify the design or the interval-selection rule.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The system rests on a small set of empirical assumptions about hardware behavior; none of them are new physical entities. The main risk is that the deterministic compute model and the transfer-model formula are asserted rather than demonstrated with variance data.

assumptions (3)
  • domain assumption The computation time of each decoder layer is deterministic and identical across layers and iterations for a fixed batch size and sequence length.
    Stated in Section 4.3 as the basis for offline profiling; no repeated-measurement variance data is provided, and real GPU clocks or contention can violate it.
  • domain assumption Offline measurements of per-layer compute and transfer time on a dedicated server transfer to online servers aside from PCIe contention.
    Section 4.4 uses these measurements to build the performance record; thermal, clock, and neighbor workload effects are not quantified.
  • ad hoc to paper The formula L_offload = floor(t_compute(1+delta)/t_trans) and interval = floor(L/L_offload) correctly characterizes SLO feasibility.
    Section 4.4 introduces this relation without a derivation; its dimension and boundary behavior are unclear, including division by zero when L_offload is zero.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Memory Offloading for Large Language Model Inference with Latency SLO Guarantees." pith.science (2026). https://pith.science/paper/DWL4V6KA

@misc{pith2026250208182,
  author       = {Pith},
  title        = {Pith review of: Memory Offloading for Large Language Model Inference with Latency SLO Guarantees},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DWL4V6KA}},
  note         = {Machine review of arXiv:2502.08182}
}
read the original abstract

Offloading large language models (LLMs) state to host memory during inference promises to reduce operational costs by supporting larger models, longer inputs, and larger batch sizes. However, the design of existing memory offloading mechanisms does not take latency service-level objectives (SLOs) into consideration. As a result, they either lead to frequent SLO violations or underutilize host memory, thereby incurring economic loss and thus defeating the purpose of memory offloading. This paper presents Select-N, a latency-SLO-aware memory offloading system for LLM serving. A key challenge in designing Select-N is to reconcile the tension between meeting SLOs and maximizing host memory usage. Select-N overcomes it by exploiting a unique characteristic of modern LLMs: during serving, the computation time of each decoder layer is deterministic. Leveraging this, Select-N introduces offloading interval, an internal tunable knob that captures the tradeoff between SLOs and host memory usage, thereby reducing the aforementioned challenge to pick an optimal offloading interval. With that, Select-N proposes a two-stage approach to automatically pick the offloading interval. The first stage is offline that generates the range of optimal offloading interval, while the second stage adjusts offloading interval at the granularity of inference iteration based on runtime hardware status. Our evaluation shows that Select-N consistently meets SLOs and improves the serving throughput over existing mechanisms by 1.85X due to maximizing the use of host memory.

Figures

Figures reproduced from arXiv: 2502.08182 by the authors.

Figure 1
Figure 1. Memory demands of modern LLMs using float16 preci￾sion, with input sequences of 2,048 tokens. The grey dashed line represents the GPU memory capacity (80GB) of the NVIDIA A100. ple, with production chatbot services, one latency SLO is that each token must be generated within hundreds of mil￾liseconds to ensure timely feedback that aligns with human reading speeds [26]. Failing to meet the SLO would cause the users t… view at source ↗
Figure 2
Figure 2. (a) Serving latency (normalized by the target SLO) with DeepSpeed. (b) The average computation and transfer time for a single layer. Model: Qwen2-beta-7B, sequence length: 256, batch size: 4. 4 8 16 32 0 100 200 300 Batch Size Throughput (tokens/sec) Select-N DeepSpeed [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Throughput of Select-N and DeepSpeed with varying batch sizes. Model: Qwen2-beta-7B resulting in frequent SLO violations for all evaluated models. In addition to latency, since the GPU waits for the data most of the time, such an approach also reduces throughput by up to 8.2×, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The actual serving latency vs. the one estimated by Flex￾Gen. Model: OPT-13B. 8 32 0 2 4 6 8 10 12 7.77 11.97 4.80 5.57 Batch Size Memory Usage (GB) (a) Memory Usage on Offloading Devices Select-N FlexGen 8 32 0 10 20 30 40 50 60 21.97 57.14 11.85 35.46 Batch Size Thro…
Figure 5
Figure 5. Figure 5: Comparison of Select-N and FlexGen in (a) Memory usage on the offloading devices and (b) Throughput. Model: OPT￾13B. current system status. Therefore, to avoid SLO violations, with FlexGen, one has to conservatively estimate for the worst case, making FlexGen underutil…
Figure 6
Figure 6. Figure 6: The architecture and workflow of Select-N. floading interval. 3 The offloading interval and the requests are passed to the bandwidth coordinator, which generates an adjusted offloading interval for each GPU instance that shares the bus considering their bandwidth utili…
Figure 7
Figure 7. Figure 7: An overview of how a memory manager works. The upper part represents the compute stream, where each block denotes a layer with pink blocks being the offloaded layers. The lower part is the copy stream. The two streams execute in parallel, with "S" meaning synchronizati…
Figure 8
Figure 8. Figure 8: The adjustment algorithm performed by the coordinator. throughput during inference. We encapsulated Select-N into a Python library that al￾lows seamless integration with Transformer-based models. This library dynamically manages the decoder layers of the model, enablin…
Figure 10
Figure 10. Figure 10: presents the results of the comparison. FlexGen’s memory-saving capability is consistently inferior to that of Select-N at the same batch size due to inaccuracies in its estimation of transfer and computation latencies, resulting in suboptimal offloading decisions. In…
Figure 9
Figure 9. Figure 9: Comparison of TTFT and TPOT between Select-N and DeepSpeed under the OPT-6.7B and Qwen2-beta-7B models. The y-axis represents the ratio of the observed latency to the corre￾sponding SLO target latency, where a value of 1 indicates that the latency matches the SLO targe…
Figure 11
Figure 11. Figure 11: The TTFT, TPOT, and memory usage of Select-N under different offloading interval configurations. The red dashed lines represent the SLO. The optimal offloading interval is 3 in (a) and 8 in (b). 8 16 32 0 20 40 60 80 100 120 Batch Size TPOT (ms) Select-N-OPT Select-N-…
Figure 14
Figure 14. Figure 14: Maximum prompt length the model can process under different offloading interval settings. The dashed line represents the maximum length in the naive mode. balance compute and memory resources. These two works cannot handle models that exceed the GPU memory capacity, m…
Figure 13
Figure 13. Figure 13: TPOT of OPT-13B and LLaMA-13B models using Select-N. like vLLM, and is designed to work in parallel with other approaches to further enhance performance and resource efficiency. Offloading techniques. Existing works have explored various techniques to improve large-sc…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

87 extracted references · 43 canonical work pages

  1. [1]

    URL https://huggingface.co/ docs/accelerate/index

    Hugging face accelerate., 2025. URL https://huggingface.co/ docs/accelerate/index

    2025

  2. [2]

    URL https://sharegpt.com/

    Sharegpt teams., 2025. URL https://sharegpt.com/

    2025

  3. [3]

    Acharya, B

    A. Acharya, B. Singh, and N. Onoe. Llm based generation of item- description for recommendation system. In Proceedings of the 17th ACM Conference on Recommender Systems, RecSys ’23, page 1204–1207, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702419. doi: 10.1145/3604915.3610647. URL https: //doi.org/10.1145/3604915.3610647

    work page arXiv 2023

  4. [4]

    SYMPHONY: Improving Memory Management for LLM Inference Workloads

    S. Agarwal, A. Mao, A. Akella, and S. Venkataraman. Symphony: Im- proving memory management for llm inference workloads, 2024. URL https://arxiv.org/abs/2412.16434

    work page Pith review arXiv 2024

  5. [5]

    Agrawal, A

    A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, and R. Ram- jee. Sarathi: Efficient llm inference by piggybacking decodes with chun- ked prefills, 2023. URL https://arxiv.org/abs/2308.16369

    arXiv 2023

  6. [6]

    Alizadeh, I

    K. Alizadeh, I. Mirzadeh, D. Belenko, K. Khatamifard, M. Cho, C. C. D. Mundo, M. Rastegari, and M. Farajtabar. Llm in a flash: Efficient large language model inference with limited memory, 2024. URL https://arxiv.org/abs/2312.11514

    arXiv 2024

  7. [7]

    Alomari, N

    A. Alomari, N. Idris, A. Q. M. Sabri, and I. Alsmadi. Deep reinforcement and transfer learning for abstractive text summa- rization: A review. Computer Speech & Language , 71:101276,

  8. [8]

    R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley, and Y. He. Deepspeed- infer- ence: Enabling efficient inference of transformer models at unprece- dented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2022. doi: 10.1109/SC4...

    arXiv 2022

Show all 87 references
  1. [9]

    Bambhaniya, R

    A. Bambhaniya, R. Raj, G. Jeong, S. Kundu, S. Srinivasan, M. Elavazha- gan, M. Kumar, and T. Krishna. Demystifying platform requirements for diverse llm inference use cases, 2024. URL https://arxiv.org/ abs/2406.01698

    2024 arXiv

  2. [10]

    Cao and L

    S. Cao and L. Wang. Awesome: Gpu memory-constrained long doc- ument summarization using memory mechanism and global salient content, 2023. URL https://arxiv.org/abs/2305.14806

    2023 arXiv

  3. [11]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Ka- plan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter...

    2021 arXiv

  4. [12]

    S. Chen, Z. Guan, Y. Liu, and P. B. Gibbons. Practical offloading for fine-tuning llm on commodity gpu via learned subspace projectors,

  5. [13]

    Cheng, Y

    R. Cheng, Y. Peng, Y. Lai, X. Wei, R. Chen, and H. Chen. Kunserve: Elastic and efficient large language model serving with parameter- centric memory management, 2024. URL https://arxiv.org/abs/ 2412.18169

    2024

  6. [14]

    Choquette, W

    J. Choquette, W. Gandhi, O. Giroux, N. Stam, and R. Krashinsky. Nvidia a100 tensor core gpu: Performance and innovation. IEEE Micro, 41(2): 29–35, 2021. doi: 10.1109/MM.2021.3061394

    2021

  7. [15]

    Crankshaw, X

    D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonza- lez, and I. Stoica. Clipper: A Low-Latency online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17) , pages 613–627, Boston, MA, Mar. 2017. USENIX Associatio...

    2017

  8. [16]

    Crankshaw, G.-E

    D. Crankshaw, G.-E. Sela, X. Mo, C. Zumar, I. Stoica, J. Gonzalez, and A. Tumanov. Inferline: latency-aware provisioning and scaling for pre- diction serving pipelines. In Proceedings of the 11th ACM Symposium on Cloud Computing , SoCC ’20, page 477–491, New York, NY, USA,

  9. [17]

    S. K. Dam, C. S. Hong, Y. Qiao, and C. Zhang. A complete survey on llm-based ai chatbots, 2024. URL https://arxiv.org/abs/2406. 16937

    2024

  10. [18]

    T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxiv.org/abs/2307. 08691

    2023

  11. [19]

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 16344–...

    2022

  12. [20]

    Donthi, M

    S. Donthi, M. Spencer, O. Patel, J. Doh, and E. Rodan. Improving llm abilities in idiomatic translation, 2024. URL https://arxiv.org/ abs/2407.03518

    2024 arXiv

  13. [21]

    J. Duan, S. Zhang, Z. Wang, L. Jiang, W. Qu, Q. Hu, G. Wang, Q. Weng, H. Yan, X. Zhang, X. Qiu, D. Lin, Y. Wen, X. Jin, T. Zhang, and P. Sun. Ef- ficient training of large language models on distributed infrastructures: A survey, 2024. URL https://arxiv.org/abs/2407.20018

    2024 arXiv

  14. [22]

    Elliott, M

    M. Elliott, M. Luciano, and J. Fortes. Integrating large language models and the idigbio portal for conversational data exploration and retrieval. Biodiversity Information Science and Standards , 8:e142696, 2024

    2024

  15. [23]

    Gambhir and V

    M. Gambhir and V. Gupta. Recent automatic text summarization techniques: a survey. Artificial Intelligence Review, 47(1):1–66, 2017

    2017

  16. [24]

    B. Gao, Z. He, P. Sharma, Q. Kang, D. Jevdjic, J. Deng, X. Yang, Z. Yu, and P. Zuo. Cost-Efficient large language model serv- ing for multi-turn conversations with CachedAttention. In 2024 USENIX Annual Technical Conference (USENIX ATC 24) , pages 111– 126, Santa Clara, CA, Ju...

    2024

  17. [25]

    S. Gao, Y. Chen, and J. Shu. Fast state restoration in llm serving with hcache, 2024. URL https://arxiv.org/abs/2410.05004

    2024 arXiv

  18. [26]

    Griggs, X

    T. Griggs, X. Liu, J. Yu, D. Kim, W.-L. Chiang, A. Cheung, and I. Stoica. Mélange: Cost efficient large language model serving by exploiting gpu heterogeneity, 2024. URL https://arxiv.org/abs/2404.14527

    2024 arXiv

  19. [27]

    Q. Gu. Llm-based code generation method for golang compiler testing. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ESEC/FSE 2023, page 2201–2203, New York, NY, USA, 2023. Association for...

    2023

  20. [28]

    Gujarati, R

    A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kaufmann, Y. Vigfusson, and J. Mace. Serving DNNs like clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Sys- tems Design and Implementation (OSDI 20) , pages 443–462. USENIX Associati...

    2020

  21. [29]

    Gujarati, R

    A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kaufmann, Y. Vigfusson, and J. Mace. Serving DNNs like clockwork: Performance predictability 14 from the bottom up. In 14th USENIX Symposium on Operating Sys- tems Design and Implementation (OSDI 20) , pages 443–462. USENIX Associ...

    2020

  22. [30]

    Holmes, M

    C. Holmes, M. Tanaka, M. Wyatt, A. A. Awan, J. Rasley, S. Rajbhan- dari, R. Y. Aminabadi, H. Qin, A. Bakhtiari, L. Kurilenko, and Y. He. Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference, 2024. URL https://arxiv.org/abs/ 2401.08671

    2024 arXiv

  23. [31]

    S. Hong, Y. Lin, B. Liu, B. Liu, B. Wu, C. Zhang, C. Wei, D. Li, J. Chen, J. Zhang, J. Wang, L. Zhang, L. Zhang, M. Yang, M. Zhuge, T. Guo, T. Zhou, W. Tao, X. Tang, X. Lu, X. Zheng, X. Liang, Y. Fei, Y. Cheng, Z. Gou, Z. Xu, and C. Wu. Data interpreter: An llm agent for data ...

    2024 arXiv

  24. [33]

    C. Hu, H. Huang, L. Xu, X. Chen, J. Xu, S. Chen, H. Feng, C. Wang, S. Wang, Y. Bao, N. Sun, and Y. Shan. Inference without interference: Disaggregate llm inference for mixed downstream workloads, 2024. URL https://arxiv.org/abs/2401.11181

    2024 arXiv

  25. [34]

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

    2021 arXiv

  26. [35]

    Jayaram Subramanya, D

    S. Jayaram Subramanya, D. Arfeen, S. Lin, A. Qiao, Z. Jia, and G. R. Ganger. Sia: Heterogeneity-aware, goodput-optimized ml-cluster scheduling. In Proceedings of the 29th Symposium on Operating Systems Principles , SOSP ’23, page 642–657, New York, NY, USA,

  27. [36]

    Jiang, Y

    X. Jiang, Y. Zhou, S. Cao, I. Stoica, and M. Yu. Neo: Saving gpu mem- ory crisis with cpu offloading for online llm inference, 2024. URL https://arxiv.org/abs/2411.01142

    2024 arXiv

  28. [37]

    H. Jin, R. Lai, C. F. Ruan, Y. Wang, T. C. Mowry, X. Miao, Z. Jia, and T. Chen. A system for microserving of llms, 2024. URL https: //arxiv.org/abs/2412.12488

    2024 arXiv

  29. [38]

    Y. Jin, T. Wang, H. Lin, M. Song, P. Li, Y. Ma, Y. Shan, Z. Yuan, C. Li, Y. Sun, T. Wu, X. Chu, R. Huan, L. Ma, X. You, W. Zhou, Y. Ye, W. Liu, X. Xu, Y. Zhang, T. Dong, J. Zhu, Z. Wang, X. Ju, J. Song, H. Cheng, X. Li, J. Ding, H. Guo, and Z. Zhang. P/d-serve: Serving disaggr...

    2024 arXiv

  30. [39]

    Kasner and O

    Z. Kasner and O. Dusek. Beyond traditional benchmarks: Analyz- ing behaviors of open LLMs on data-to-text generation. In L.-W. Ku, A. Martins, and V. Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2024 doi

  31. [40]

    Koshkin, K

    R. Koshkin, K. Sudoh, and S. Nakamura. Transllama: Llm-based simul- taneous translation system, 2024. URL https://arxiv.org/abs/ 2402.04636

    2024 arXiv

  32. [41]

    Koziolek, S

    H. Koziolek, S. Grüner, R. Hark, V. Ashiwal, S. Linsbauer, and N. Es- kandani. Llm-based and retrieval-augmented control code genera- tion. In Proceedings of the 1st International Workshop on Large Lan- guage Models for Code, LLM4Code ’24, page 22–29, New York, NY, USA,

  33. [42]

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gon- zalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles , SOSP ’23, page 611–626, New...

    2023

  34. [43]

    W. Lee, J. Lee, J. Seo, and J. Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache man- agement. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages 155–172, Santa Clara, CA, July

  35. [44]

    C. Li, Y. Sun, L. Jin, L. Xu, Z. Cao, P. Fan, D. Kaeli, S. Ma, Y. Guo, and J. Yang. Priority-based pcie scheduling for multi-tenant multi-gpu systems. IEEE Computer Architecture Letters, 18(2):157–160, 2019. doi: 10.1109/LCA.2019.2955119

    2019

  36. [45]

    Li and Y

    K. Li and Y. Zhang. Planning first, question second: An LLM-guided method for controllable question generation. In L.-W. Ku, A. Mar- tins, and V. Srikumar, editors, Findings of the Association for Com- putational Linguistics: ACL 2024 , pages 4715–4729, Bangkok, Thai- land, Au...

    2024 doi

  37. [46]

    ISBN 9798400705793

    Association for Computing Machinery. ISBN 9798400705793. doi: 10.1145/3643795.3648384. URL https://doi.org/10.1145/ 3643795.3648384

  38. [47]

    B. Lin, C. Zhang, T. Peng, H. Zhao, W. Xiao, M. Sun, A. Liu, Z. Zhang, L. Li, X. Qiu, S. Li, Z. Ji, T. Xie, Y. Li, and W. Lin. Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache,

  39. [48]

    S.-C. Liu, S. Wang, W. Lin, C.-W. Hsiung, Y.-C. Hsieh, Y.-P. Cheng, S.-H. Luo, T. Chang, and J. Zhang. Jarvix: A llm no code plat- form for tabular data analysis and optimization, 2023. URL https: //arxiv.org/abs/2312.02213

    2023 arXiv

  40. [49]

    ISBN 978-1-939133-40-3

    USENIX Association. ISBN 978-1-939133-40-3. URL https: //www.usenix.org/conference/osdi24/presentation/lee

  41. [50]

    P. Ma, R. Ding, S. Wang, S. Han, and D. Zhang. InsightPilot: An LLM-empowered automated data exploration system. In Y. Feng and E. Lefever, editors, Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing: System Demonstra- tions, pages 346–352...

    2023 doi

  42. [51]

    Patel, E

    P. Patel, E. Choukse, C. Zhang, A. Shah, Í. Goiri, S. Maleki, and R. Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Sym- posium on Computer Architecture (ISCA) , pages 118–132, 2024. doi: 10.1109/ISCA5...

    2024

  43. [52]

    Z. Li, L. Zheng, Y. Zhong, V. Liu, Y. Sheng, X. Jin, Y. Huang, Z. Chen, H. Zhang, J. E. Gonzalez, and I. Stoica. AlpaServe: Statistical multiplex- ing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSD...

    2023

  44. [53]

    R. Peng, K. Liu, P. Yang, Z. Yuan, and S. Li. Embedding-based retrieval with llm for effective agriculture information extracting from unstruc- tured data, 2023. URL https://arxiv.org/abs/2308.03107

    2023 arXiv

  45. [54]

    URL https://arxiv.org/abs/2401.02669

    arXiv

  46. [55]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 (8):9, 2019

    2019

  47. [56]

    Y. Lu, W. Zhu, L. Li, Y. Qiao, and F. Yuan. Llamax: Scaling linguis- tic horizons of llm by enhancing translation capabilities beyond 100 languages, 2024. URL https://arxiv.org/abs/2407.05975

    2024 arXiv

  48. [57]

    H. Shen, L. Chen, Y. Jin, L. Zhao, B. Kong, M. Philipose, A. Krish- namurthy, and R. Sundaram. Nexus: a gpu cluster engine for ac- celerating dnn-based video analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles , SOSP ’19, page 322–337, New York, ...

    2019

  49. [58]

    Sheng, L

    Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Re, I. Stoica, and C. Zhang. FlexGen: High-throughput gener- ative inference of large language models with a single GPU. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scar- lett, edit...

    2023

  50. [59]

    Patke, D

    A. Patke, D. Reddy, S. Jha, H. Qiu, C. Pinto, C. Narayanaswami, Z. Kalbarczyk, and R. Iyer. Queue management for slo-oriented large language model serving. In Proceedings of the 2024 ACM Sympo- sium on Cloud Computing , SoCC ’24, page 18–35, New York, NY, USA,

    2024

  51. [60]

    ISBN 9798400712869

    Association for Computing Machinery. ISBN 9798400712869. doi: 10.1145/3698038.3698523. URL https://doi.org/10.1145/ 3698038.3698523

  52. [61]

    Strati, S

    F. Strati, S. Mcallister, A. Phanishayee, J. Tarnawski, and A. Klimovic. Déjàvu: Kv-cache streaming for fast, fault-tolerant generative llm serv- ing, 2024. URL https://arxiv.org/abs/2403.01876

    2024 arXiv

  53. [62]

    A. Qiao, S. K. Choe, S. J. Subramanya, W. Neiswanger, Q. Ho, H. Zhang, G. R. Ganger, and E. P. Xing. Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning. In 15th USENIX Symposium on 15 Operating Systems Design and Implementation (OSDI 21) , pages 1–18. U...

    2021

  54. [63]

    Taori, I

    R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: an instruction-following llama model (2023). URL https://github. com/tatsu-lab/stanford_alpaca, 1(9), 2023

    2023

  55. [64]

    J. Ren, S. Rajbhandari, R. Y. Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, and Y. He. ZeRO-Offload: Democratizing Billion-Scale model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21) , pages 551–564. USENIX Association, July 2021. ISBN 978-1-939133-23-6....

    2021

  56. [65]

    Ugare, T

    S. Ugare, T. Suresh, H. Kang, S. Misailovic, and G. Singh. Syncode: Llm generation with grammar augmentation, 2024. URL https: //arxiv.org/abs/2403.01632

    2024 arXiv

  57. [66]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need.(nips), 2017. arXiv preprint arXiv:1706.03762, 10:S0140525X16001837, 2017

    2017 arXiv

  58. [67]

    M. Side, F. Yao, and Z. Zhang. Lockeddown: Exploiting contention on host-gpu pcie bus for fun and profit. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P) , pages 270–285, 2022. doi: 10.1109/EuroSP53844.2022.00025

    2022

  59. [68]

    Sivakumar

    S. Sivakumar. Performance optimization of large language models (llms) in web applications. International Journal of Advanced Scientific Research, 8:1077–1096, 02 2024

    2024

  60. [69]

    G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks, 2024. URL https://arxiv. org/abs/2309.17453

    2024 arXiv

  61. [70]

    B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin. Llum- nix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 173–191, Santa Clara, CA, July 2024. USENIX Asso- ciation. ...

    2024

  62. [71]

    Xiong, H

    Y. Xiong, H. Wu, C. Shao, Z. Wang, R. Zhang, Y. Guo, J. Zhao, K. Zhang, and Z. Pan. Layerkv: Optimizing large language model serving with layer-wise kv cache management, 2024. URL https://arxiv.org/ abs/2410.00428

    2024 arXiv

  63. [72]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hoss...

    2023 arXiv

  64. [73]

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun. Orca: A dis- tributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 22), pages 521–538, Carlsbad, CA, July 2022. USENIX Associati...

    2022

  65. [74]

    Z. Yuan, Y. Shang, Y. Zhou, Z. Dong, Z. Zhou, C. Xue, B. Wu, Z. Li, Q. Gu, Y. J. Lee, Y. Yan, B. Chen, G. Sun, and K. Keutzer. Llm in- ference unveiled: Survey and roofline model insights, 2024. URL https://arxiv.org/abs/2402.16363

    2024 arXiv

  66. [75]

    Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Ha- jishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023. URL https://arxiv.org/abs/2212.10560

    2023 arXiv

  67. [76]

    B. Wu, Y. Zhong, Z. Zhang, S. Liu, F. Liu, Y. Sun, G. Huang, X. Liu, and X. Jin. Fast distributed inference serving for large language models,

  68. [77]

    URL https://arxiv.org/abs/2305.05920

    arXiv

  69. [78]

    Zhong, S

    Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages 193–210, Santa C...

    2024

  70. [79]

    Xiao and X

    L. Xiao and X. Chen. Enhancing llm with evolutionary fine tuning for news summary generation, 2023. URL https://arxiv.org/abs/ 2307.02839

    2023 arXiv

  71. [80]

    J. Zhu, Y. Zhou, J. Zhang, and C. Zong. Attend, translate and sum- marize: An efficient method for neural cross-lingual summarization. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors,Proceed- ings of the 58th Annual Meeting of the Association for Computational ...

    2020 doi

  72. [81]

    Xu and G

    R. Xu and G. Li. A comparative study of offline models and online llms in fake news detection. arXiv preprint arXiv:2409.03067, 2024

    2024 arXiv

  73. [84]

    Zhang, Y

    H. Zhang, Y. Tang, A. Khandelwal, and I. Stoica. SHEPHERD: Serving DNNs in the wild. In 20th USENIX Symposium on Net- worked Systems Design and Implementation (NSDI 23) , pages 787– 808, Boston, MA, Apr. 2023. USENIX Association. ISBN 978- 1-939133-33-5. URL https://www.usenix...

    2023

  74. [85]

    Zhang, S

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shus- ter, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL https:...

    2022 arXiv

  75. [86]

    P. Zhao, H. Zhang, F. Fu, X. Nie, Q. Liu, F. Yang, Y. Peng, D. Jiao, S. Li, J. Xue, Y. Tao, and B. Cui. Efficiently training 7b llm with 1 million sequence length on 8 gpus, 2024. URL https://arxiv.org/abs/ 2407.12117

    2024 arXiv

  76. [88]

    X. Zhou, X. Zhao, and G. Li. Llm-enhanced data management. arXiv preprint arXiv:2402.02643, 2024

    2024 arXiv

  77. [2020]

    ISBN 9781450381376

    Association for Computing Machinery. ISBN 9781450381376. doi: 10.1145/3419111.3421285. URL https://doi.org/10.1145/ 3419111.3421285

  78. [2022]

    doi: https://doi.org/10.1016/j.csl.2021

    ISSN 0885-2308. doi: https://doi.org/10.1016/j.csl.2021. 101276. URL https://www.sciencedirect.com/science/ article/pii/S0885230821000796

    2021 doi

  79. [2024]

    URL https://arxiv.org/abs/2406.10181

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.