Pith. sign in

REVIEW 4 major objections 4 minor 54 references

Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FASTLIBRA claims that storing each LoRA adapter together with its KV caches in a unified, dependency-ordered pool cuts time-to-first-token by 63.4% and raises peak throughput by 35.2%.

desk verdict A real dependency-aware caching idea for multi-LoRA serving, but the evaluation needs tightening before I'd trust the stated numbers. read the letter →

arxiv 2505.03756 v1 pith:F22MPUSC submitted 2025-04-19 cs.AR cs.AIcs.LGcs.PF

Hang Zhang , Jiuchen Shi , Yixiao Wang , Quan Chen , Yizhou Shan , Minyi Guo This is my paper · ORCID
classification cs.ARcs.AIcs.LGcs.PF
keywords multi-LoRAservingKVcachemanagementdependencytreeunifiedmemorypooltime-to-first-tokenLLMinferencelow-rankadaptersswappolicy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the main source of lost performance in multi-LoRA LLM serving is not raw HBM capacity but the way adapters and their key-value caches are cached in isolation. It proposes FASTLIBRA, which stores LoRA adapters and KV blocks in one block-wise pool organized as a dependency tree, so a prefix's KV blocks are present only when the LoRA that produces them is resident. A cost model then decides swap-in and swap-out in terms of expected benefit to time-to-first-token. On Llama-7B, 13B, and 34B models across chatbot, translation, and personal-agent workloads, the paper reports TTFT down 63.4%, TPOT down 40.1%, and peak throughput up 35.2% compared with two state-of-the-art Multi-LoRA serving systems. The stakes are practical: multi-LoRA serving is how many task-specific models run from one base model, and TTFT is what users experience as waiting time.

What carries the argument

The carrying object is a unified dependency tree over a block-wise memory pool. Every node, whether a LoRA adapter or a KV block, lives in equally sized HBM or main-memory blocks, with adapters split along the rank dimension to align with KV blocks; a virtual root connects independent LoRA subtrees, and prefix matching progresses depth-first so a KV block is usable only when its adapter and its earlier tokens are resident. Swap operations preserve this invariant by evicting leaves first and loading roots first. The identity that drives decisions is the cost model $\mathrm{Eval}_i = \mathrm{LoRA\_Eval}_i \times \mathrm{Retain\_Eval}_i$, where $\mathrm{LoRA\_Eval}_i = \max(1, L_{\mathrm{lora}}/N_{\mathrm{now}})$ with $L_{\mathrm{lora}} = \sum_i (1-(1-p_i)^{BS})$ computed from the last five seconds of batch composition, and $\mathrm{Retain\_Eval}_i = c_i p_i (1-\mathrm{sigmoid}(t_i))$ combines transfer cost, historical visit probability, and recency. The swapper re-evaluates this score every 100 ms and moves nodes in or out until HBM crosses its 95% and 70% thresholds.

What would settle it

Run FASTLIBRA on a workload where, after an hour of stable adapter popularity, the hottest adapter suddenly goes completely idle and a long-cold adapter receives all queries within one 100 ms window; if the cost model's frequency term still favors the old hot adapter and TTFT spikes toward the baseline level, the recency signal does not track real load dynamics.

Watch

Extended reading notes

Core claim

The central claim is that the usage dependency between a LoRA and its KV caches is the missing structure in existing caching. FASTLIBRA's dependency-aware cache manager builds a unified tree with a virtual root, LoRA nodes at the second layer, and KV cache nodes as descendants; queries match the LoRA first and then match prefixes by depth-first search. Because eviction removes only leaf nodes and swap-in adds only root nodes, the tree stays connected, so no KV block can remain in HBM after the adapter that produced it has been evicted. The performance-driven cache swapper ranks every candidate node with a cost model that multiplies a LoRA-quantity reward by an expected retention benefit combining transfer cost, visit frequency, and recency decay. The paper's reported result is a 63.4% average reduction in TTFT, a 40.1% reduction in TPOT, and a 35.2% improvement in peak serving throughput, with ablations showing that dependency tracking and the cost model each contribute separately.

Load-bearing premise

The cost model assumes that recent visit frequencies and the batch composition of the last five seconds predict the next moments of demand; if the query mix shifts faster than the 100 ms monitoring interval, eviction and prefetch decisions can be wrong and the claimed gains can degrade.

Editorial extensions

If this is right

  • Every cached KV block in the dependency tree is guaranteed to belong to a resident LoRA, which frees the HBM fraction that static systems waste on invalid entries; the paper measures up to 48.1% invalid KV caches in the baseline.
  • A unified block-wise pool lets HBM shift dynamically between adapters and KV blocks without redeployment, so load changes that exhaust a static LoRA partition or KV partition can be absorbed.
  • The swapper's LoRA-reward term keeps enough distinct adapters resident to cover the expected batch composition, reducing queueing and LoRA cold-start latency in dynamic scenarios.
  • The reported average gains follow directly: TTFT down 63.4%, TPOT down 40.1%, and peak throughput up 35.2% against state-of-the-art baselines.
  • Ablations show the two components are complementary: dropping dependency tracking raises TTFT by 1.27x, replacing the cost model with LRU raises it by 1.24x, and removing only the LoRA-quantity reward raises it by 1.13x.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the recency-and-frequency cost model would be stress-tested by abrupt workload reversals, where the hottest adapter of the past hour suddenly goes idle and a long-cold adapter receives all queries within one 100 ms window; the paper's traces are slowly varying, so they cannot separate the recency signal from a simpler reactive rule.
  • Editorial inference: the same scoring structure could be re-targeted from TTFT to end-to-end latency by weighting prefill and decode token counts differently, which would change which KV blocks the system keeps when conversations are long.
  • Editorial inference: since the cost model already quantifies transfer costs, the unified pool could extend naturally to heterogeneous memory tiers or disaggregated prefill and decode; the paper evaluates only a flat HBM-to-main-memory hierarchy.
  • Editorial inference: multiplying the LoRA-quantity reward by the retention score means that when the expected adapter count is far above the current count, even moderately cold adapters are favored for prefetching; the paper does not isolate whether that prefetch ever sacrifices KV hit rate during load spikes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents FASTLIBRA, a caching system for multi-LoRA LLM serving that manages LoRA adapters and KV caches in a unified HBM/main-memory pool. It maintains usage dependencies between each LoRA and its KV caches via a dependency tree, and it decides swap-in/swap-out actions with a periodic cost model based on visit frequency, recency, transfer cost, and a target number of loaded LoRAs (Eqs. 3-6). The system is implemented on top of vLLM and evaluated with Llama-7B/13B/34B on NPUs across chatbot, translation, and personal-agent workloads, using vLLM and S-LoRA as baselines. The paper reports large reductions in TTFT and TPOT and increases in peak throughput, with ablations for the dependency manager, the cost model, and the LoRA-count term.

Significance. If the reported gains hold, FASTLIBRA is a practical contribution to multi-LoRA serving: the dependency-tree idea directly addresses a real inefficiency, namely cached KVs that are unusable because their required LoRA is not resident. The paper has notable strengths: it evaluates three model sizes, three scenarios built from public traces, LoRA counts from 20 to 2000, two baselines, and it ablates each major component (WOM, WOS, WOL). The HBM-utilization and cache-hit-rate analyses provide plausible mechanism evidence for the gains. However, the headline numbers are not auditable from the per-baseline results, the cost model's predictive assumption is not tested in the dynamic regime that motivates the design, and some evaluation details (error bars, synthetic LoRAs, baseline modifications) need clarification. The central idea is defensible, but the current evidence does not yet support the headline claims as stated.

major comments (4)
  1. [Abstract, §8 vs §6.3] The abstract and conclusion state that FASTLIBRA reduces TTFT and TPOT by 63.4% and 40.1% on average and improves peak throughput by 35.2%, but §6.3 reports TTFT/TPOT reductions of 60.3%/33.9% against vLLM and 50.1%/28.6% against S-LoRA, with throughput ratios of 1.7x and 1.6x. No aggregation rule described in the paper turns the per-baseline numbers into the headline numbers; for example, averaging the two TTFT reductions gives 55.2%, not 63.4%. Because the central claim is quantitative, please report the exact aggregation formula or correct the abstract and conclusion to match §6.3.
  2. [§5, Eqs. (3)-(6); §6.2] The cost model is a predictor: prob_i is a historical visit frequency, BS is a 5-second batch average, t_i is the time since last use, and the swapper acts every 100 ms. The evaluation, however, never exercises the dynamic regime that motivates the design: the translation and agent traces take Azure Function timing but map functions to LoRAs statically, and the chatbot trace preserves LMSYS timing without constructing an abrupt change in the hot set. If popularity shifts faster than the 100 ms monitor interval or the 5 s batch window, Eval_i is based on stale statistics and can evict the newly hot LoRA's KVs or fail to preload the newly hot LoRA. Please add a workload with a sudden popularity shift (e.g., a synthetic hot-set switch) and a sensitivity analysis over the monitor interval, batch window, and sigmoid time scale, or explicitly narrow the claim to workloads whose popularity changes on the monitored timescale.
  3. [§5.2, Eqs. (4)-(6)] LoRA_Eval_i in Eq. (4) is a function of Lowlora and NowLoRA only and does not depend on i, yet Eq. (6) multiplies it into Eval_i for a generic 'KV cache or LoRA i'. If it is applied to KV nodes, then whenever NowLoRA is below the target, all KV caches receive an extra retention boost, which pushes against the stated goal of freeing HBM space for LoRAs; if it is intended to apply only to LoRA nodes, the equations should say so explicitly with an indicator. Also, NowLoRA appears in the denominator without a stated guard against NowLoRA=0. Please clarify the intended domain of LoRA_Eval_i and provide evidence, or a correction, that the combined scoring behaves as claimed.
  4. [§6.1, §6.3] The evaluation section does not state how many independent runs are averaged, and no error bars or variance information are given for TTFT, TPOT, or peak throughput; under queueing, these metrics are noisy, so the reported average improvements need a measure of dispersion. In addition, the LoRA adapters are random matrices ('parameters of the LoRAs are randomly generated using a normal distribution'), which is acceptable for a memory-management study only if the authors argue that adapter values do not affect the sizes, access patterns, or transfer costs being measured. Finally, the vLLM baseline is described as 'adapted' with an LRU-based swap-out policy; please specify exactly which components of vLLM were modified and confirm that S-LoRA was run unmodified, so the comparison is auditable.
minor comments (4)
  1. [Abstract and throughout] The abstract contains 'ELORA' where 'FASTLIBRA' is meant, and 'Time-To-First-Toke' should be 'Time-To-First-Token'; Section 1 has 'It is challenging to to balance'; and several figure captions use 'Pernonal Agents' instead of 'Personal Agents'.
  2. [§1, §2.3.1, §6.6] The invalid-KV statistics are inconsistent: the Introduction says vLLM suffers 'up to 46.5% invalid KV caches', §2.3.1 says '48.1% invalid KV caches on average', and §6.6 reports an average of 48.6% for FASTLIBRA-WOM. Please reconcile these numbers and state whether they refer to the same measurement.
  3. [§7] The related-work text attributes ChunkAttention to reference [53], but [53] is the SGLang paper; please provide the correct citation for ChunkAttention and avoid relying on a single reference for two distinct systems.
  4. [§5.1] Equation (3) is described as computing 'the expected number of LoRAs required for inference', which is correct if queries are sampled with replacement; please state this assumption explicitly, since the subsequent cost model is sensitive to it.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: FASTLIBRA's cost model is a stated heuristic evaluated on external traces and ablations; the reported inconsistencies are auditability issues, not constructional reductions.

full rationale

FASTLIBRA's derivation chain is self-contained and its headline results are benchmarked, not fitted. The cost model in Eqs. (3)-(6) is a stated heuristic: prob_i, BS, and t_i are runtime observations from the dependency tree; no parameters are calibrated against the reported TTFT/TPOT numbers, and thresholds such as 95%/70% HBM usage and the 100 ms monitor interval are specified before evaluation rather than chosen to reproduce the measured gains. The evaluation compares against external systems (vLLM, S-LoRA) on external traces (LMSYS-33k, OPUS-100, Taskmaster, Azure Function trace), and the ablations FASTLIBRA-WOM, -WOS, and -WOL show that each component contributes to the measured gains, which is the opposite of a circular reduction: the target result is not an input to the design. The claim that the dependency tree eliminates invalid KV caches is true by the tree invariant (swaps start at leaves, LoRAs are second-layer nodes), but that invariant is a system property, not the source of the time savings by itself; the time savings are evidenced by comparative latency experiments. No load-bearing uniqueness claim is imported from author-overlapping prior work: the intro citation to Caraserve [24] is contextual, not foundational. The abstract's 'ELORA' typo and the mismatch between the headline 63.4%/40.1% averages and the body's per-baseline numbers (60.3%/33.9% vs vLLM; 50.1%/28.6% vs S-LoRA; 1.7x/1.6x throughput) are reporting and auditability defects that should be examined as correctness risk, but they are not instances of a prediction reducing to its inputs by construction. The untested assumption that recent history predicts near-future popularity is an experimental gap, not a circularity, because the cost model is not claimed to be derived from the measured improvement.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced; the virtual root of the dependency tree is a data structure element, not a postulated phenomenon. The central claim rests on hand-set caching thresholds, a heuristic cost model, and the domain assumptions above.

free parameters (5)
  • HBM upper threshold = 95%
    Cache swapper triggers swap-out when HBM usage exceeds 95 percent; set in Section 5.3 without sensitivity analysis.
  • HBM lower threshold = 70%
    Swap-in is triggered below 70 percent to avoid ping-pong swapping; chosen in Section 5.3 with no sensitivity study.
  • monitor interval = 100ms
    Period at which the cache swapper re-evaluates swap decisions; set in Section 5.3 with no sensitivity analysis.
  • batch size window = 5 seconds
    Equation 3 uses the batch size from the last 5 seconds to estimate the target LoRA count; window length is hand-chosen in Section 5.1.
  • sigmoid time scale = implicitly 1 second
    Equation 5 uses 1 minus sigmoid(t_i) with t_i in seconds and no explicit scale, so the decay rate is a hidden choice.
assumptions (4)
  • domain assumption LoRA-specific KV caches: each LoRA modifies Q/K projections so KV caches are adapter-specific (Equation 2).
    Taken from LoRA and transformer background; this makes the dependency tree meaningful.
  • domain assumption A query cannot start inference until its LoRA is in HBM, so KV caches without their LoRA are invalid.
    Used in the motivation in Section 2.3.1 and Figure 3; consistent with vLLM and S-LoRA behavior, though not formally proven for all serving engines.
  • domain assumption Historical visit frequency and recency predict future query demand.
    Core of the cost model in Equations 3 to 5; not validated for sudden workload shifts.
  • domain assumption Swap cost is linear in node size and PCIe bandwidth.
    Used in Equation 5 and Section 5.2; ignores queuing, overlap, and other transfer effects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management." pith.science (2026). https://pith.science/paper/F22MPUSC

@misc{pith2026250503756,
  author       = {Pith},
  title        = {Pith review of: Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F22MPUSC}},
  note         = {Machine review of arXiv:2505.03756}
}
read the original abstract

Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA adapters in high bandwidth memory of accelerations can improve inference performance. However, existing Multi-LoRA inference systems fail to optimize serving performance like Time-To-First-Toke (TTFT), neglecting usage dependencies when caching LoRAs and KVs. We therefore propose FASTLIBRA, a Multi-LoRA caching system to optimize the serving performance. FASTLIBRA comprises a dependency-aware cache manager and a performance-driven cache swapper. The cache manager maintains the usage dependencies between LoRAs and KV caches during the inference with a unified caching pool. The cache swapper determines the swap-in or out of LoRAs and KV caches based on a unified cost model, when the HBM is idle or busy, respectively. Experimental results show that ELORA reduces the TTFT by 63.4% on average, compared to state-of-the-art works.

Figures

Figures reproduced from arXiv: 2505.03756 by the authors.

Figure 1
Figure 1. An example caching state to show the Usage De￾pendencies between LoRAs and KV caches. KVs of a LoRA may also prevent useful KVs of other LoRAs from being cached. For instance, KV2-2 is not cached while LoRA-1’s KVs are invalid, preventing queries of LoRA-2 from running. Our experiments show that vLLM [22] suffers from up to 46.5% invalid KV caches. In the inter-LoRA aspect, the required number of LoRAs and the hotne… view at source ↗
Figure 2
Figure 2. The TTFT of vLLM for various scenarios. Trace [35, 50] to adapt its query arriving timing information. Moreover, we use Llama-7B/13B/34B as the base models, and conduct the experiments on four High-performance NPUs. Ta￾ble 1 summarizes the hardware and software configurations [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples of serving two queries under: (a) without usage dependencies, and (b) maintaining usage dependencies. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: The utilization rate of the HBM space allocated to [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The relationships among the visited frequency, swap [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Design overview of FASTLIBRA. 3 FASTLIBRA Methodology In this section, we summarize the challenges of FASTLIBRA, and introduce the overview of FASTLIBRA. 3.1 Challenges of FASTLIBRA According to the above analysis, two technical challenges should be addressed to resolv…
Figure 7
Figure 7. Figure 7: The constructing process of the usage dependencies [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 9
Figure 9. Figure 9: The TTFT of vLLM under different HBM allocation [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: The operation workflow of the cache swapper. [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: The average TTFT, TPOT, and supported peak throughput of [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: The breakdown of the average queue, LoRA cold [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 14
Figure 14. Figure 14: The HBM allocation over time of FASTLIBRA and baselines in different application scenarios. We also compare the average KV cache and LoRA hit rates of FASTLIBRA and baselines across different scenarios, as shown in Figure 13b. FASTLIBRA increases the cache hit rate by…
Figure 15
Figure 15. Figure 15: The TTFT and TPOT of the variants of FASTLIBRA in different application scenarios, respectively. gradually decrease in this period as they are swapped out to free HBM for running KVs when the sending rate rises. From 900s to 1300s in (c), FASTLIBRA swaps-out all his￾t…
Figure 16
Figure 16. Figure 16: The TTFT and TPOT of FASTLIBRA and baselines with different LoRA numbers and distributions. The x-axis represents the combination of LoRA number and distribution. LoRA distributions: 1) Uniform, where queries have an equal usage probability for each LoRA. 2) Distinct,…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 20 canonical work pages

  1. [1]

    Sarathi: Efficient llm inference by piggy- backing decodes with chunked prefills

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachan- dran Ramjee. Sarathi: Efficient llm inference by piggy- backing decodes with chunked prefills. arXiv preprint arXiv:2308.16369, 2023

    arXiv 2023

  2. [2]

    Llm in a flash: Efficient large language model inference with lim- ited memory

    Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. Llm in a flash: Efficient large language model inference with lim- ited memory. arXiv preprint arXiv:2312.11514, 2023

    arXiv 2023

  3. [3]

    Introducing apple’s on-device and server foun- dation models, 2025

    Apple. Introducing apple’s on-device and server foun- dation models, 2025

    work page 2025

  4. [4]

    The costly dilemma: generalization, evaluation and cost- optimal deployment of large language models

    Abi Aryan, Aakash Kumar Nain, Andrew McMahon, Lucas Augusto Meyer, and Harpreet Singh Sahota. The costly dilemma: generalization, evaluation and cost- optimal deployment of large language models. arXiv preprint arXiv:2308.08061, 2023

    arXiv 2023

  5. [5]

    Taskmaster-1:toward a realistic and diverse dialog dataset

    Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. Taskmaster-1:toward a realistic and diverse dialog dataset. In 2019 Conference on Empiri- cal Methods in Natural Language Processing and 9th International Joint Conference on Natural Langua...

    work page 2019

  6. [6]

    Punica: Multi-tenant lora serving

    Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. Punica: Multi-tenant lora serving. Proceedings of Machine Learning and Systems, 6:1–13, 2024

    work page 2024

  7. [7]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anas- tasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024

    arXiv 2024

  8. [8]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023

    2023

Show all 54 references
  1. [9]

    Introduction to tpus, 2023

    Google Cloud. Introduction to tpus, 2023

    2023

  2. [10]

    sglang: A fast serving framework for large language models and vision language models., 2024

    SGL Community. sglang: A fast serving framework for large language models and vision language models., 2024

    2024

  3. [11]

    High bandwidth memory, 2023

    Wikipedia contributors. High bandwidth memory, 2023

    2023

  4. [12]

    Trie, 2023

    Wikipedia contributors. Trie, 2023

    2023

  5. [13]

    Nvidia a100 tensor core gpu, 2024

    NVIDIA Corporation. Nvidia a100 tensor core gpu, 2024

    2024

  6. [14]

    Qlora: Efficient finetuning of quan- tized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quan- tized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  7. [15]

    Gpt-3: Its nature, scope, limits, and consequences

    Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681–694, 2020

    2020

  8. [16]

    Attentionstore: Cost-effective atten- tion reuse across multi-turn conversations in large lan- guage model serving

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Attentionstore: Cost-effective atten- tion reuse across multi-turn conversations in large lan- guage model serving. arXiv preprint arXiv:2403.19708, 2024

    2024 arXiv

  9. [17]

    Prompt cache: Modular attention reuse for low-latency inference

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. Pro- ceedings of Machine Learning and Systems, 6:325–338, 2024

    2024

  10. [19]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  11. [20]

    Lorahub: Efficient cross- task generalization via dynamic lora composition

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross- task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269, 2023

    2023 arXiv

  12. [21]

    Chameleon: Adaptive caching and scheduling for many- adapter llm inference environments

    Nikoleta Iliakopoulou, Jovan Stojkovic, Chloe Alverti, Tianyin Xu, Hubertus Franke, and Josep Torrellas. Chameleon: Adaptive caching and scheduling for many- adapter llm inference environments. arXiv preprint arXiv:2411.17741, 2024

    2024

  13. [22]

    Efficient memory man- agement for large language model serving with page- dattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with page- dattention. In Proceedings of the 29th Symposium on Operating Systems Principle...

    2023

  14. [23]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  15. [24]

    Caraserve: Cpu-assisted and rank- aware lora serving for generative llm inference

    Suyi Li, Hanfeng Lu, Tianyuan Wu, Minchen Yu, Qizhen Weng, Xusheng Chen, Yizhou Shan, Binhang 13 Yuan, and Wei Wang. Caraserve: Cpu-assisted and rank- aware lora serving for generative llm inference. arXiv preprint arXiv:2401.11240, 2024

    2024 arXiv

  16. [25]

    Prefix-tuning: Optimiz- ing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimiz- ing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  17. [26]

    Personal llm agents: Insights and survey about the capability, efficiency and security

    Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459, 2024

    2024 arXiv

  18. [27]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  19. [28]

    Instruct-tune llama on consumer hardware using alpaca-lora, 2023

    Alpaca lora team. Instruct-tune llama on consumer hardware using alpaca-lora, 2023

    2023

  20. [29]

    Arena learning: Build data flywheel for llms post-training via simulated chatbot arena

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Qingwei Lin, Jianguang Lou, Shifeng Chen, Yansong Tang, and Weizhu Chen. Arena learning: Build data flywheel for llms post-training via simulated chatbot arena. arXiv preprint arXiv:2407.10627, 2024

    2024 arXiv

  21. [30]

    Language models are few-shot learners

    Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 1, 2020

    2005 arXiv

  22. [31]

    The impact of hyperparameters on large language model inference performance: An evalu- ation of vllm and huggingface pipelines

    Matias Martinez. The impact of hyperparameters on large language model inference performance: An evalu- ation of vllm and huggingface pipelines. arXiv preprint arXiv:2408.01050, 2024

    2024 arXiv

  23. [32]

    Chatgpt, 2020

    OpenAI. Chatgpt, 2020

    2020

  24. [33]

    torch.stream — pytorch 2.0.1 documentation, 2023

    PyTorch Contributors. torch.stream — pytorch 2.0.1 documentation, 2023

    2023

  25. [34]

    Moon- cake: Kimi’s kvcache-centric architecture for llm serv- ing

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Moon- cake: Kimi’s kvcache-centric architecture for llm serv- ing. arXiv e-prints, pages arXiv–2407, 2024

    2024

  26. [35]

    Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider

    Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Go- har Chaudhry, Paul Batum, Jason Cooke, Eduardo Lau- reano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In 2...

    2020

  27. [36]

    Fast transformer decoding: One write- head is all you need

    Noam Shazeer. Fast transformer decoding: One write- head is all you need. arXiv preprint arXiv:1911.02150, 2019

    1911 arXiv

  28. [37]

    Slora: Scalable serving of thousands of lora adapters

    Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. Slora: Scalable serving of thousands of lora adapters. Proceedings of Machine Learning and Systems, 6:296–311, 2024

    2024

  29. [38]

    Understanding lstm–a tutorial into long short-term memory recurrent neural networks

    Ralf C Staudemeyer and Eric Rothstein Morris. Understanding lstm–a tutorial into long short-term memory recurrent neural networks. arXiv preprint arXiv:1909.09586, 2019

    1909 arXiv

  30. [39]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  31. [40]

    Discovering finance keywords via continuous-space lan- guage models

    Ming-Feng Tsai, Chuan-Ju Wang, and Po-Chuan Chien. Discovering finance keywords via continuous-space lan- guage models. ACM Transactions on Management In- formation Systems (TMIS), 7(3):1–17, 2016

    2016

  32. [41]

    vllm: A high-throughput and memory-efficient inference and serving engine for llms

    vLLM Community. vllm: A high-throughput and memory-efficient inference and serving engine for llms

  33. [42]

    Lora-pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407.18242, 2024

    Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407.18242, 2024

    2024 arXiv

  34. [43]

    {dLoRA}: Dynamically or- chestrating requests and adapters for{LoRA}{LLM} serving

    Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xu- anzhe Liu, and Xin Jin. {dLoRA}: Dynamically or- chestrating requests and adapters for{LoRA}{LLM} serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 911–927, 2024

    2024

  35. [44]

    Cacheblend: Fast large language model serving with cached knowledge fusion

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yi- hua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving with cached knowledge fusion. arXiv preprint arXiv:2405.16444, 2024

    2024 arXiv

  36. [45]

    Orca: A distributed serving system for transformer-based generative mod- els

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soo- jeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative mod- els. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, 2022

    2022

  37. [46]

    Stateful large language model serving with pensieve

    Lingfan Yu, Jinkun Lin, and Jinyang Li. Stateful large language model serving with pensieve. arXiv preprint arXiv:2312.05516, 2023. 14

    2023 arXiv

  38. [47]

    Improving massively multilingual neural machine translation and zero-shot translation

    Biao Zhang, Philip Williams, Ivan Titov, and Rico Sen- nrich. Improving massively multilingual neural machine translation and zero-shot translation. arXiv preprint arXiv:2004.11867, 2020

    2004 arXiv

  39. [48]

    Google’s neural machine translation system: Bridging the gap between human and machine transla- tion

    Bill Zhang. Google’s neural machine translation system: Bridging the gap between human and machine transla- tion. arXiv preprint arXiv:1609.08144, 11, 2016

    2016 arXiv

  40. [49]

    Adalora: Adaptive bud- get allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive bud- get allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023

    2023 arXiv

  41. [50]

    Faster and cheaper serverless computing on harvested resources

    Yanqi Zhang, Íñigo Goiri, Gohar Irfan Chaudhry, Ro- drigo Fonseca, Sameh Elnikety, Christina Delimitrou, and Ricardo Bianchini. Faster and cheaper serverless computing on harvested resources. InProceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, pages...

    2021

  42. [51]

    Lora land: 310 fine-tuned llms that rival gpt-4, a technical report

    Justin Zhao, Timothy Wang, Wael Abid, Geoffrey An- gus, Arnav Garg, Jeffery Kinnison, Alex Sherstinsky, Piero Molino, Travis Addair, and Devvret Rishi. Lora land: 310 fine-tuned llms that rival gpt-4, a technical report. arXiv preprint arXiv:2405.00732, 2024

    2024 arXiv

  43. [52]

    Judging llm-as- a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuo- han Li, Dacheng Li, Eric Xing, et al. Judging llm-as- a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595– 46623, 2023

    2023

  44. [53]

    Efficiently programming large language models using sglang

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Chris- tos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Efficiently programming large language models using sglang. arXiv e-prints, pages arXiv–2312, 2023

    2023

  45. [54]

    Dist- serve: Disaggregating prefill and decoding for goodput- optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Dist- serve: Disaggregating prefill and decoding for goodput- optimized large language model serving. arXiv preprint arXiv:2401.09670, 2024

    2024 arXiv

  46. [55]

    Multilingual machine translation with large language models: Empirical results and analysis

    Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models: Empirical results and analysis. arXiv preprint arXiv:2304.04675, 2023. 15

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.