REVIEW 12 cited by
S-LoRA: Serving Thousands of Concurrent LoRA Adapters
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The "pretrain-then-finetune" paradigm is commonly adopted in the deployment of large language models. Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is often employed to adapt a base model to a multitude of tasks, resulting in a substantial collection of LoRA adapters derived from one base model. We observe that this paradigm presents significant opportunities for batched inference during serving. To capitalize on these opportunities, we present S-LoRA, a system designed for the scalable serving of many LoRA adapters. S-LoRA stores all adapters in the main memory and fetches the adapters used by the currently running queries to the GPU memory. To efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes Unified Paging. Unified Paging uses a unified memory pool to manage dynamic adapter weights with different ranks and KV cache tensors with varying sequence lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and highly optimized custom CUDA kernels for heterogeneous batching of LoRA computation. Collectively, these features enable S-LoRA to serve thousands of LoRA adapters on a single GPU or across multiple GPUs with a small overhead. Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with naive support of LoRA serving), S-LoRA can improve the throughput by up to 4 times and increase the number of served adapters by several orders of magnitude. As a result, S-LoRA enables scalable serving of many task-specific fine-tuned models and offers the potential for large-scale customized fine-tuning services. The code is available at https://github.com/S-LoRA/S-LoRA
Forward citations
Cited by 12 Pith papers
-
Metis: Memory Foundation Model
Metis puts a trainable fixed-size memory matrix inside a frozen LLM backbone and learns to remember, update, forget, and reflect across turns without replaying original context.
-
How Context Attribution Handles What the Model Already Knows
Context attribution methods cannot disentangle in-context from in-weight knowledge and assign unfaithful scores under overlap; new metrics and WMDP-Cyber++ quantify the failure.
-
Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems
State management across streaming, serving, retrieval, and learning systems is best modeled as one coupled runtime control loop spanning access, execution, and evolution.
-
CeRA: Breaking the Linear Ceiling of Low-Rank Adaptation with Non-linearity Retained at Inference
CeRA overcomes LoRA's linear ceiling by injecting non-linear SiLU gating and dropout, outperforming high-rank LoRA on complex math reasoning with 1/8 the parameters.
-
InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata
InstantInfer refactors vLLM's cold start into a concurrent state-machine pipeline, speeding up startup by up to 7.2×.
-
JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Models
A service-oriented multi-tenant architecture with schema-compatible group batching reduces aggregate GPU time for VLA post-training by about 28% in simulation.
-
Tensorized Clustered LoRA Merging for Multi-Task Interference
Clustering training data by embedding similarity and jointly CP-decomposing LoRA adapters cuts multi-task merging interference: +1.4% on Phi-3 and +2.3% on Mistral-7B over SVD baselines.
-
On Evaluating Performance of LLM Inference Serving Systems
A systematic review identifies eight anti-patterns in LLM inference evaluation and proposes a checklist, with a speculative decoding case study demonstrating how conventional metrics mislead.
-
EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices
EdgeLoRA combines automatic adapter routing, LRU caching with a memory pool, and grouped LoRA batching to serve thousands of LoRA adapters on edge devices with up to 4x higher throughput than llama.cpp.
-
LARA: Lightweight Adapters in the Residual Stream for Composable Adaptation and Alignment
Low-rank residual-stream adapters match LoRA at near-equal parameter counts on code fine-tuning and DPO, and support inference-time scaling plus per-token routing of seven behaviors on one frozen 1.5B model.
-
Get Experience from Practice: LLM Agents with Record & Replay
AgentRR is a proposed paradigm that records agent traces, generalizes them into multi-level experiences, and replays them under safety checks to make LLM agents cheaper, faster, and more reliable.
-
PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpoint
PHLoRA extracts LoRA-compatible adapters from full-rank fine-tuned models via truncated SVD of the weight delta, matching full-rank performance on several benchmarks with no gradients or training data.
Discussion (0). Continue with ORCID to comment.