Pith. sign in

REVIEW 5 major objections 5 minor 42 references

A data-driven pipeline can place LLM adapters on GPUs so workloads run on the fewest GPUs, cutting GPU use by 60 percent on average.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:04 UTC pith:CNZEAXKZ

load-bearing objection A genuinely useful pipeline for adapter placement, but the 'minimum GPUs' claim is contradicted by the paper's own Fig. 11 and the 60% average reduction is never derived in the body. the 5 major comments →

arxiv 2602.24044 v2 pith:CNZEAXKZ submitted 2026-02-27 cs.DC cs.AIcs.CLcs.LG

Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving

Ferran Agullo , Joan Oliveras , Chen Wang , Alberto Gutierrez-Torre , Olivier Tardieu , Alaa Youssef , Jordi Torres , Josep Ll. Berral This is my paper
classification cs.DC cs.AIcs.CLcs.LG
keywords adapter cachingLoRA adaptersLLM servingGPU efficiencydigital twinmachine learning surrogategreedy placementMaxpack
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the adapter caching problem — deciding which lightweight model adapters live on which GPU, and how many adapters a GPU should hold — can be solved for predictable workloads by a data-driven pipeline. The pipeline identifies each GPU's maximum feasible packing point (Maxpack), the highest throughput before request starvation or out-of-memory errors set in, and then packs adapters greedily up to that point. The authors argue that a CPU-based digital twin plus a distilled machine-learning surrogate can predict throughput and starvation accurately enough (below 5 percent error for the twin, below 8 percent for the learned models) and fast enough for periodic reconfiguration, and they report a 60 percent average reduction in GPUs needed across tested scenarios. If true, this turns GPU provisioning for multi-adapter LLM serving from a manual, heuristic exercise into a planable optimization.

Core claim

The central claim is that the optimal packing point Maxpack is learnable and that reaching it on each GPU minimizes total GPU count without violating service constraints. The authors construct this by emulating an online LLM-adapter serving loop in a Digital Twin that runs on CPU and reproduces batch scheduling, KV-cache allocation, adapter swapping, and model-forward latency; the twin is calibrated with a small set of real benchmarks and then generates a large synthetic dataset. From that dataset, two ML models — a throughput regressor and a starvation classifier — are trained and optionally distilled into shallow decision trees that run in microseconds. A greedy first-fit-decreasing placem

What carries the argument

The load-bearing objects are Maxpack (the per-GPU adapter count/throughput point at which throughput peaks before starvation; operationally, the highest measured throughput above 90% of the incoming token rate) and the three-stage pipeline built to find it: a Digital Twin whose four predictive models (memory capacity, scheduler latency, adapter loading latency, model forward latency) reproduce the continuous batching loop; ML surrogates (a throughput regressor and a starvation classifier) trained on twin-generated data; and a greedy first-fit-decreasing placement algorithm with periodic feasibility tests that select the per-GPU Amax. The key mechanism is the transfer of expensive real-system

Load-bearing premise

The pipeline assumes the future workload is known in advance as per-adapter Poisson arrival rates with predictable request-length statistics; if real arrivals are bursty or the token-length distribution shifts, the predicted Maxpack points and the minimum-GPU guarantee are unvalidated.

What would settle it

Run the pipeline on a workload whose arrivals are bursty (e.g., log-normally distributed with high variance, or switching rates every few minutes) or whose request lengths come from a distribution far from the training data, then execute the recommended placement on real GPUs; starvation episodes, memory errors, or throughput estimation error above the reported single-digit percentages would falsify the claim that the pipeline serves the workload with minimum GPUs without starvation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • For workloads that can be forecast, operators can precompute an adapter-to-GPU placement that uses the minimum number of GPUs while avoiding starvation and memory errors.
  • Periodic re-invocation of the pipeline lets a serving system follow long-term workload shifts, freeing GPUs for other workloads or power-down.
  • The Digital Twin alone is a reusable tool that estimates throughput, inter-token latency, and time-to-first-token for adapter serving about 90x faster than real benchmarks, enabling broader scheduling and configuration studies.
  • The refined fast variant computes placements in under 3 ms on four GPUs, making the approach viable for fast reconfiguration.
  • The same learned predictors can be repurposed for latency-oriented allocation, showing the pipeline is not tied to GPU-count minimization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the method's validity rests on the workload model; under bursty arrivals or shifted token-length distributions, the ML models and greedy placements would need retraining and revalidation, and the claimed 60% reduction is an upper bound for the tested distribution class.
  • Editorial inference: the greedy algorithm's zigzag ordering and discrete Amax candidates are heuristic choices; a natural extension is to couple the learned predictors with a search over ordering policies or an ILP for larger clusters.
  • Editorial inference: the Digital Twin could be inverted into an online controller — periodically re-simulating the current queue state to recommend dynamic Amax changes within a reconfiguration interval, which the paper treats as static.
  • Editorial inference: since Maxpack depends on arrival rates, the placement changes the load on each GPU; a workload-placement feedback loop could be tested by iterating the pipeline on the realized arrivals from the previous placement.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper addresses the "adapter caching problem" in distributed LLM-adapter serving: given an expected workload (adapters with sizes and Poisson arrival rates), it computes an adapter-to-GPU placement and per-GPU Amax configuration that maximizes per-GPU throughput (reaching Maxpack), thereby minimizing the number of GPUs while avoiding request starvation and memory errors. The proposed pipeline has three stages: (i) a Digital Twin (DT) that emulates vLLM's continuous batching, KV-cache allocation, and adapter swapping on CPU; (ii) ML models (KNN, RF, SVM, plus a refined decision-tree variant) trained on DT-generated data to predict throughput and starvation risk; and (iii) a greedy First-Fit-Decreasing placement algorithm that uses these predictions. The DT is validated against real vLLM/H100 executions with two backbone LLMs, reporting <5% throughput SMAPE and up to 90x speedup. The pipeline is evaluated on single- and 4-GPU scenarios and compared against MaxBase, MaxBase*, Random, and dLoRA, claiming improvements in GPU efficiency and a 60% average GPU reduction in the abstract.

Significance. If the claims hold, this is a practically valuable contribution to LLM-adapter serving: it attacks an under-explored resource-efficiency problem, introduces a Digital Twin that captures adapter-caching dynamics, and validates it on real H100 hardware with two popular backbone models. The DT's low throughput error and fast execution are genuine strengths, and the refinement into interpretable decision trees with sub-millisecond inference is a useful engineering contribution. The real-system validation and the inclusion of a comparison with dLoRA strengthen the empirical study. However, the headline claims outrun the evidence: the "minimum number of GPUs" guarantee is contradicted by one of the paper's own experiments, the 60% average reduction is not reported in the body, and the ML models are not independently validated against the DT's calibration process. With corrected claims and a sharper evaluation protocol, the core pipeline has solid value.

major comments (5)
  1. [§8.4.2, Fig. 11] In the rightmost scenario (Llama, low sizes, mixed rates), the Proposed curve terminates at 128 adapters while the Random baseline reaches 160 adapters without starvation on the same 4-GPU system. Because Random's 160-adapter allocation is feasible, the pipeline's infeasibility prediction at 160 is a false positive. The paper's central claim (Abstract, §1, §8.4) that the pipeline "serves the workload with the minimum number of GPUs without incurring starvation" is therefore empirically contradicted: at 160 adapters the proposed pipeline does not serve the workload at all. The dismissal in §8.4.2 ("expected given the stochastic nature of Random") does not address the contradiction; it indicates that the ML/DT predictions are not reliable enough to certify infeasibility. Please either modify the prediction/algorithm to eliminate such false positives or temper the claim to "reduces GPU coun
  2. [Abstract / §8.4.2] The abstract states that the pipeline reduces "the number of GPUs required to sustain target workloads by 60% on average across the evaluated scenarios." I could not find this average anywhere in the body. §8.4.2 reports scenario-by-scenario curves (Fig. 11) but computes no aggregate reduction. This is a headline quantitative claim and must be either derived from the data — with a precise definition of "reduction" and the reference baseline — or removed from the abstract.
  3. [§8.3 / §8.2] The ML models are trained on DT-generated data, yet §8.3 says they are evaluated "against the same real-system executions employed for the validation of the Digital Twin." Since the ML models are distilled surrogates of the DT, this evaluation does not independently test the ML surrogate; it largely re-tests the DT's fidelity on a set already used in the DT's validation loop. The greedy placement (Algorithm 2) relies on these ML predictions, so the reported accuracy in Table 3 does not establish end-to-end reliability for unseen workload conditions. Please evaluate the ML models on a held-out set not used in DT parameterization/validation, or explicitly analyze how ML and DT errors compound in the final placement decisions.
  4. [§7 / Abstract] The problem is NP-hard and the proposed algorithm is a heuristic FFD variant with no optimality guarantee. The paper nevertheless claims the output is "the minimum number of GPUs" (Abstract, §1, §8.4). No lower bound, optimal baseline, or exhaustive search on small instances is provided, so the optimality claim is unsupported. Please either add such an analysis (e.g., exact ILP for small workloads) or rephrase the claim to "small" or "near-minimum" GPU count.
  5. [§6 / §9.1 / Abstract] The decision engine (ML models and greedy placement) is trained and validated only for Poisson arrivals with a fixed request-length distribution. Although the DT is stress-tested under non-stationary arrivals (§8.2), the ML models and placement algorithm are not, as §9.1 admits. Since the placement's starvation/throughput predictions come from these ML models, the abstract's unqualified claim of a starvation-free minimum-GPU placement "for a given workload" is unvalidated outside the Poisson/fixed-length class. Please either test the ML/placement engine under the same non-stationary conditions already used for the DT or explicitly scope the abstract and title claims to the evaluated workload class.
minor comments (5)
  1. [§2.3 vs §8.1] Section 2.3 says "Smax is configured to match the adapter size used in each experiment," while §8.1 says "we set Smax as the maximum adapter size found in every tested scenario." These descriptions are inconsistent and should be reconciled, as Smax directly affects memory partitioning.
  2. [Algorithm 1] Line 27 says "repeat lines 12-14" without specifying the exact loop semantics. The pseudocode would be clearer if the commit/allocation logic were factored into a subroutine, especially since the same logic appears in two places.
  3. [Table 4] The Qwen starvation inference time for Small Tree** is reported as 10.5e-4 ms (1.05 microseconds), while the text states that Numba-optimized implementations achieve "inference times below 100ns per prediction." These numbers are inconsistent; please correct the table or the text.
  4. [§7] The zigzag rate ordering is described as selected "empirically" with no sensitivity analysis. Since this ordering is a free design choice that affects the packing, a brief ablation (e.g., ascending, descending, zigzag sizes) would improve confidence in the algorithm's robustness.
  5. [Algorithm 1/2] The pseudocode labels contain a formatting artifact: "Star v ation" (with a space) appears in the error messages and comments. This should be fixed to "Starvation" for readability.

Circularity Check

0 steps flagged

No significant circularity: the pipeline is calibrated to real profiling data and validated against real-system executions; the few self-citations are not load-bearing, and the ML/DT evaluation overlap and Fig. 11 anomaly are validity concerns, not definitional reductions.

full rationale

The derivation chain is an empirical calibration-plus-validation loop, not a tautology. The Digital Twin uses profiled constants (Eq. 1, K1–K7, Memmax) and is then checked against real vLLM/H100 executions with reported SMAPE (Table 1); the ML surrogate is trained on DT-generated data and checked against the same real executions (Table 3, §8.3). Because the final check is against real system behavior rather than the fitted curves, the central throughput/starvation predictions are not forced by construction. One evaluation-dependence concern is that §8.3 says ML models are "evaluated against the same real-system executions employed for the validation of the Digital Twin"; this reduces the independence of the ML test set, but it does not make the ML output equal to its training input, and the paper does not state that the DT parameterization experiments are inside that validation set. The self-citation [20] (§3.3) is only a prior workshop version and is not load-bearing. The paper's own Fig. 11 shows Random reaching 160 adapters where Proposed stops at 128, which undermines the unqualified "minimum number of GPUs" claim, but that is a correctness/optimality issue about greedy search, not a circularity. Admitted limitations in §9.1 (Poisson arrivals, fixed request-length distribution for ML training) are generalization limits, not circular steps. Overall score 2: no definitional or self-citation-induced circularity; a minor evaluation-overlap concern keeps it slightly above 0.

Axiom & Free-Parameter Ledger

7 free parameters · 7 axioms · 0 invented entities

The DT is a calibrated simulator: every latency model enters with constants fit from profiling on the same H100 hardware where fidelity is then measured. That is legitimate calibration, but it means the <5% fidelity claim covers workload variation on one hardware/model family, not extrapolation to new hardware or to departures from the Poisson/fixed-length workload class. The ML stage adds a layer trained entirely on DT output. The only genuinely free quantities the greedy itself uses are hand-chosen grids (testing points, Amax candidates, zigzag ordering) and the 90% starvation definition. No invented entities.

free parameters (7)
  • Latency-model constants K1–K7 = not reported
    Calibrated constants in Eq. (1) — Lat_sched = K1B + K2RP + K3·RP·AB/A and Lat_model = (K4B+K5)(K6A+K7) — fitted from profiling benchmarks on the target GPU/model (§5.2). Central to the digital twin's fidelity claim.
  • Memory estimator Memmax (Tmax) = empirical lookup
    Derived directly from profiled data (Fig. 4, left) rather than an analytical formula; converts Amax/Smax into maximum token capacity (§5.2).
  • Adapter loading latency LA = profiled per adapter size
    Lat_load = LA from Fig. 6 profiling; per-adapter-size loading cost used in DT swap events (§5.2).
  • Starvation threshold 90% = 0.9
    Maxpack and the ML starvation label are defined as throughput falling below 90% of incoming token rate (§2.3); hand-chosen definition inherited by the DT, the ML models, and the validation.
  • Greedy testing points and Amax candidate grid = [8,16,32,64,96,128,160,192,256,320,384]
    Fixed arrays for provisional-allocation testing and Amax candidates (§7, Algorithm 2); chosen by hand, so the Amax optimization is coarse-grained.
  • Zigzag rate ordering = high-low alternating
    PrioritySorting's zigzag order was "selected empirically" because it consistently improved throughput (§7).
  • Smax = max adapter size per scenario = 8 / 16 / 32
    vLLM assigns a uniform memory footprint per adapter; setting Smax to the scenario maximum is the default behavior and inflates the memory cost of small adapters (§2.2, §8.1).
axioms (7)
  • domain assumption Backbone prefill/decode latency is a linear function of batch size
    Eq. (1) Lat_model = (K4B+K5), justified by Fig. 4 (right) and prior work [38,27]; load-bearing for DT fidelity outside the calibrated batch range.
  • domain assumption Adapter computational overhead is a linear function of the number of adapters
    Eq. (1) Overhead_A = (K6A+K7), from Fig. 5; the paper explicitly notes this differs from CaraServe's model [17].
  • domain assumption Workloads are predictable Poisson arrivals with known per-adapter rates and average token lengths
    §6 training data and §4 problem formulation; acknowledged in §9.1 as a limitation. The DT was tested under non-stationary arrivals, but the ML stage and placement were not.
  • domain assumption Using mean input/output token lengths in the DT preserves fidelity
    §5 requires output length as DT input; the Mean variant is validated (Table 1) with modest SMAPE increase, but only for the tested distributions.
  • domain assumption vLLM's greedy KV-cache allocation and Amax static partition are faithfully emulated by the DT's code-based simulation
    §5 DT architecture; the fidelity claims depend on this emulation matching vLLM v0.8.5 behavior.
  • standard math Bin packing is NP-hard, so FFD-style greedy is an appropriate approximation
    §7 cites Garey-Johnson [8] and Johnson [14]; standard, uncontroversial.
  • domain assumption The throughput-plateau phenomenon applies to this serving regime
    §5.1.1 relies on the plateau (diminishing returns of batch size) from [25,26,2] to explain why adapter memory costs may not reduce throughput.

pith-pipeline@v1.3.0-alltime-deepseek · 22722 in / 20558 out tokens · 185177 ms · 2026-08-02T20:04:26.076676+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving." pith.science (2026). https://pith.science/paper/CNZEAXKZ

@misc{pith2026260224044,
  author       = {Pith},
  title        = {Pith review of: Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CNZEAXKZ}},
  note         = {Machine review of arXiv:2602.24044}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently. While prior work has largely focused on latency and throughput optimization, minimizing GPU resource requirements through near-peak utilization remains largely underexplored. This paper presents a data-driven pipeline that, for a given workload, computes an adapter placement that serves the workload with the minimum number of GPUs while avoiding request starvation and GPU memory errors. To that end, the approach identifies the maximum feasible throughput attainable on each GPU by leveraging accurate performance predictions learned from real serving behavior. The proposed pipeline integrates three components: (i) a Digital Twin (DT) tailored to LLM-adapter serving, (ii) a distilled machine learning (ML) model trained on DT-generated data, and (iii) a greedy placement algorithm that exploits ML-based performance estimates to maximize GPU efficiency. The DT emulates real system dynamics with high fidelity, achieving below 5% throughput estimation error while executing up to 90x faster than full LLM benchmarking across both predictable and unpredictable workloads. The learned ML models further accelerate performance estimation with marginal accuracy degradation, enabling scalable optimization. Experimental results demonstrate that the pipeline substantially improves GPU efficiency, reducing the number of GPUs required to sustain target workloads by 60\% on average across the evaluated scenarios. Beyond GPU efficiency, the pipeline can be adapted to alternative objectives, such as latency minimization, highlighting its versatility for future large-scale LLM serving infrastructures.

Figures

Figures reproduced from arXiv: 2602.24044 by Alaa Youssef, Alberto Gutierrez-Torre, Chen Wang, Ferran Agullo, Joan Oliveras, Jordi Torres, Josep Ll. Berral, Olivier Tardieu.

Figure 1
Figure 1. Figure 1: Throughput as a function of the number of served adapters [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Proposed data-driven pipeline to address the adapter caching problem (right), shown alongside its expected usage within a production [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Digital Twin behavior and architecture. greedy adapter caching algorithm. The greedy algorithm produces the final placement decision, relying on perfor￾mance predictions generated by the ML models. Trained offline, these models estimate the achievable throughput of a GPU under a given adapter placement and Amax config￾uration, and predict whether starvation may arise. An op￾tional refinement phase can simp… view at source ↗
Figure 4
Figure 4. Figure 4: Evolution of batch size and throughput with increasing [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: Scheduler time relative to the average per-step execution [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 6
Figure 6. Figure 6: Loading times for varying adapter sizes, shown relative [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparison between Digital Twin predictions and real [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: We follow the same evaluation methodology as in the predictable-arrival experiments; however, for efficiency reasons, the number of evaluated scenarios is reduced. Adapter sizes are fixed to 8, Amax is set to 32, and the number of served adapters is varied among 32, 64, and 128. Arrival rates follow the same ranges as in the previous ex￾periments. The resulting SMAPE values are reported in the right part o… view at source ↗
Figure 9
Figure 9. Figure 9: Execution with Llama-3.1-8B under initial high arrival [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: (Top) Achieved throughput and (Bottom) configured [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Number of GPUs required by each baseline and by the proposed pipeline on a 4-GPU system, across heterogeneous workloads with [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Number of GPUs required, throughput, and ITL achieved by each method on a 4-GPU system, across two heterogeneous [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 1 canonical work pages

  1. [1]

    Vidur: A large-scale simulation framework for llm inference

    Agrawal, A., Kedia, N., Mohan, J., Panwar, A., Kwa- tra, N., Gulavani, B.S., Ramjee, R., Tumanov, A., 2024a. Vidur: A large-scale simulation framework for llm inference. Proceedings of Machine Learning and Systems 6, 351–366

  2. [2]

    Taming throughput-latency tradeoff in llm in- ference with sarathi-serve, in: 18th USENIX Sympo- sium on Operating Systems Design and Implementa- tion (OSDI 24), pp

    Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwa- tra, N., Gulavani, B., Tumanov, A., Ramjee, R., 2024b. Taming throughput-latency tradeoff in llm in- ference with sarathi-serve, in: 18th USENIX Sympo- sium on Operating Systems Design and Implementa- tion (OSDI 24), pp. 117–134

  3. [3]

    Clean sharegpt dataset

    anon8231489123, 2023. Clean sharegpt dataset. URL:https://huggingface.co/datasets/ anon8231489123/ShareGPT_Vicuna_unfiltered

    2023

  4. [4]

    Language models are few-shot learners

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Ka- plan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al., 2020. Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901

    2020

  5. [5]

    Compress then serve: Serving thousands of lora adapters with little overhead

    Brüel-Gabrielsson, R., Zhu, J., Bhardwaj, O., Choshen, L., Greenewald, K., Yurochkin, M., Solomon, J., 2024. Compress then serve: Serving thousands of lora adapters with little overhead. arXiv preprint arXiv:2407.00066

    Pith/arXiv arXiv 2024

  6. [6]

    Punica: Multi-tenant lora serving

    Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., Krishna- murthy, A., 2024. Punica: Multi-tenant lora serving. Proceedings of Machine Learning and Systems 6, 1– 13

    2024

  7. [7]

    Llmservingsim: A hw/sw co-simulation infrastructure for llm inference serving at scale, in: 2024 IEEE Inter- national Symposium on Workload Characterization (IISWC), IEEE

    Cho, J., Kim, M., Choi, H., Heo, G., Park, J., 2024. Llmservingsim: A hw/sw co-simulation infrastructure for llm inference serving at scale, in: 2024 IEEE Inter- national Symposium on Workload Characterization (IISWC), IEEE. pp. 15–29

    2024

  8. [8]

    Computers and Intractability: A Guide to the Theory of NP- Completeness

    Garey, M.R., Johnson, D.S., 1979. Computers and Intractability: A Guide to the Theory of NP- Completeness. W. H. Freeman and Company, San Francisco

    1979

  9. [9]

    The llama 3 herd of models

    Grattafiori, A., et al., 2024. The llama 3 herd of models. URL:https://arxiv.org/abs/2407. 21783,arXiv:2407.21783

    Pith/arXiv arXiv 2024

  10. [10]

    Guo, D., Rush, A., Kim, Y., 2021. Parameter-efficient transfer learning with diff pruning, in: Proceedings of the 59th annual meeting of the association for com- putational linguistics and the 11th international joint conference on natural language processing (volume 1: Long papers), pp. 4884–4896

    2021

  11. [11]

    Parameter-efficient transfer learning fornlp, in: Internationalconferenceonmachinelearn- ing, PMLR

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S., 2019. Parameter-efficient transfer learning fornlp, in: Internationalconferenceonmachinelearn- ing, PMLR. pp. 2790–2799

    2019

  12. [12]

    Lora: Low-rank adaptation of large language models

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al., 2022. Lora: Low-rank adaptation of large language models. ICLR 1, 3

    2022

  13. [13]

    Chameleon: Adap- tive Caching and Scheduling for Many-Adapter LLM Inference Environments

    Iliakopoulou, N., Stojkovic, J., Alverti, C., Xu, T., Franke, H., Torrellas, J., 2025. Chameleon: Adap- tive Caching and Scheduling for Many-Adapter LLM Inference Environments. Association for Computing Machinery, New York, NY, USA. p. 217–231. URL: https://doi.org/10.1145/3725843.3756083

    arXiv 2025

  14. [14]

    Fast algorithms for bin packing

    Johnson, D.S., 1974. Fast algorithms for bin packing. Journal of Computer and System Sciences 8, 272–314. URL:https://www.sciencedirect.com/science/ article/pii/S0022000074800267, doi:https: //doi.org/10.1016/S0022-0000(74)80026-7

    work page doi:10.1016/s0022-0000(74)80026-7 1974

  15. [15]

    Ef- ficient memory management for large language model serving with pagedattention, in: Proceedings of the 29th Symposium on Operating Systems Principles, pp

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.H., Gonzalez, J., Zhang, H., Stoica, I., 2023. Ef- ficient memory management for large language model serving with pagedattention, in: Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626

    2023

  16. [16]

    Numba: A llvm-based python jit compiler, in: Proceedings of the Second Workshop on the LLVM Compiler Infrastruc- ture in HPC, pp

    Lam, S.K., Pitrou, A., Seibert, S., 2015. Numba: A llvm-based python jit compiler, in: Proceedings of the Second Workshop on the LLVM Compiler Infrastruc- ture in HPC, pp. 1–6

    2015

  17. [17]

    Caraserve: Cpu- assistedandrank-awareloraservingforgenerativellm inference

    Li, S., Lu, H., Wu, T., Yu, M., Weng, Q., Chen, X., Shan, Y., Yuan, B., Wang, W., 2024. Caraserve: Cpu- assistedandrank-awareloraservingforgenerativellm inference. arXiv preprint arXiv:2401.11240

    Pith/arXiv arXiv 2024

  18. [18]

    Prefix-tuning: Optimizing continuouspromptsforgeneration

    Li, X.L., Liang, P., 2021. Prefix-tuning: Optimizing continuouspromptsforgeneration. Proceedingsofthe 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , 4582–4597URL:https://api. semanticscholar.org/CorpusID:230433941. 17 True False Fal...

    2021

  19. [19]

    Few-shot parameter- efficient fine-tuning is better and cheaper than in- context learning

    Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., Raffel, C.A., 2022. Few-shot parameter- efficient fine-tuning is better and cheaper than in- context learning. Advances in Neural Information Processing Systems 35, 1950–1965

    2022

  20. [20]

    López, F.A., Oliveras, J., Wang, C., Gutierrez-Torre, A., Tardieu, O., Youssef, A., Torres, J., Berral, J.L.,

  21. [21]

    DeepSpeed-MII

    Microsoft, 2022–2025. DeepSpeed-MII. GitHub repository. URL:https://github.com/ deepspeedai/DeepSpeed-MII

    2022

  22. [22]

    TensorRT-LLM

    NVIDIA, 2023–2025. TensorRT-LLM. GitHub repository. URL:https://github.com/NVIDIA/ TensorRT-LLM

    2023

  23. [23]

    Scikit-learn: Machine learning in Python

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duch- esnay, E., 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830

    2011

  24. [24]

    Qin, G., Eisner, J., 2021. Learninghowtoask: Query- ing lms with mixtures of soft prompts, in: Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pp. 5203– 5212

    2021

  25. [25]

    Mind the memory gap: Unveiling gpu bottlenecks in large- batch llm inference, in: 2025 IEEE 18th International Conference on Cloud Computing (CLOUD), IEEE

    Recasens, P.G., Agullo, F., Zhu, Y., Wang, C., Lee, E.K., Tardieu, O., Torres, J., Berral, J.L., 2025. Mind the memory gap: Unveiling gpu bottlenecks in large- batch llm inference, in: 2025 IEEE 18th International Conference on Cloud Computing (CLOUD), IEEE. pp. 277–287

    2025

  26. [26]

    Towards pareto optimal throughput in small language model serving, in: Proceedings of the 4th Workshop on Machine Learning and Systems, pp

    Recasens, P.G., Zhu, Y., Wang, C., Lee, E.K., Tardieu, O., Youssef, A., Torres, J., Berral, J.L., 2024. Towards pareto optimal throughput in small language model serving, in: Proceedings of the 4th Workshop on Machine Learning and Systems, pp. 144–152

    2024

  27. [27]

    Shen, H., Chen, L., Jin, Y., Zhao, L., Kong, B., Philipose, M., Krishnamurthy, A., Sundaram, R.,

  28. [28]

    EdgeLoRA: An Efficient Multi- Tenant LLM Serving System on Edge Devices

    Shen, Z., He, Y., Wang, Z., Zhang, Y., Sun, G., Ye, W., Li, A., 2025. EdgeLoRA: An Efficient Multi- Tenant LLM Serving System on Edge Devices. As- sociation for Computing Machinery, New York, NY, USA. p. 138–153. URL:https://doi.org/10.1145/ 3711875.3729141

    arXiv 2025

  29. [29]

    Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., et al.,

  30. [30]

    Lst: Ladder side-tuning for parameter and memory efficient trans- fer learning

    Sung, Y.L., Cho, J., Bansal, M., 2022. Lst: Ladder side-tuning for parameter and memory efficient trans- fer learning. Advances in Neural Information Process- ing Systems 35, 12991–13005

    2022

  31. [31]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., et al., 2023. Llama 2: Open foundation and fine-tuned chat models. URL:https://arxiv. org/abs/2307.09288,arXiv:2307.09288

    Pith/arXiv arXiv 2023

  32. [32]

    Parameter-efficient fine-tuning in large language models: a survey of methodologies

    Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., Yang, F., 2025. Parameter-efficient fine-tuning in large language models: a survey of methodologies. Artificial Intelligence Review 58, 227. URL:https: //doi.org/10.1007/s10462-025-11236-4, doi:10. 1007/s10462-025-11236-4

    doi:10.1007/s10462-025-11236-4 2025

  33. [33]

    Finance lora adapter for llama-3.1-8b instruct

    Wengwengwhale, 2024. Finance lora adapter for llama-3.1-8b instruct. URL:https: //huggingface.co/Wengwengwhale/llama_3.1_ 8B_Instruct_Finance_lora_adapter

    2024

  34. [34]

    dlora: Dynamically orchestrating requests and adapters for lora llm serving, in: 18th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 24), pp

    Wu, B., Zhu, R., Zhang, Z., Sun, P., Liu, X., Jin, X., 2024. dlora: Dynamically orchestrating requests and adapters for lora llm serving, in: 18th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 24), pp. 911–927

    2024

  35. [35]

    Qwen2.5 technical re- port

    Yang, A., et al., 2025. Qwen2.5 technical re- port. URL:https://arxiv.org/abs/2412.15115, arXiv:2412.15115

    Pith/arXiv arXiv 2025

  36. [36]

    Sql lora for llama-2-7b

    yard1, 2024. Sql lora for llama-2-7b. URL:https://huggingface.co/yard1/ llama-2-7b-sql-lora-test

    2024

  37. [37]

    Orca: A distributed serving system for transformer-based generative models, in: 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp

    Yu, G.I., Jeong, J.S., Kim, G.W., Kim, S., Chun, B.G., 2022. Orca: A distributed serving system for transformer-based generative models, in: 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521–538

    2022

  38. [38]

    Shepherd: Serving dnns in the wild, in: 20th USENIX Symposium on Networked Systems Design and Imple- mentation (NSDI 23), pp

    Zhang, H., Tang, Y., Khandelwal, A., Stoica, I., 2023. Shepherd: Serving dnns in the wild, in: 20th USENIX Symposium on Networked Systems Design and Imple- mentation (NSDI 23), pp. 787–808

    2023

  39. [39]

    Medical lora for qwen2.5-7b- instruc

    zjudai, 2025. Medical lora for qwen2.5-7b- instruc. URL:https://huggingface.co/zjudai/ flowertune-medical-lora-qwen2.5-7b-instruct. 19

    2025

  40. [2019]

    Nexus: a gpu cluster engine for accelerat- ing dnn-based video analysis, in: Proceedings of the 27th ACM Symposium on Operating Systems Prin- ciples, Association for Computing Machinery, New York, NY, USA. p. 322–337. doi:10.1145/3341301. 3359658. 18

    doi:10.1145/3341301

  41. [2024]

    Proceedings of Machine Learning and Sys- tems 6, 296–311

    Slora: Scalable serving of thousands of lora adapters. Proceedings of Machine Learning and Sys- tems 6, 296–311

  42. [2025]

    Poster session, San Diego, CA

    A data-driven ml approach for maximizing per- formance in llm-adapter serving, in: 9th Machine Learning for Systems (ML for Systems) Workshop, NeurIPS 2025. Poster session, San Diego, CA

    2025