Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

LithOS: An Operating System for Efficient Machine Learning on GPUs

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Atomizing GPU kernels into schedulable chunks cuts tail latency 13x

desk verdict The system design is ambitious and the evaluation broad, but the kernel atomization mechanism as written cannot deliver the fine-grained spatial scheduling it claims, leaving the headline results unsupported. read the letter →

arxiv 2504.15465 v1 pith:4EU67LSE submitted 2025-04-21 cs.OS cs.LG

Patrick H. Coppock , Brian Zhang , Eliot H. Solomon , Vasilis Kypriotis , Leon Yang , Bikash Sharma , Dan Schatzberg , Todd C. Mowry
show 1 more author
Dimitrios Skarlatos
This is my paper · ORCID
classification cs.OScs.LG
keywords GPUoperatingsystemkernelatomizationthreadprocessingclusterschedulingmultitenancyCUDAdriverinterpositionright-sizingpowermanagementtaillatency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LithOS argues that GPU underutilization in datacenters can be attacked at the operating-system layer rather than the framework or hardware layer. It interposes at the CUDA driver boundary and, without modifying applications, splits long-running kernels into atoms, contiguous ranges of thread blocks, and schedules those atoms onto individual thread processing clusters (TPCs), lending idle TPCs to other workloads. The paper reports that this transparent scheduling reduces P99 tail latency by 13x versus MPS and 3x versus the best prior software baseline for inference stacking, and by 4.7x versus MPS and 1.18x versus TGS for hybrid inference/training, while aggregate throughput rises by 1.6x and 1.35x respectively. It also claims that right-sizing TPC counts saves a quarter of GPU capacity on average for under a 4% latency cost, and that transparent frequency scaling saves a quarter of GPU energy for a 7% cost. A sympathetic reader would care because the result suggests that an OS-style resource manager, not new hardware or rewritten models, could provide MIG-like isolation together with high utilization.

What carries the argument

The load-bearing mechanism is the Kernel Atomizer, implemented by a prelude kernel (Algorithm 1): a wrapper launched with the same grid configuration and resources as the original kernel that checks each block index against an atom's metadata and invokes the original kernel entry point only for blocks in the atom's range. Because the grid is partitioned into non-overlapping block-index ranges, a monolithic kernel becomes many independently schedulable atoms, which the TPC Scheduler places onto individual thread processing clusters with TPC stealing. Two supporting models carry the resource-efficiency claims: a scaling curve of the form $l = m/t + b$ fitted from measured latencies at one TPC and at all TPCs, used to right-size each kernel under a user-specified latency slip $k$, and a sequence-based DVFS model that weights each kernel's frequency sensitivity by its share of a stream's total runtime to pick a safe final frequency.

What would settle it

Run LithOS on kernels that use grid-wide synchronization, cooperative groups, or cross-block communication and compare outputs bit-for-bit against monolithic launches; any divergence shows atomization is not universally correct. Separately, with MPS active, use hardware counters to check whether a high-priority workload's TPC allocation is truly respected while a best-effort kernel is running.

Watch

Extended reading notes

Core claim

The central claim is that a transparent OS layer for GPUs can decouple kernel submission from execution by atomizing kernels at runtime and scheduling atoms at TPC granularity, eliminating head-of-line blocking without relying on hardware preemption. Atomization works by launching a prelude kernel that reads each thread block's index and calls the original kernel's entry point only for blocks in the atom's index range, which is correct provided thread blocks are independent. The TPC scheduler uses predicted atom durations and per-TPC timers to steal idle TPCs from underutilized workloads while protecting latency-critical work through layered priorities and limits on outstanding atoms. On top of this, a two-point Amdahl-style scaling model right-sizes TPC allocation per kernel, and a sequence-weighted frequency model guides DVFS, together yielding the paper's reported latency, capacity, and energy numbers.

Load-bearing premise

Two premises carry LithOS: that thread blocks in a kernel are independent enough that splitting a grid by block-index ranges and letting out-of-range blocks exit early never changes the result, and that a CUDA driver-interposition library on top of MPS can actually enforce per-TPC allocation; if either fails, the reported latency and isolation numbers do not generalize.

Editorial extensions

If this is right

  • Datacenter GPUs can be shared among latency-critical and best-effort ML workloads without application, framework, or compiler changes; a driver-level library is sufficient.
  • A long-running training kernel no longer blocks latency-critical inference for its full duration, because atomization bounds head-of-line blocking to the duration of a single atom.
  • Kernel-level right-sizing captures capacity savings that whole-model partitioning misses, since individual kernels within one model scale very differently with TPC count.
  • Transparent DVFS can save about a quarter of GPU energy at a 7% P99 latency cost without offline profiling, by weighting kernels according to their contribution to stream runtime.
  • TPC-level spatial isolation plus stealing gives SLO attainment comparable to hardware partitioning while still allowing best-effort throughput that static partitions cannot support.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If atomization is safe in production, GPU scheduling gains a software-only analogue of CPU preemption, which could unlock standard OS policies such as priority inheritance, fairness, and work conservation on GPUs; the paper only hints at these policies.
  • Atomization correctness depends on thread-block independence, so the natural test is to run grids that use grid-wide synchronization or cooperative groups; the paper does not identify or test such kernels.
  • The reported right-sizing savings should grow on future GPUs with more TPCs, since finer scheduling granularity makes it easier to match allocation to a kernel's intrinsic parallelism.
  • The same interposition layer could extend the atom/TPC abstraction to other contended resources such as memory bandwidth or L2 capacity, but the paper does not evaluate that direction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents LithOS, a userspace GPU resource-management layer that interposes on the CUDA driver API and claims to provide transparent, fine-grained spatial scheduling of ML workloads. Its main components are a TPC-level scheduler with TPC stealing, a kernel atomizer that splits kernels into thread-block ranges, a right-sizing mechanism that reduces TPC allocations, and a DVFS power-management mechanism. The evaluation, on an A100 with inference and training workloads, claims that LithOS reduces tail latencies by 13x versus MPS and by up to 3x versus the best prior software baseline for inference stacking, while also providing capacity and energy savings of roughly a quarter. The central design claim is that atomization is transparent and requires no source, PTX, compiler, or runtime modifications.

Significance. If the central mechanism worked, LithOS would be a notable step toward OS-like GPU management: it targets a real production utilization problem, evaluates across many models and frameworks, and compares against four NVIDIA mechanisms plus three research baselines. The evaluation breadth is a genuine strength, as is the attempt to make all ML-stack components unmodified. However, the contribution stands or falls on the kernel atomizer and TPC-level placement, and the manuscript as written does not provide a credible mechanism for either. The right-sizing and DVFS results are also of limited independent value until the scheduling foundation is established. The paper is therefore potentially significant but currently lacks support for its load-bearing premise.

major comments (4)
  1. [§4.4, Algorithm 1; also §1 and §7.1] The load-bearing atomization mechanism is not expressible in CUDA as described. In Algorithm 1, the prelude kernel calls `atom->kernel_entrypoint(*args)` from device code; a __global__ function cannot be called as an ordinary device function and its address cannot be taken for device-side invocation in the CUDA programming model, and the paper explicitly states (§4.4, §5) that LithOS has no source or PTX access. This makes it impossible to "call into the original kernel" for arbitrary cuDNN or TensorRT kernels. Furthermore, every atom is launched with the original full grid configuration, so the GPU's hardware block scheduler still distributes blocks across all TPCs; therefore the TPC Scheduler cannot confine an atom to a chosen subset of TPCs, which is exactly what TPC Stealing and per-TPC isolation require. The headline claims in §1 and §7.1 (13x and 3x tail-latency reductions, 1.35x aggregate throughput) all depend on this mechanism.
  2. [§4.4] Atomization by block-index range preserves semantics only if thread blocks are independent and have no grid-wide interactions. Kernels using cooperative-groups grid.sync(), persistent-CTA work queues, atomic-counter barriers, or cross-block producer-consumer patterns can deadlock, double-execute work, or skip required finalization when split by block-index ranges. The paper reports no audit of the §7 kernels for such patterns and no output-equivalence check against unatomized execution; without this, the evaluated results may reflect incorrect executions. Please provide evidence of semantic preservation or restrict the claim accordingly.
  3. [§7.2] The reported right-sizing accuracy is computed in-sample: the text says the model fits each kernel curve and then computes R² for the "curves we fit" on the same data. With the two-parameter form l = m/t + b and the two-point interpolation described in §4.5, R² would be 1 by construction, which is inconsistent with the reported 0.92–0.99 range. Please clarify whether the R² values come from held-out data and report prediction error on unseen TPC allocations; otherwise the 26% capacity-savings claim is not supported by the accuracy evidence.
  4. [§6 and §7.1] The comparison to Orion and REEF uses the authors' reimplementations instead of the original systems, with no validation of those reimplementations (e.g., against the papers' reported numbers), and all multitenancy results are point estimates without confidence intervals. Moreover, §5 states that "some low-level details" of TPC-level control are deferred to a technical report, so neither the TPC placement mechanism nor the reimplemented baselines can be independently checked from the manuscript. Given that the 3x and 1.18x "best SotA" claims depend on exact baseline numbers, this is a reproducibility gap for the central evaluation.
minor comments (4)
  1. [§4.6] The symbol k is used both for the configured latency slip parameter and for the per-kernel relative slowdown in the DVFS formula, which is confusing; please use a distinct symbol for the slowdown.
  2. [§4.4] The discussion of prelude resources should state explicitly how shared memory, registers, and dynamic shared memory are replicated from the original kernel, since the atomization overhead claims depend on this.
  3. [Figure 19] The legend "MPS + TPC Scheduling + Kernel Atomization" is not labeled as cumulative; please clarify that each line adds one feature.
  4. [References] Reference [23] contains a typo: "unjie Qian" should be "Junjie Qian."

Circularity Check

1 steps flagged · score 2.0 of 10

Minor in-sample circularity in the right-sizing accuracy metric; headline multitenancy results are external measurements and self-contained.

  1. fitted input called prediction [Section 7.2, "Accuracy" (R2 of fitted curves), with the model defined in Section 4.5, "Modeling Kernel Scaling"]
    "To quantify the accuracy of our prediction technique, we compute the kernel-execution-time weighted average of the R2 values for the curves we fit (i.e., for kernels where the possible TPCs value exceeds the threshold). Across all of the evaluated workloads, the average R2 values range from 0.92 (Llama finetuning) to 0.99 (RetinaNet inference), indicating that our technique is highly accurate."

    Section 4.5 constructs the scaling model by fitting l = m/t + b to measured kernel latencies. Section 7.2 then reports the R2 of those same fitted curves as the "accuracy of our prediction technique." R2 computed on the data used for the fit measures how well the curve reproduces its own inputs; it is a goodness-of-fit statistic, not a held-out predictive validation. The claimed "prediction accuracy" therefore reduces, by construction, to the self-consistency of the fitted curve rather than to any independent forecast. The impact is limited because the headline multitenancy results and the capacity/energy savings are external measurements, not derived from this R2 metric.

full rationale

The central claims are not circular: the multitenancy results of Section 7.1 are measured end-to-end against external baselines (MPS, MIG, time slicing, TGS, REEF, Orion), and the right-sizing and DVFS benefits in Sections 7.2 and 7.3 are evaluated by their measured capacity, energy, and latency effects rather than by re-inserting LithOS's own fitted values as predicted outcomes. The scaling forms l = m/t + b and the first-order Taylor DVFS model are explicit modeling assumptions, not results imported from the authors' prior work via self-citation, and no uniqueness theorem or load-bearing self-citation appears. The one reduction-to-input I can exhibit is localized to Section 7.2: the R2 statistic for the fitted kernel-scaling curves is presented as predictive accuracy even though it is an in-sample fit-quality measure. This is a genuine but minor methodological circularity that does not support the headline performance comparisons. Separately, Section 5 defers low-level implementation details to a technical report, and Algorithm 1's device-side call to a __global__ entrypoint raises CUDA-expressibility questions; these are correctness and completeness risks, not circularity. Overall, the derivation chain is largely self-contained, so the circularity score is low.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a software mechanism and an evaluation. The main free parameters are the atomization time target, the latency-slip policy knob, and per-kernel parameters fitted at runtime for right-sizing and DVFS. The key axiomatic premise is that arbitrary CUDA kernels can be split by block-index ranges without changing results; this is assumed in Algorithm 1 and not justified for kernels with grid-wide dependencies. No new physical entities are introduced.

free parameters (4)
  • atom_duration = not reported
    Tunable target atom length that controls how aggressively kernels are split; the paper says it is adjusted dynamically but gives no default or tuning protocol.
  • latency_slip_k = 1.1 in evaluation
    User-specified multiplicative performance loss bound used by both right-sizing and DVFS; the reported capacity and energy savings depend on this choice.
  • per-kernel right-sizing parameters m and b = fit at runtime from all-TPC and one-TPC latencies
    The curve l = m/t + b is fitted per kernel from two observed points; capacity savings are derived from this fit.
  • per-kernel frequency sensitivity s = learned online from observed kernel responses
    DVFS decisions use s = k / (fmax/fth - 1), updated during execution; energy savings are derived from this learned parameter.
assumptions (4)
  • domain assumption Atomizing a kernel into non-overlapping block-index ranges preserves the kernel's semantics and makes early-exiting blocks cheap.
    Algorithm 1 assumes thread blocks are independent; no handling is given for grid-wide synchronization, persistent kernels, or cross-block dependencies.
  • domain assumption LithOS can enforce per-TPC spatial allocation and stealing through the CUDA driver and MPS layer without hardware changes.
    Section 5 says the implementation builds on MPS and defers low-level details to a technical report; this premise is load-bearing for the TPC scheduler claim.
  • domain assumption Kernel launch ordinal indices uniquely identify operator nodes across batches.
    Section 4.7 uses the kth kernel after batch start as an operator identity, which assumes deterministic and stable launch order.
  • ad hoc to paper A two-point Amdahl-style curve l = m/t + b captures per-kernel TPC scaling.
    Section 4.5 fits this form without validating it against alternative scaling models; an outlier filter handles kernels that deviate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LithOS: An Operating System for Efficient Machine Learning on GPUs." pith.science (2026). https://pith.science/paper/4EU67LSE

@misc{pith2026250415465,
  author       = {Pith},
  title        = {Pith review of: LithOS: An Operating System for Efficient Machine Learning on GPUs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4EU67LSE}},
  note         = {Machine review of arXiv:2504.15465}
}
read the original abstract

The surging demand for GPUs in datacenters for machine learning (ML) has made efficient GPU utilization crucial. However, meeting the diverse needs of ML models while optimizing resource usage is challenging. To enable transparent, fine-grained GPU management that maximizes utilization and energy efficiency while maintaining strong isolation, an operating system (OS) approach is needed. This paper introduces LithOS, a first step toward a GPU OS. LithOS includes the following new abstractions and mechanisms for efficient GPU resource management: (i) a novel TPC Scheduler that supports spatial scheduling at the granularity of individual TPCs, unlocking efficient TPC stealing between workloads; (ii) transparent kernel atomization to reduce head-of-line blocking and enable dynamic resource reallocation mid-execution; (iii) a lightweight hardware right-sizing mechanism that determines the minimal TPC resources needed per atom; and (iv) a transparent power management mechanism that reduces power consumption based on in-flight work behavior. We implement LithOS in Rust and evaluate its performance across extensive ML environments, comparing it to state-of-the-art solutions from NVIDIA and prior research. For inference stacking, LithOS reduces tail latencies by 13x compared to MPS; compared to the best SotA, it reduces tail latencies by 3x while improving aggregate throughput by 1.6x. In hybrid inference-training stacking, LithOS reduces tail latencies by 4.7x compared to MPS; compared to the best SotA, it reduces tail latencies 1.18x while improving aggregate throughput by 1.35x. Finally, for a modest performance hit under 4%, LithOS's right-sizing provides a quarter of GPU capacity savings on average, while for a 7% hit, its power management yields a quarter of a GPU's energy savings. Overall, LithOS increases GPU efficiency, establishing a foundation for future OS research on GPUs.

Figures

Figures reproduced from arXiv: 2504.15465 by the authors.

Figure 1
Figure 1. GPU utilization metrics over a week in a production Ads inference service at Meta. remain significantly underutilized. Public reports from Mi￾crosoft and Alibaba cite average and median GPU utilization rates of just 52% [23] and 10% [49], respectively. Our analysis of a production Ads service at Meta reveals similarly low utilization, averaging just 27%, as shown in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. GPU timeline showcasing the pitfalls of MPS. sharing between containerized applications. In practice, un￾cooperative tasks and limited application-specific informa￾tion make transparent multitasking a serious challenge. Temporal multitenancy. Temporal multitenancy ded￾icates the entire GPU to a single task at a time via native time slicing or software scheduling. Some approaches work at the level of entire inference… view at source ↗
Figure 4
Figure 4. Mean normalized traffic. A B C D E F G H I J K L M Model ID 1 10 100 1000 Model Frequency (log scale) [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (14 more)
Figure 5
Figure 5. Figure 5: Model frequency distribution. A B C D E F G H I J K L M Model ID 0 2 4 6 8 10 12 Normalized Model Size [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Model size distribution. popular model M. Over-provisioning GPUs for such a wide request distribution can lead to underutilization, particularly for less popular models. Model Sizes. To better understand GPU utilization, we examine the sizes of the most commonly used m…
Figure 7
Figure 7. Figure 7: LithOS architecture overview. 4 LithOS Design We propose LithOS, an OS designed to address GPU ineffi￾ciencies in datacenters. LithOS operates transparently across the ML stack, enabling efficient machine learning on GPUs. 4.1 Architecture Overview [PITH_FULL_IMAGE:fi…
Figure 8
Figure 8. Figure 8: LithOS operations overview. maintains per-TPC timers informed by a latency predic￾tion module, estimating kernel (and atom) durations at sub￾mission time. These timers help avoid stealing from long￾running TPCs. As tasks complete, sync queues are cleared and timers upd…
Figure 9
Figure 9. Figure 9: GPU timeline for two workloads showcasing (a) TPC Scheduling, (B) Stealing, and (C) Atomization. 0 10 20 30 40 Training GPU Memory Usage (GiB) 0 10 20 30 P99 K ern el L ate n c y (m s) DLRM BERT MobileNet ResNet VGG Llama 3 GPT-J Inference Model 0 1 2 3 4 5 S M L [PIT…
Figure 10
Figure 10. Figure 10: (a) 𝑃99 kernel latency at different training batch sizes normalized to memory usage. (b) 𝑃99 kernel latency for different inference prompt sequence lengths for LLMs. needs to know the entry point to the Conv kernel. The Kernel Atomizer passes this information to the P…
Figure 11
Figure 11. Figure 11: LithOS’s interpolated TPC scaling curves. modeling and scaling techniques offer a robust and accurate solution—as we will see in Section 7.2. 4.6 Transparent Power Management LithOS is well-positioned to enable transparent and effi￾cient power management via DVFS. Jus…
Figure 14
Figure 14. Figure 14: Inference-only multitenancy: goodput by app [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]
Figure 13
Figure 13. Figure 13: SLO attainment and throughput by system. 7.1 Performance in Multitenant Environments In the following experiments, we disable right-sizing and power management features of LithOS to provide an apples￾to-apples comparison to other systems in terms of scheduling efficie…
Figure 15
Figure 15. Figure 15: Inference stacking multitenancy: HP A tail latencies by model. Llama 3 RetinaNet GPT-J BERT YOLO High-priority Inference Model 0 2 4 P99 L ate ncy (×) 5 4 7 9545 Llama 3 RetinaNet GPT-J BERT YOLO High-priority Inference Model 0.0 0.5 1.0 1.5 Throughput (×) BE HP ideal…
Figure 16
Figure 16. Figure 16: Hybrid multitenancy: (a) 𝑃99 service latency and (b) aggregate throughput. Hybrid Inference/Training Multitenancy. In this ex￾periment, we stack an HP that has a latency-oriented SLO with a training BE app. Similar to the inference-stacking experiment, resources unuse…
Figure 17
Figure 17. Figure 17: Hardware right-sizing GPU capacity savings. tail latencies to reach 8.93×. In contrast, LithOS maintains a tail latency within 20% of the ideal. On average, this is a 2.34× and 1.18× over REEF and TGS, respectively. Com￾pared to the native MPS solution, LithOS reduces…
Figure 18
Figure 18. Figure 18: Power management GPU energy savings. Llama 3 RetinaNet GPT-J BERT YOLO High-priority Inference Model 0 2 4 P99 L ate ncy (×) 5 4 7 9 MPS + TPC Scheduling + Kernel Atomization [PITH_FULL_IMAGE:figures/full_fig_p012_18.png]
Figure 19
Figure 19. Figure 19: Breakdown of LithOS features for inf-train. execution time of each inference or training iteration is spent inside a GPU kernel; this does not impede tuning in practice. Accuracy. To quantify the accuracy of our prediction technique, we compute the kernel-execution-ti…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. REACH: Reinforcement Learning for Efficient Allocation in Community and Heterogeneous Networks

    cs.NI 2025-08 unverdicted novelty 5.0 of 10

    REACH is claimed to improve task completion by up to 17%, double high-priority success, and cut bandwidth penalties by over 80% in simulations of community GPU scheduling.

Reference graph

Works this paper leans on

63 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    2020.{PipeSwitch}: Fast pipelined context switching for deep learning applications

    Zhihao Bai, Zhen Zhang, Yibo Zhu, and Xin Jin. 2020.{PipeSwitch}: Fast pipelined context switching for deep learning applications. In14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). 499–514

    work page 2020

  2. [2]

    Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao

    work page

  3. [3]

    Qichen Chen, Hyerin Chung, Yongseok Son, Yoonhee Kim, and Heon Young Yeom. 2021. smCompactor: a workload-aware fine- grained resource management framework for GPGPUs. In Proceedings of the 36th Annual ACM Symposium on Applied Computing (Virtual Event, Republic of Korea) (SAC ’21). Association for Computing Ma- chinery, New York, NY, USA, 1147–1155. doi:10...

    arXiv 2021

  4. [4]

    Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. 2022. Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Shar- ing. In 2022 USENIX Annual Technical Conference (USENIX ATC 22) . USENIX Association, Carlsbad, CA, 199–216. https://www.usenix. org/conference/atc22/presentation/choi-seungbeom

    work page 2022

  5. [5]

    Marcus Chow, Ali Jahanshahi, and Daniel Wong. 2023. KRISP: En- abling Kernel-wise RIght-sizing for Spatial Partitioned GPU Inference Servers. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 624–637. doi:10.1109/HPCA56546.2023. 10071121

    arXiv 2023

  6. [6]

    Marcus Chow and Daniel Wong. 2024. CoFRIS: Coordinated Frequency and Resource Scaling for GPU Inference Servers. In Proceedings of the 14th International Green and Sustainable Computing Conference (Toronto, ON, Canada) (IGSC ’23). Association for Computing Machin- ery, New York, NY, USA, 45–51. doi:10.1145/3634769.3634808

    arXiv 2024

  7. [7]

    NVIDIA Corporation. [n. d.]. Multi-Process Service. https://docs. nvidia.com/deploy/mps/index.html. Accessed: April 14, 2025

    work page 2025

  8. [8]

    NVIDIA Corporation. 2023. NVIDIA H100 Tensor Core GPU Architec- ture. Technical Report. NVIDIA Corporation, Santa Clara, CA

    work page 2023

Show all 63 references
  1. [9]

    NVIDIA Corporation. 2024. Triton Inference Server. https://developer. nvidia.com/triton-inference-server. Accessed: May 8, 2024

    2024

  2. [10]

    NVIDIA Corporation. 2025. NVIDIA Multi-Instance GPU User Guide. https://docs.nvidia.com/datacenter/tesla/mig-user-guide/ index.html. Accessed: April 14, 2025

    2025

  3. [11]

    NVIDIA Corporation. 2025. NVIDIA RTX BLACKWELL GPU ARCHI- TECTURE. https://images.nvidia.com/aem-dam/Solutions/geforce/ blackwell/nvidia-rtx-blackwell-gpu-architecture.pdf

    2025

  4. [12]

    Franklin, Joseph E

    Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: A Low-Latency Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17) . USENIX As- sociation, Boston, MA...

    2017

  5. [13]

    Weihao Cui, Han Zhao, Quan Chen, Ningxin Zheng, Jingwen Leng, Jieru Zhao, Zhuo Song, Tao Ma, Yong Yang, Chao Li, and Minyi Guo

  6. [14]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

  7. [15]

    Aditya Dhakal, Sameer G Kulkarni, and K. K. Ramakrishnan. 2020. GSLICE: Controlled Spatial Sharing of GPUs for a Scalable Inference Platform. In Proceedings of the 11th ACM Symposium on Cloud Comput- ing (Virtual Event, USA) (SoCC ’20). Association for Computing Ma- chinery, N...

    2020

  8. [16]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Ka- dian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and Angela et al. Fan. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  9. [17]

    Joshua Fried, Zhenyuan Ruan, Amy Ousterhout, and Adam Belay. 2020. Caladan: Mitigating Interference at Microsecond Timescales. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, 281–297. https://www.usenix.org/ conference/osd...

    2020

  10. [18]

    Yanjie Gao, Yichen He, Xinze Li, Bo Zhao, Haoxiang Lin, Yoyo Liang, Jing Zhong, Hongyu Zhang, Jingzhou Wang, Yonghua Zeng, et al

  11. [19]

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kauf- mann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like Clockwork: Performance Predictability from the Bottom Up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USEN...

    2020

  12. [20]

    Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. 2022. Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 539–

    2022

  13. [21]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs.CV] https://arxiv.org/abs/1512.03385

    2015 arXiv

  14. [22]

    Saksham Jain, Iljoo Baek, Shige Wang, and Ragunathan Rajkumar. 2019. Fractional GPUs: Software-based compute and memory bandwidth reservation for GPUs. In2019 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS). IEEE, 29–41

    2019

  15. [23]

    Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, unjie Qian, Wencong Xiao, and Fan Yang. 2019. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. In Pro- ceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference (Renton, WA...

    2019

  16. [24]

    Andreas Kosmas Kakolyris, Dimosthenis Masouros, Petros Vavarout- sos, Sotirios Xydis, and Dimitrios Soudris. 2024. SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving. arXiv:2408.05235 [cs.DC] https://arxiv.org/abs/2408.05235

    2024

  17. [25]

    Yunseong Kim, Yujeong Choi, and Minsoo Rhu. 2022. PARIS and ELSA: an elastic scheduling algorithm for reconfigurable multi-GPU 13 P. H. Coppock, B. Zhang, E. H. Solomon, V. Kypriotis, L. Yang, B. Sharma, D. Schatzberg, T. C. Mowry, and D. Skarlatos inference servers. In Procee...

    2022

  18. [26]

    Beth Kindig. 2024. AI power consumption: Rapidly becoming mission- critical. https://www.forbes.com/sites/bethkindig/2024/06/20/ai- power-consumption-rapidly-becoming-mission-critical/

    2024

  19. [27]

    Baolin Li, Tirthak Patel, Siddharth Samsi, Vijay Gadepally, and Devesh Tiwari. 2022. MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant GPU Clusters. In Proceedings of the 13th Symposium on Cloud Computing (San Francisco, California) (SoCC ’22). Association for Com...

    2022

  20. [28]

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár

  21. [29]

    Xuanzhe Liu, Yihao Zhao, Shufan Liu, Xiang Li, Yibo Zhu, Xin Liu, and Xin Jin. 2024. MuxFlow: efficient GPU sharing in production- level clusters with more than 10000 GPUs. Science China Information Sciences 67, 12 (2024), 222101. doi: 10.1007/s11432-024-4227-2

    2024 doi

  22. [30]

    Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krish- namoorthi, Ansha Yu, V...

    2019 arXiv

  23. [31]

    Microsoft Network. 2024. Dell exec reveals Nvidia has a 1,000 watt GPU in the works. https://www.msn.com/en-us/lifestyle/other/dell- exec-reveals-nvidia-has-a-1-000-watt-gpu-in-the-works/ar- BB1jlE8f. Accessed: June 24, 2024

    2024

  24. [32]

    Kelvin K. W. Ng, Henri Maxime Demoulin, and Vincent Liu

  25. [33]

    NVIDIA Corporation. [n. d.].NVIDIA CUDA Driver API Documentation: Occupancy. NVIDIA Corporation. https://docs.nvidia.com/cuda/cuda- driver-api/group__CUDA__OCCUPANCY.html

  26. [34]

    Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke

  27. [35]

    Amy Ousterhout, Joshua Fried, Jonathan Behrens, Adam Belay, and Hari Balakrishnan. 2019. Shenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter Workloads. In 16th USENIX Sym- posium on Networked Systems Design and Implementation (NSDI 19) . USENIX Association...

    2019

  28. [36]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Brijesh Warrier, Nithish Mahalingam, and Ricardo Bianchini. 2024. Charac- terizing Power Management Opportunities for LLMs in the Cloud. In Proceedings of the 29th ACM International Conference on Architectural Support f...

    2024

  29. [37]

    Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew Kalbarczyk, Tamer Başar, and Ravishankar K. Iyer. 2024. Power-aware Deep Learning Model Serving with𝜇-Serve. In 2024 USENIX Annual Technical Conference (USENIX ATC 24). USENI...

    2024

  30. [38]

    Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B

    Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, It...

    2019 arXiv

  31. [39]

    Yadwadkar, and Christos Kozyrakis

    Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021. INFaaS: Automated Model-less Inference Serv- ing. In 2021 USENIX Annual Technical Conference (USENIX ATC 21) . USENIX Association, 397–411. https://www.usenix.org/conference/ atc21/presentation/romero

    2021

  32. [40]

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition . 4510–4520

    2018

  33. [41]

    Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. 2019. Nexus: A GPU Cluster Engine for Accelerating DNN-Based Video Analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (H...

    2019

  34. [42]

    Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556 [cs.CV] https://arxiv.org/abs/1409.1556

    2015 arXiv

  35. [43]

    Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. 2024. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency. arXiv:2408.00741 [cs.AI] https: //arxiv.org/abs/2408.00741

    2024

  36. [44]

    Foteini Strati, Xianzhe Ma, and Ana Klimovic. 2024. Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications. In Proceedings of the Nineteenth European Conference on Computer Sys- tems (<conf-loc>, <city>Athens</city>, <country>Greece</country>, </conf-loc>) (Eu...

    2024

  37. [45]

    Cheng Tan, Zhichao Li, Jian Zhang, Yu Cao, Sikai Qi, Zherui Liu, Yibo Zhu, and Chuanxiong Guo. 2021. Serving DNN Models with Multi- Instance GPUs: A Case of the Reconfigurable Machine Scheduling Problem. arXiv:2109.11067 [cs.DC]

    2021 arXiv

  38. [46]

    Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Param- eter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax

    2021

  39. [47]

    Tianyu Wang, Sheng Li, Bingyao Li, Yue Dai, Ao Li, Geng Yuan, Yufei Ding, Youtao Zhang, and Xulong Tang. 2024. Improving GPU Multi-Tenancy Through Dynamic Multi-Instance GPU Reconfigura- tion. arXiv preprint arXiv:2407.13126 (2024)

    2024 arXiv

  40. [48]

    Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, and Xin Jin. 2023. Transparent GPU Sharing in Container Clouds for Deep Learning Workloads. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) . USENIX Association, Boston, MA, 69–

    2023

  41. [49]

    Wencong Xiao, Shiru Ren, Yong Li, Yang Zhang, Pengyang Hou, Zhi Li, Yihui Feng, Wei Lin, and Yangqing Jia. 2020. AntMan: Dynamic Scaling on GPU Clusters for Deep Learning. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI’20). US...

    2020

  42. [50]

    Fei Xu, Jianian Xu, Jiabin Chen, Li Chen, Ruitao Shang, Zhi Zhou, and Fangming Liu. 2023. iGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloud. IEEE Transactions on Parallel and Distributed Systems 34, 3 (2023), 812–827. doi:10.1109/...

    2023

  43. [51]

    Hangchen Yu, Arthur Michener Peters, Amogh Akshintala, and Christopher J Rossbach. 2020. AvA: Accelerated virtualization of ac- celerators. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. 807–825

    2020

  44. [52]

    Yijia Zhang, Qiang Wang, Zhe Lin, Pengxiang Xu, and Bingqiang Wang. 2024. Improving GPU Energy Efficiency through an Application-transparent Frequency Scaling Policy with Performance Assurance. In Proceedings of the Nineteenth European Conference on Computer Systems (Athens, G...

    2024 doi

  45. [53]

    Yongkang Zhang, Haoxuan Yu, Chenxia Han, Cheng Wang, Baotong Lu, Yunzhe Li, Zhifeng Jiang, Yang Li, Xiaowen Chu, and Huaicheng Li. 2025. SGDRC: Software-Defined Dynamic Resource Control for Concurrent DNN Inference on NVIDIA GPUs. In Proceedings of the 30th ACM SIGPLAN Annual ...

    2025

  46. [54]

    Xia Zhao, Magnus Jahre, and Lieven Eeckhout. 2020. HSM: A Hy- brid Slowdown Model for Multitasking GPUs. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Pro- gramming Languages and Operating Systems (Lausanne, Switzerland) (ASPLOS ’20)....

    2020

  47. [85]

    https://www.usenix.org/conference/nsdi23/presentation/wu

  48. [558]

    https://www.usenix.org/conference/osdi22/presentation/han

  49. [2017]

    arXiv:1712.06139 [cs.DC]

    TensorFlow-Serving: Flexible, High-Performance ML Serving. arXiv:1712.06139 [cs.DC]

    arXiv

  50. [2018]

    arXiv:1708.02002 [cs.CV] https://arxiv.org/abs/1708.02002

    Focal Loss for Dense Object Detection. arXiv:1708.02002 [cs.CV] https://arxiv.org/abs/1708.02002

    arXiv

  51. [2019]

    arXiv:1810.04805 [cs.CL] https://arxiv.org/ abs/1810.04805

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs.CL] https://arxiv.org/ abs/1810.04805

    arXiv

  52. [2020]

    arXiv:2004.10934 [cs.CV] https://arxiv.org/abs/2004.10934

    YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv:2004.10934 [cs.CV] https://arxiv.org/abs/2004.10934

    2004 arXiv

  53. [2021]

    In Proceedings of the International Conference for High Performance Computing, Network- ing, Storage and Analysis (St

    Enable simultaneous DNN services based on deterministic op- erator overlap and precise latency prediction. In Proceedings of the International Conference for High Performance Computing, Network- ing, Storage and Analysis (St. Louis, Missouri) (SC ’21). Association for Computin...

  54. [2023]

    In Proceedings of the 29th Symposium on Oper- ating Systems Principles (<conf-loc>, <city>Koblenz</city>, <coun- try>Germany</country>, </conf-loc>) (SOSP ’23)

    Paella: Low-latency Model Serving with Software-defined GPU Scheduling. In Proceedings of the 29th Symposium on Oper- ating Systems Principles (<conf-loc>, <city>Koblenz</city>, <coun- try>Germany</country>, </conf-loc>) (SOSP ’23) . Association for Computing Machinery, New Yo...

  55. [2024]

    In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

    An Empirical Study on Low GPU Utilization of Deep Learning Jobs. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.