Pith. sign in

REVIEW 4 major objections 5 minor 55 references

ElastiCo: Elastic Configuration and Interference-Aware Orchestration for GPU Clusters

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ElastiCo argues that treating a job's configuration as elastic rather than fixed is the missing lever for GPU cluster efficiency, and shows that jointly reshaping jobs, pricing resources, and predicting interference cuts average job…

desk verdict A genuinely new combination of configuration reshaping, shadow pricing, and interference-aware co-location, but the headline JCT claims rest on an undefined training-completion criterion and same-distribution evaluation; it deserves serious review, not immediate acceptance. read the letter →

arxiv 2608.07971 v1 pith:QZUGGD3A submitted 2026-08-08 cs.DC

Jinghao Wang , Yihang Zhou , Xiaoyang Sun , Chunming Hu , Tianyu Wo , Xu Wang , Albert Y. Zomaya , Renyu Yang This is my paper · ORCID
classification cs.DC
keywords GPUclusterschedulingjobco-locationinterference-awareelasticconfigurationshadowpricingdeeplearningtrainingLLMinferenceKubernetes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ElastiCo claims that GPU clusters serving both deep-learning training and offline LLM inference waste most of their capacity because each job is locked into one fixed resource configuration. The paper proposes to treat configuration flexibility as a first-class scheduling dimension: each job is profiled into a family of feasible resource-performance shapes, the scheduler prices every resource with dynamic shadow prices so each job picks its best shape independently, and a learned model predicts how much any two jobs will slow each other down if co-located. On a 64-GPU A100 testbed and in trace-driven simulations up to 512 GPUs, this combined mechanism reduces average job completion time by up to 2.94x, raises cluster throughput by 2.02x, and lifts GPU SM utilization from about 25% to 46% compared with a static scheduler. The ablation shows the largest single contribution comes from reshaping jobs' resource demands rather than from job ordering or packing.

What carries the argument

The load-bearing object is the Lagrangian decomposition of the joint configuration-and-allocation integer program together with the interference penalty that couples the two halves. Concretely, RST builds a profile family for each job (batch size, activation checkpointing, mixed precision, KV-cache settings), ESP assigns a shadow price $\lambda_r$ to each resource and lets every job pick the profile minimizing performance cost plus resource rent plus interference penalty, and IAC trains a three-layer DNN to predict slowdown $\hat{S}_{i|j}$ from hardware counters, task attributes, and pairwise features, with the constraint that predictions stay $\ge 1$. The Phase-Aware Disaggregated Scheduling module closes the loop by reserving capacity for the queue, re-running price updates and interference checks for a few rounds, and then applying configuration switches or migrations. The whole epoch costs roughly $O(M \cdot I \cdot N \cdot \bar{K} + N^2 F)$ and stays under 15 ms at 200 jobs.

What would settle it

Run ElastiCo on a cluster whose workload mix includes a model architecture with a markedly different hardware utilization profile (for example, mixture-of-experts or an image-diffusion model) that was never in the training set, and compare the IAC predictor's slowdown estimates against measured co-location throughput; if the predictor's MAPE well exceeds the reported 7.6% or if the scheduler admits pairs whose true performance decay falls below 0.6, the end-to-end JCT and utilization gains would not transfer to that mix.

Watch

Extended reading notes

Core claim

The paper's central claim is that configuration selection, multi-resource allocation, and interference awareness are three interdependent parts of one problem, and that solving them jointly is what unlocks the idle capacity in GPU clusters. Its mechanism is a closed loop: Resource Shape Transformation (RST) exposes each job as a set of discrete profiles trading throughput against memory and compute; Elastic Shadow Pricing (ESP) relaxes the capacity constraints with Lagrange multipliers so each job independently minimizes a score that includes a resource rent and an interference penalty; and Interference-Aware Co-location (IAC) supplies those penalties from a neural network that predicts pairwise slowdown from 36 hardware-counter and task-level features. The loop repeats until the placement is capacity-feasible and interference-safe. The paper argues that this joint design, not any single component, delivers the reported 2.94x JCT reduction, 2.02x throughput gain, and 44% reduction in extra GPU instances, and the ablation study attributes the largest individual gain to RST, with a 53% JCT increase when it is disabled.

Load-bearing premise

The interference predictor, trained on pairwise measurements of the twelve evaluation workloads, must keep its accuracy when the cluster runs models and sharing patterns outside that training set, because the scheduler's co-location safety decisions rest on those predictions.

Editorial extensions

If this is right

  • Schedulers that treat a job's resource request as fixed leave a large, exploitable fraction of GPU capacity idle; exposing configuration families is a direct way to recover it.
  • A market-style pricing loop with per-resource shadow prices can make configuration selection tractable even when each job has dozens of feasible profiles and the cluster runs hundreds of jobs.
  • Interference prediction must be configuration-aware: the same two jobs can be a safe pair under one set of knobs and a harmful pair under another, so co-location admission should be based on predicted slowdown, not just memory budgets.
  • Co-location decisions can be steered by charging jobs for the interference they impose, converting a combinatorial safety check into a penalty term in an otherwise decomposable optimization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The pricing formulation suggests a natural extension to multi-GPU parallelism: if data, tensor, and pipeline parallel strategies are added as profile dimensions, the same shadow-price loop could co-select parallelism degree and per-GPU knobs, which the paper itself lists as future work.
  • Because the IAC predictor's accuracy is the main transfer risk, an online-learning variant that updates on newly measured co-location pairs would plausibly let ElastiCo absorb novel architectures (for example, mixture-of-experts or diffusion models) without offline re-training.
  • The 44% reduction in extra GPU instances implies a direct hardware-cost consequence: a cluster operator buying capacity for a fixed workload mix could buy roughly half as many additional GPUs, which is a sharper economic statement than the JCT or utilization numbers alone.
  • The weak negative correlation between training and inference demand observed in the traces suggests the approach would benefit any cluster whose workload classes have infrequently coinciding peaks; testing on traces with strong positive correlation would bound the generality of the utilization gain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. ElastiCo is presented as a Kubernetes-native middleware for GPU clusters that jointly schedules deep-learning training and offline LLM inference workloads. It combines three mechanisms: Resource Shape Transformation (RST), which exposes each job as a family of resource/performance configurations; Elastic Shadow Pricing (ESP), a Lagrangian-relaxation-based pricing scheme that decomposes the allocation problem into per-job configuration selection; and Interference-Aware Co-location (IAC), a DNN-based predictor of pairwise slowdown under GPU sharing. A fourth module, Phase-Aware Disaggregated Scheduling (PDS), closes the loop with queue-aware capacity reservation, reconfiguration, and migration. The paper evaluates ElastiCo on a 64-GPU A100 testbed and in trace-driven simulations up to 512 GPUs, reporting average JCT reductions of up to 2.94x, cluster throughput improvements of 2.02x, and GPU SM utilization rising from about 25% to 46% relative to static scheduling.

Significance. If the results hold, ElastiCo makes a strong contribution to elastic GPU cluster scheduling by treating intra-job configuration flexibility as a first-class dimension and coupling it with interference-aware pricing. The ablation study usefully attributes the largest gain to RST (53% JCT increase when disabled), and the IAC predictor is reported with a 7.6% MAPE and R^2=0.89. The system is implemented without user-code modification and the paper includes a careful accounting of scheduling latency and reconfiguration overhead. These strengths make the paper of clear interest to the systems community. However, the headline JCT and throughput numbers rest on an undefined training completion criterion and on an evaluation that is in-distribution with respect to the interference predictor, so the quantitative claims need substantial strengthening before acceptance.

major comments (4)
  1. [§3.1.2, §3.2.1, §5.2, Table 4] The headline JCT result is not established because the manuscript never defines what 'completion' means for training jobs. Section 3.1.2 defines T_j(c) as 'training throughput (samples or iterations per second)', the ESP objective (Eq. 1) maximizes this throughput subject only to a throughput floor L_j (Eq. 4), and Table 4's attainment metric is the fraction of jobs meeting the throughput target, not an accuracy or convergence target. When RST selects a smaller micro-batch (Table 2 lists batch sizes down to 32) or switches configurations mid-run, the samples-per-gradient-step changes, and no learning-rate scaling is mentioned anywhere in the paper. If jobs run to a fixed iteration count, a reshaped job with a smaller batch completes after fewer samples and does less useful training work; if jobs run to a fixed epoch count, T_j(c) in samples/sec is not a valid proxy for time-to-convergence because smaller batches typically require more iterations to reach the same accuracy. The 2.94x JCT reduction and 2.02x throughput improvement are therefore not shown to be improvements in real training completion time. The authors must state the termination criterion and demonstrate that all selected configurations reach the same target accuracy, or adopt a goodput-style model (as in Pollux) that accounts for statistical efficiency.
  2. [§3.2.2 (Eq. 6) vs §3.2.3 (Eq. 7)] The implemented scoring function does not match the claimed Lagrangian decomposition. The per-job subproblem in Eq. (6) maximizes w_j T_j(c) - sum_r lambda_r R_j(c)[r], where w_j is a per-job fairness weight (e.g., inverse remaining iterations). Eq. (7) instead minimizes Score_j(c) = 1 - T_j(c)/T_j(c_max) + sum_r lambda_r R_j(c)[r] + Gamma_j(c), which is equivalent to maximizing T_j(c)/T_j(c_max) - sum_r lambda_r R_j(c)[r] and drops w_j entirely. Thus the 'weighted throughput' objective of Eq. (1) is not what the scheduler optimizes, and the fairness rationale for w_j is not realized. Please either incorporate w_j into the implemented score or explicitly state that the practical scheduler uses an unweighted, normalized objective.
  3. [§5.5, §7 (Limitation 4)] The interference predictor is trained and evaluated in-distribution with respect to the workloads used in the end-to-end tests. Section 3.3.2 states that training data is generated by sampling pairs of the workloads listed in Table 2; Section 5.5 cross-validates on the same 12 workload families; and the 512-GPU simulation uses the same predictor and, per §5.1, is calibrated on the identical testbed trace. The 7.6% MAPE therefore does not demonstrate that IAC generalizes to workloads outside the evaluation mix, and the §7 limitation 4 concession ('accuracy may degrade on model architectures with substantially different hardware utilization profiles') applies directly to the reported JCT and throughput gains. Please add a held-out-workload evaluation (e.g., MoE or diffusion models) and show that the scheduler's admission decisions remain correct outside the training distribution.
  4. [Table 4, §5.2, §5.6] The central quantitative claims are reported without error bars, confidence intervals, or any indication of run-to-run variance. Table 4 reports a single 24-hour replay for each system, and the scalability numbers in Table 8 likewise appear to be single simulation runs. Because scheduling outcomes under dynamic workload arrivals can be highly variable, a 2.94x JCT improvement from a single trace does not support the strength of the abstract's claims. Please report the mean and spread (or at least multiple seeds) for the testbed and simulation experiments, and state the number of independent runs.
minor comments (5)
  1. [Figures 6, 8, 9] The figure text in the manuscript contains long placeholder strings such as '/uni00000013/uni00000011/...' that appear to be PDF-extraction artifacts; these should be replaced with the intended axis labels and legends.
  2. [§3.3.2] For reproducibility, the DNN hyperparameters should be reported: layer widths, learning rate, number of epochs, batch size, and early-stopping criteria are all absent.
  3. [Table 3] The 'Open source' column lists a dash for ElastiCo; please clarify whether the code is planned to be released, since several baselines are excluded for lack of public implementations.
  4. [§3.3.3, Eq. (11)] The parameter beta is introduced as controlling sensitivity but no default value is given, and beta_max is never assigned a concrete value in the experiments; please specify the values used and any sensitivity analysis.
  5. [§5.7] The reconfiguration overhead is reported as consuming less than 0.3% of total GPU-hours, but it is unclear whether the time spent in checkpoint-resume transitions is included in the reported JCT values; please clarify whether the 2.3 transitions/hour/GPU are part of the completion time.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central JCT/throughput claims are measured system outcomes, and the fitted IAC predictor is evaluated by cross-validation with self-admitted generalization limits.

full rationale

The paper's headline results (2.94x JCT reduction, 2.02x throughput, 25% to 46% SM utilization) are measured on a 64-GPU testbed and in a simulator validated against that testbed, rather than being quantities derived from the fitted models. RST profiles are obtained by direct measurement of each configuration (Section 3.1.2), and ESP solves the resulting allocation problem via Lagrangian shadow prices (Eqs. 6-8); neither step defines its output in terms of the claimed outcome. The IAC interference predictor is trained on co-location measurements of the 12 evaluation workloads and tested via 5-fold cross-validation (Section 5.5), which is standard in-distribution model evaluation rather than a fitted input being renamed as a prediction; the paper explicitly concedes in Limitation 4 that accuracy may degrade on architectures with different utilization profiles. The simulator is calibrated against testbed measurements and validated to within 7% (Section 5.1), so the 512-GPU scalability results are extrapolations from a checked model, not circular restatements. No load-bearing self-citation chain appears: references such as Pollux and SIRIUS are contextual comparisons, and no uniqueness theorem is imported from the authors' prior work. The reviewer concern that JCT comparisons may be confounded by an undefined training-completion criterion and unmodeled learning-rate scaling is a substantive correctness and validity risk, but it is not a circularity: the paper never equates its throughput objective with JCT by construction, and JCT is measured rather than predicted from T_j(c). Under the required standard of exhibiting a specific equation-level reduction or a fitted parameter renamed as a prediction, no circular step is present.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on domain assumptions about profiling representativeness, Lagrangian dual feasibility, memory isolation, predictor generalization, and simulator fidelity. The free parameters are scheduler and model hyperparameters that are chosen by hand and mostly unreported, so they are not independently grounded. No new physical or conceptual entities are introduced; RST, ESP, IAC, and PDS are system components, not postulated entities requiring separate evidence.

free parameters (7)
  • Shadow price step size eta_t = eta_0/sqrt(t), eta_0 not reported
    Controls how fast ESP prices respond to capacity shortages; affects which configurations are selected and ultimately the reported JCT and throughput gains (Eq. 8).
  • Queue reservation aggressiveness gamma
    Reserves capacity for queued jobs in Eq. 12; the value in [0,1] is chosen by hand and not reported, and it directly affects admission and migration behavior.
  • Interference penalty coefficient alpha
    Scales IAC slowdown predictions in the ESP score (Eq. 7); no value or sensitivity analysis is reported, yet it controls how aggressively the scheduler avoids interference.
  • Tolerance sensitivity beta and cap beta_max
    Adaptive tolerance threshold in Eq. 11; determines how much slowdown is accepted under load, and values are not reported.
  • Switching cost gamma_0
    Penalty for changing a job's configuration in Eq. 7; not reported in the evaluation, and it affects how often ElastiCo reconfigures jobs.
  • IAC DNN hyperparameters = 2 hidden layers, dropout 0.2; width, learning rate, and epochs not reported
    Affect interference prediction accuracy (7.6% MAPE) and hence co-location safety; training details are under-specified.
  • IAC training coverage fraction = 20-30% of candidate pairs
    Stratified sampling of co-location pairs; the chosen fraction is heuristic and affects predictor generalization.
assumptions (5)
  • domain assumption Post-warmup sampling of a few iterations represents steady-state throughput and resource demand for jobs that run for hours or days.
    Invoked in Section 3.1.2 to justify building RST profiles without profiling entire executions; if false, configuration selection uses incorrect T_j(c) and R_j(c) values.
  • domain assumption The Lagrangian dual and subgradient price updates on the integer allocation problem lead to feasible, high-quality allocations after bin-packing and penalty rounds.
    Sections 3.2.2 and 3.2.3; no optimality or convergence bound is proven for the integer program, so practical performance relies on this assumption.
  • domain assumption Co-located jobs can be safely isolated through framework memory caps and MPS limits.
    Section 4.2; the memory budget derivation assumes no allocator or driver-level interference beyond what IAC predicts.
  • domain assumption The IAC predictor generalizes from 12 training and inference workloads to the workloads in the evaluation and in production clusters.
    Sections 3.3.2 and 5.5; the predictor is trained and cross-validated on the same workload families used in the end-to-end and simulation evaluation, and the paper's own limitation 4 admits degradation on MoE or diffusion models.
  • domain assumption The discrete-event simulator calibrated within 7% on a 64-GPU trace accurately predicts performance at 128, 256, and 512 GPUs.
    Sections 5.1 and 5.6; the scalability claims depend on this extrapolation beyond the physical testbed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ElastiCo: Elastic Configuration and Interference-Aware Orchestration for GPU Clusters." pith.science (2026). https://pith.science/paper/QZUGGD3A

@misc{pith2026260807971,
  author       = {Pith},
  title        = {Pith review of: ElastiCo: Elastic Configuration and Interference-Aware Orchestration for GPU Clusters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QZUGGD3A}},
  note         = {Machine review of arXiv:2608.07971}
}
read the original abstract

Modern GPU clusters must simultaneously serve deep learning training and offline large language model inference workloads, yet existing schedulers treat these as isolated resource consumers with rigid, static allocations. This leaves substantial GPU capacity underutilized: training jobs reserve entire devices despite periodic idle phases, while offline inference tasks over-provision GPUs despite bursty demand patterns. We present ElastiCo, an elastic co-location framework that enables training and inference workloads to safely share GPUs through three integrated mechanisms. First, Resource Shape Transformation exposes each job as a family of feasible resource-performance configurations. Second, Elastic Shadow Pricing decomposes the resulting multi-resource allocation problem into per-job configuration selection subproblems via dynamic per-resource shadow prices. Third, Interference-Aware Co-location uses a predictor trained on hardware-counter and task-level features to estimate pairwise performance degradation under GPU sharing. Implemented as native Kubernetes middleware requiring no user-code modifications, ElastiCo is evaluated on a 64-GPU testbed and through large-scale trace-driven simulations (up to 512 GPUs), reducing the average JCT by up to 2.94x, increasing the cluster throughput by 2.02x, and increasing the GPU utilization from approximately 25% to 46%.

Figures

Figures reproduced from arXiv: 2608.07971 by the authors.

Figure 1
Figure 1. CDF of GPU SM utilization across two production systems. (a) Alibaba PAI Cluster [2]: training vs. inference jobs. (b) GenAI Serving System [11]: per-pod time-series samples. 10 20 30 40 Time (days) 0 20 40 60 80 100 Median GPU Utilisation (%) r = -0.16 (a) PAI Cluster: Train vs Infer Training Inference 0.0 0.2 0.4 0.6 0.8 Time (days) 0 2 4 6 Aggregate QPS CV = 0.87 (b) GenAI Serving: Request Rate [PITH_FULL_IMAGE:… view at source ↗
Figure 2
Figure 2. Temporal demand characteristics. (a) Median GPU utilization in the PAI cluster, aggregated over 6-hour intervals; (b) Total QPS of the GenAI serving infrastructure. serving system follows a comparable trend: its GPU pods achieve an average utilization of 7.0%, with 97.7% of time￾series measurements under 50% and a median that is effectively zero, pointing to long idle periods between request bursts. These results al… view at source ↗
Figure 3
Figure 3. Job states in the PAI cluster. (a) Distribution of GPU requests; (b) Intra-group configuration diversity. periods between bursts. Each such idle stretch is a period during which GPUs reserved for inference are completely unused - this is exactly the temporal slack. 2.3. Opportunity for Reconfigurability Production traces show that workloads have substantial, but largely untapped, configuration flexibility. Bursty de… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Job configurations in the PAI cluster. (a) GPU request vs usage. (b) CDF of within-group GPU flexibility ratio. This diversity offers indirect evidence of configuration flexibility: when jobs in the same group succeed with varying GPU allocations, the workload likely s…
Figure 5
Figure 5. Figure 5: The architecture of ElastiCo [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Job-level performance comparison on the 64-GPU testbed under mixed workload. 5.2. Overall Performance Figure 6a shows the CDF of normalized JCT for training jobs. ElastiCo reduces average JCT by 2.94× compared to Volcano and by roughly 1.35× compared to Lucid, while ac…
Figure 7
Figure 7. Figure 7: Cluster-wide GPU SM Active over a 24-hour trace replay. ElastiCo sustains around 40–50% utilization, compared to roughly 20–30% for Volcano [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Interference characteristics of workload co-location. single A100. The scheduler serializes them, leaving one GPU idle while the other runs. ElastiCo enables CK for BERT (reducing its footprint to 24 GB) and AMP for ResNet-50 (reducing to 14 GB). The combined 38 GB fit…
Figure 9
Figure 9. Figure 9: Heatmap of pairwise co-location interference across workload variants [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 25 canonical work pages

  1. [1]

    W.Chen,C.Lu,H.Xu,K.Ye,C.Xu, MultiplexingdynamicdeeplearningworkloadswithSLO-awarenessinGPUclusters, in:Proceedingsof theTwentiethEuropeanConferenceonComputerSystems(EuroSys’25),2025,pp.589–604.URL:https://doi.org/10.1145/3689031. 3696074. doi:10.1145/3689031.3696074

    arXiv 2025

  2. [2]

    Q.Weng,W.Xiao,Y.Yu,W.Wang,C.Wang,J.He,Y.Li,L.Zhang,W.Lin,Y.Ding, MLaaSinthewild:Workloadanalysisandscheduling in large-scale heterogeneous GPU clusters, in: 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), 2022, pp. 945–960. URL:https://www.usenix.org/conference/nsdi22/presentation/weng

    work page 2022

  3. [3]

    Y. Gao, Y. He, X. Li, B. Zhao, H. Lin, Y. Liang, J. Zhong, H. Zhang, J. Wang, Y. Zeng, K. Gui, J. Tong, M. Yang, An empirical study on low GPUutilizationofdeeplearningjobs, in:ProceedingsoftheIEEE/ACM46thInternationalConferenceonSoftwareEngineering(ICSE’24),

    work page

  4. [4]

    Q.Hu,Z.Ye,Z.Wang,G.Wang,M.Zhang,Q.Chen,P.Sun,D.Lin,X.Wang,Y.Luo,Y.Wen,T.Zhang, Characterizationoflargelanguage modeldevelopmentinthedatacenter, in:21stUSENIXSymposiumonNetworkedSystemsDesignandImplementation(NSDI24),2024,pp. 709–729. URL:https://www.usenix.org/conference/nsdi24/presentation/hu

    work page 2024

  5. [5]

    C.Lv,X.Shi,D.Liang,W.Tan,X.Zhao, SpecInF:ExploitingidleGPUresourcesindistributedDLtrainingviaspeculativeinferencefilling, in: Network and Parallel Computing: 20th IFIP WG 10.3 International Conference, NPC 2024, Proceedings, Part I, 2025, pp. 146–158. URL: https://doi.org/10.1007/978-981-96-2830-8_12. doi:10.1007/978-981-96-2830-8_12

    work page doi:10.1007/978-981-96-2830-8_12 2024

  6. [6]

    J. Wang, Y. Wang, M. Han, R. Chen, Colocating ML inference and training with fast GPU memory handover, in: 2025 USENIX Annual TechnicalConference(USENIXATC25),2025,pp.1657–1675.URL:https://www.usenix.org/conference/atc25/presentation/ wang-jiali

    work page 2025

  7. [7]

    J. Liu, Z. Cai, Y. Liu, H. Li, Z. Zhang, R. Ma, R. Buyya, SMore: Enhancing GPU utilization in deep learning clusters by serverless-based co-location scheduling, IEEE Transactions on Parallel and Distributed Systems 36 (2025) 903–917. doi:10.1109/TPDS.2025.3548320

    arXiv 2025

  8. [8]

    G. Chen, S. Subramaniyan, X. Wang, Latency-guaranteed co-location of inference and training for reducing data center expenses, in: Proc. IEEE International Conference on Distributed Computing Systems (ICDCS), 2024, pp. 473–484. doi:10.1109/ICDCS60910.2024.00051

    arXiv 2024

Show all 55 references
  1. [9]

    A. Qiao, S. K. Choe, S. J. Subramanya, W. Neiswanger, Q. Ho, H. Zhang, G. R. Ganger, E. P. Xing, Pollux: Co-adaptive cluster scheduling forgoodput-optimizeddeeplearning, in:15thUSENIXSymposiumonOperatingSystemsDesignandImplementation(OSDI21),2021,pp. 1–18. URL:https://www.usen...

    2021

  2. [10]

    Q. Hu, M. Zhang, P. Sun, Y. Wen, T. Zhang, Lucid: A non-intrusive, scalable and interpretable scheduler for deep learning training jobs, in: Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 ...

    2023

  3. [11]

    Y. Lin, S. Wu, S. Luo, H. Xu, H. Shen, C. Ma, M. Shen, L. Chen, C. Xu, L. Qu, K. Ye, Understanding diffusion model serving in production: A top-down analysis of workload, scheduling, and resource efficiency, in: Proceedings of the 2025 ACM Symposium on Cloud Computing (SoCC ’2...

    2025

  4. [12]

    arXiv:1604.06174, preprint

    T.Chen,B.Xu,C.Zhang,C.Guestrin,Trainingdeepnetswithsublinearmemorycost,2016.URL:https://arxiv.org/abs/1604.06174. arXiv:1604.06174, preprint

    2016 arXiv

  5. [13]

    Micikevicius, S

    P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, H. Wu, Mixed precision training, in: International Conference on Learning Representations, 2018. URL:https://openreview.net/forum?id= r1gs9JgRZ

    2018

  6. [14]

    Zheng, L

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, Y. Sheng, SGLang: Efficient execution of structured language model programs, in: Proceedings of the 38th International Conference on Neural Information Processing...

    2024

  7. [16]

    K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778. doi:10.1109/CVPR.2016.90. J. Wang et al.:Preprint submitted to ElsevierPage 24 of 27 ElastiCo: Elastic ...

    2016 doi

  8. [17]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Vo...

    2019 doi

  9. [18]

    A.Radford,L.Metz,S.Chintala,Unsupervisedrepresentationlearningwithdeepconvolutionalgenerativeadversarialnetworks,2016.URL: https://arxiv.org/abs/1511.06434.arXiv:1511.06434, preprint

    2016 arXiv

  10. [19]

    R. Q. Charles, H. Su, M. Kaichun, L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 77–85. doi:10.1109/CVPR.2017.16

    2017 doi

  11. [20]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, I. Polosukhin, Attention is all you need, in: I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems,...

    2017

  12. [21]

    Hochreiter, J

    S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Computation 9 (1997) 1735–1780. URL:https://doi.org/10.1162/ neco.1997.9.8.1735. doi:10.1162/neco.1997.9.8.1735

    1997 doi

  13. [22]

    X. He, L. Liao, H. Zhang, L. Nie, X. Hu, T.-S. Chua, Neural collaborative filtering, in: Proceedings of the 26th International Conference on World Wide Web, WWW ’17, International World Wide Web Conferences Steering Committee, 2017, pp. 173–182. URL:https: //doi.org/10.1145/30...

    2017

  14. [23]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms, 2017. URL:https://arxiv.org/ abs/1707.06347.arXiv:1707.06347, preprint

    2017 arXiv

  15. [24]

    DeepSeek-AI, :, X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, H. Gao, K. Gao, W. Gao, R. Ge, K. Guan, D. Guo, J. Guo, G. Hao, Z. Hao, Y. He, W. Hu, P. Huang, E. Li, G. Li, J. Li, Y. Li, Y. K. Li, W. Liang, F. Lin, A. X. Liu, B. Liu, W. Liu,...

    2024 arXiv

  16. [25]

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P.W...

    2024 arXiv

  17. [26]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, W. E. Sayed, Mistral 7b, 2023. URL:https://arxiv.org/ abs/23...

    2023 arXiv

  18. [27]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru,...

    2024 arXiv

  19. [28]

    CNCF Incubating Project

    Volcano Community, Volcano: Cloud native batch computing platform,https://volcano.sh/, 2024. CNCF Incubating Project

    2024

  20. [29]

    Y. Qiao, S. Anzai, S. Yu, H. Ma, S. Yang, Y. Wang, M. Kim, Y. Wu, Y. Zhou, J. Xing, J. E. Gonzalez, I. Stoica, H. Xu, ConServe: Fine- grainedGPUharvestingforLLMonlineandofflineco-serving,2025.URL:https://arxiv.org/abs/2410.01228.arXiv:2410.01228, preprint

    2025 arXiv

  21. [30]

    B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, W. Lin, Llumnix: Dynamic scheduling for large language model serving, in: 18th USENIXSymposiumonOperatingSystemsDesignandImplementation(OSDI24),2024,pp.173–191.URL:https://www.usenix.org/ conference/osdi24/presentation/sun-biao

    2024

  22. [31]

    Y. Li, Z. Li, Y. Zhu, C. Liu, LeMix: Unified scheduling for LLM training and inference on multi-GPU systems, 2025. URL:https: //arxiv.org/abs/2507.21276.arXiv:2507.21276, preprint

    2025 arXiv

  23. [32]

    Jayaram Subramanya, D

    S. Jayaram Subramanya, D. Arfeen, S. Lin, A. Qiao, Z. Jia, G. R. Ganger, Sia: Heterogeneity-aware, goodput-optimized ML-cluster scheduling, in: Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23), 2023, pp. 642–657. URL:https: //doi.org/10.1145/3600006...

    2023

  24. [33]

    Zheng, R

    P. Zheng, R. Pan, T. Khan, S. Venkataraman, A. Akella, Shockwave: Fair and efficient cluster scheduling for dynamic adaptation in machine learning, in: 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 2023, pp. 703–723. URL: https://www.usenix.or...

    2023

  25. [34]

    D. Gu, X. Xie, G. Huang, X. Jin, X. Liu, PowerFlow: Energy-efficient GPU clusters scheduling for deep learning, 2023. URL:https: //arxiv.org/abs/2304.06381.arXiv:2304.06381, preprint

    2023 arXiv

  26. [35]

    1298–1316

    Z.Zhu,C.Giannoula,M.Andoorveedu,Q.Su,K.Mangalam,B.Zheng,G.Pekhimenko, Mist:Efficientdistributedtrainingoflargelanguage modelsviamemory-parallelismco-optimization, in:ProceedingsoftheTwentiethEuropeanConferenceonComputerSystems(EuroSys’25), 2025, pp. 1298–1316. URL:https://doi....

    2025

  27. [36]

    W. Xiao, R. Bhardwaj, R. Ramjee, M. Sivathanu, N. Kwatra, Z. Han, P. Patel, X. Peng, H. Zhao, Q. Zhang, F. Yang, L. Zhou, Gandiva: Introspective cluster scheduling for deep learning, in: 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), 2018, pp. ...

    2018

  28. [37]

    J. Gu, M. Chowdhury, K. G. Shin, Y. Zhu, M. Jeon, J. Qian, H. Liu, C. Guo, Tiresias: A GPU cluster manager for distributed deep learning, in: 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’19), 2019, pp. 485–500. URL: https://www.usenix.org/confere...

    2019

  29. [38]

    doi:10.1057/palgrave.jors.2600523

    F.P.Kelly,A.K.Maulloo,D.K.H.Tan, Ratecontrolforcommunicationnetworks:Shadowprices,proportionalfairnessandstability, Journal of the Operational Research Society 49 (1998) 237–252. doi:10.1057/palgrave.jors.2600523

    1998 doi

  30. [39]

    Ghodsi, M

    A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, I. Stoica, Dominant resource fairness: Fair allo- cation of multiple resource types, in: 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’11), USENIX Association, 2011, pp. 323–336. URL:http...

    2011

  31. [40]

    Grandl, M

    R. Grandl, M. Chowdhury, A. Akella, G. Ananthanarayanan, Altruistic scheduling in multi-resource clusters, in: 12th USENIX Symposium onOperatingSystemsDesignandImplementation(OSDI16),2016,pp.65–80.URL:https://www.usenix.org/conference/osdi16/ technical-sessions/presentation/gr...

    2016

  32. [41]

    W. Xiao, S. Ren, Y. Li, Y. Zhang, P. Hou, Z. Li, Y. Feng, W. Lin, Y. Jia, AntMan: Dynamic scaling on GPU clusters for deep learning, in: 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), USENIX Association, 2020, pp. 533–548. URL: https://www.usen...

    2020

  33. [42]

    P. Yu, M. Chowdhury, Fine-grained GPU sharing primitives for deep learning applications, in: Proceedings of Machine Learning and Systems, volume 2, 2020, pp. 98–111. URL:https://proceedings.mlsys.org/paper_files/paper/2020/file/ d9cd83bc91b8c36a0c7c0fcca59228f2-Paper.pdf

    2020

  34. [43]

    Z.Bai,Z.Zhang,Y.Zhu,X.Jin, PipeSwitch:Fastpipelinedcontextswitchingfordeeplearningapplications, in:14thUSENIXSymposiumon Operating Systems Design and Implementation (OSDI 20), 2020, pp. 499–514. URL:https://www.usenix.org/conference/osdi20/ presentation/bai

    2020

  35. [44]

    NVIDIA,Multi-ProcessService,TechnicalReport,NVIDIACorporation,2019.URL:https://docs.nvidia.com/deploy/mps/index. html. J. Wang et al.:Preprint submitted to ElsevierPage 26 of 27 ElastiCo: Elastic Configuration and Interference-Aware Orchestration

    2019

  36. [45]

    URL:https://docs.nvidia.com/datacenter/tesla/ mig-user-guide/

    NVIDIA, Multi-Instance GPU, Technical Report, NVIDIA Corporation, 2020. URL:https://docs.nvidia.com/datacenter/tesla/ mig-user-guide/

    2020

  37. [46]

    Strati, X

    F. Strati, X. Ma, A. Klimovic, Orion: Interference-aware, fine-grained GPU sharing for ML applications, in: Proceedings of the Nineteenth European Conference on Computer Systems (EuroSys ’24), 2024, pp. 1075–1092. doi:10.1145/3627703.3629578

    2024

  38. [47]

    Z. Liu, Y. Cheng, C. Chen, J. Hu, R. Fu, D. Zhang, ISACPP: Interference-aware scheduling approach for deep learning training workloads based on co-location performance prediction, IEEE Transactions on Parallel and Distributed Systems 36 (2025) 1591–1607. doi:10.1109/ TPDS.2025.3577796

    2025

  39. [48]

    S.S.Shubha,H.Shen,A.Iyer,USHER:HolisticinterferenceavoidanceforresourceoptimizedMLinference,in:18thUSENIXSymposiumon Operating Systems Design and Implementation (OSDI 24), 2024, pp. 947–964. URL:https://www.usenix.org/conference/osdi24/ presentation/shubha

    2024

  40. [49]

    B.-S. Han, T. Paul, Z. Liu, A. Gandhi, KACE: Kernel-aware colocation for efficient GPU spatial sharing, in: Proceedings of the 2024 ACM Symposium on Cloud Computing (SoCC ’24), 2024, pp. 460–469. URL:https://doi.org/10.1145/3698038.3698555. doi:10.1145/3698038.3698555

    2024

  41. [50]

    Y.Zhong,S.Liu,J.Chen,J.Hu,Y.Zhu,X.Liu,X.Jin,H.Zhang, DistServe:Disaggregatingprefillanddecodingforgoodput-optimizedlarge language model serving, in: 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 193–210. URL:https://www.usenix.org/co...

    2024

  42. [51]

    Patel, E

    P. Patel, E. Choukse, C. Zhang, A. Shah, I. n. Goiri, S. Maleki, R. Bianchini, Splitwise: Efficient generative LLM inference using phase splitting, in: 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 118–132. doi:10.1109/ ISCA59077....

    2024

  43. [52]

    Z.Li,L.Zheng,Y.Zhong,V.Liu,Y.Sheng,X.Jin,Y.Huang,Z.Chen,H.Zhang,J.E.Gonzalez,I.Stoica, AlpaServe:Statisticalmultiplexing with model parallelism for deep learning serving, in: 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), 2023, pp. 663–679. URL...

    2023

  44. [53]

    X. Miao, C. Shi, J. Duan, X. Xi, D. Lin, B. Cui, Z. Jia, SpotServe: Serving generative large language models on preemptible instances, in: Proceedingsofthe29thACMInternationalConferenceonArchitecturalSupportforProgrammingLanguagesandOperatingSystems,Volume 2 (ASPLOS ’24), 2024...

    2024

  45. [54]

    Agrawal, N

    A. Agrawal, N. Kedia, J. Mohan, A. Panwar, N. Kwatra, B. S. Gulavani, R. Ramjee, A. Tumanov, Vidur: A large-scale simulation framework for LLM inference, in: P. Gibbons, G. Pekhimenko, C. D. Sa (Eds.), Proceedings of Machine Learn- ing and Systems, volume 6, 2024, pp. 351–366....

    2024

  46. [55]

    J.Hu,J.Xu,Z.Liu,Y.He,Y.Chen,H.Xu,J.Liu,J.Meng,B.Zhang,S.Wan,G.Dan,Z.Dong,Z.Ren,C.Liu,T.Xie,D.Lin,Q.Zhang,Y.Yu, H. Feng, X. Chen, Y. Shan, DeepServe: Serverless large language model serving at scale, in: 2025 USENIX Annual Technical Conference (USENIX ATC 25), 2025. URL:https:/...

    2025

  47. [2024]

    doi:10.1145/3597503.3639232

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.