Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LeMix claims that co-locating LLM training and inference on shared GPU nodes—guided by profiled latency predictions and SLO-aware prioritization—raises throughput up to 3.53×, cuts inference loss to 0.61×, and lifts response-time SLO…

desk verdict A coherent co-location scheduler for LLM training and inference, worth serious review, but the prefill-only prediction model and missing artifacts make the headline SLO gains fragile. read the letter →

arxiv 2507.21276 v1 pith:LJZ7BEFO submitted 2025-07-28 cs.AI cs.CLcs.DC

Yufei Li , Zexin Li , Yinglun Zhu , Cong Liu This is my paper
classification cs.AIcs.CLcs.DC
keywords LLMservingcontinuousretrainingpipelineparallelismGPUschedulingworkloadco-locationSLOattainmentmemory-awareidlenessprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that LLM inference serving and continuous retraining, normally run on separate dedicated servers, waste substantial GPU capacity and should instead be co-located on shared nodes under one scheduler. It shows that serving idles during request gaps and that pipeline-parallel training idles whenever heterogeneous query lengths create far dependencies, and that these two idle patterns can fill each other. LeMix uses offline-profiled latency coefficients to predict where idle time will open and which tasks can occupy it, then prioritizes inference tasks that risk missing response-time SLOs and defers or offloads memory-blocking work. If the scheduling predictions hold, production LLM systems could serve users and keep models aligned on the same hardware without sacrificing responsiveness. The paper's evaluation reports up to 3.53× higher throughput, 0.61× lower inference loss, and 2.12× higher SLO attainment over traditional separate setups.

What carries the argument

The central mechanism is task-specific execution planning built on the profiled stage-latency model $\Delta = \eta \cdot C \cdot \ell^2$, implemented as Algorithm 1's ComputeIdleness routine. For a new task on a node, it simulates the forward pass stage by stage, distinguishes usable from unusable idle intervals, reschedules around backward passes of pending training tasks (far dependencies), and returns the idleness increase (II) and estimated response time (R). These two quantities feed a node priority score $f = IP + \lambda_2 \cdot LC / (\lambda_1 \cdot R)$, where IP penalizes large idle-period increases relative to arrival intervals and LC rewards query lengths consistent with the node's historical workload; the scheduler also maintains a global queue-level rule that deprioritizes training tasks when a subsequent inference task would miss its SLO, and a runtime memory-aware wait-or-drop policy that offloads KV caches when memory pressure exceeds a threshold.

What would settle it

Measure actual per-stage forward and backward latencies across a range of batch sizes, query lengths, and memory-utilization levels on a node running concurrent training and inference; if the residuals from the $\eta \cdot C \cdot \ell^2$ fit grow systematically when memory utilization rises above the profiled threshold, or if predicted response times in Algorithm 1 diverge from measured completion times under contention, the scheduling basis breaks down.

Watch

Extended reading notes

Core claim

The paper's central claim is that the two dominant inefficiencies of separate LLM deployment—serving idleness under dynamic request arrivals and pipeline bubbles caused by heterogeneous training workloads—are complementary, and a task-aware scheduler can weave training forward/backward passes into serving idle periods and serving requests into training pipeline bubbles. LeMix operationalizes this by modeling each stage's execution time as $\Delta_F = \eta_F \cdot C \cdot \ell^2$ and $\Delta_B = \eta_B \cdot C \cdot \ell^2$ with hardware-fitted coefficients, then using an execution-planning algorithm to forecast, for each candidate node, the idleness increase (II) and response time (R) of an incoming task. Tasks are assigned by a priority score that balances utilization profit, length consistency with the node's workload, and predicted response time, while a queue-level rule deprioritizes training tasks that would push subsequent inference past the SLO deadline. At runtime, a memory-aware wait-or-drop policy defers or offloads KV cache and activations when memory would overflow. The paper concludes that this combination achieves its reported gains because it dynamically consolidates workloads onto fewer nodes during light demand and spreads them under heavy demand, with queue-level prioritization shown to be essential for SLO attainment.

Load-bearing premise

The offline latency model—where stage forward and backward times equal a hardware-fitted coefficient times batch size times query length squared—must stay accurate under online memory pressure and variable batching, because every scheduling decision, priority score, and SLO prediction is derived from it.

Editorial extensions

If this is right

  • Co-locating training and inference on shared nodes can raise average GPU utilization from about 23–32% to about 54% in the paper's Llama-8B setting, showing that idle serving gaps and pipeline bubbles are partially interchangeable resources.
  • Continuous local retraining on serving nodes reduces inference loss compared with periodic weight synchronization, and this quality gap widens as model size grows because inter-node checkpoint transfer becomes slower.
  • The queue-level deprioritization rule is load-bearing for responsiveness: removing it in the ablation collapses SLO attainment from roughly 95% to below 30% at 100 requests per second.
  • Memory-aware wait-or-drop scheduling is necessary for stable SLO attainment under high request rates; disabling it drops SLO attainment from 95% to 55% at 100 requests per second in the reported experiments.
  • If the latency model and scheduling predictions hold across deployments, operators can set the SLO deadline parameter $\tau_R$ to explicitly trade training progress against serving responsiveness rather than accepting a fixed partition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same idleness-filling logic could plausibly extend beyond full retraining to other predictable GPU workloads, such as LoRA adapter fine-tuning or per-tenant model updates, since the scheduler only needs per-stage latency and memory coefficients rather than model-specific internals.
  • The reported gains assume independent tasks with no inter-task communication, so applying LeMix to data-parallel training where gradients must be exchanged between nodes could reduce the consolidation benefit; extending the execution planner to model gradient-sync edges would be a testable next step.
  • Because the scheduler dynamically consolidates nodes under light demand, a natural extension is to combine LeMix with power capping or node sleep states, converting the utilization gains into energy savings that the paper does not quantify.
  • The length-consistency heuristic suggests a broader principle: routing requests to nodes whose recent workload matches the request's length distribution may improve both convergence and latency, a hypothesis that could be tested independently of the full scheduler.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LeMix, a distributed scheduler for co-locating LLM training and inference workloads on multi-GPU nodes. LeMix combines offline profiling of forward/backward latencies and memory usage, an execution-planning step that predicts each task's impact on node idleness and response time, a hierarchical node-allocation policy balancing utilization, serving quality, and SLO compliance, and a runtime memory-aware scheduler with wait-or-drop and KV-cache offloading. The evaluation uses GPT and Llama models from 400M to 70B parameters on A6000 and A100 testbeds, under synthetic Poisson arrivals and LMSYS-based real traces, and reports up to 3.53x higher throughput, 0.61x lower inference loss, and 2.12x higher SLO attainment relative to a SEPARATE baseline. The paper positions LeMix as the first system to jointly manage concurrent LLM training and inference in a distributed setting.

Significance. If the main claims held, this would be a timely contribution: the paper identifies a real inefficiency in phase-separated training/serving deployments and proposes a concrete co-location mechanism with a reasonable component structure. The strengths are the breadth of the evaluation (six model sizes, two hardware platforms, real traces), the presence of component ablations, the overhead analysis showing sub-millisecond scheduling costs, and the explicit attempt to separate execution planning from memory-aware runtime control. However, the evidence is currently not strong enough to support the headline quantitative claims: the execution model omits decoding work, the throughput metric appears to count prefill-only tasks, the serving-quality proxy is not validated against end-user quality, and the reported gains are single-run maxima without variance information. The central idea is defensible and potentially publishable, but several load-bearing points need substantive revision.

major comments (4)
  1. [§IV-A, §IV-B, Algorithm 1, §V-D] The execution-planning model treats an inference request as a single forward pass with duration Δ_F = η_F · C · ℓ² and defines response time R as completion of that forward pass (Algorithm 1, line 20). However, §V-D states that after prefill, decoding requests continue on the same node to preserve data locality. The trace queues and the Idleness-Increased computation in Algorithm 1 therefore contain no term for the ongoing decode iterations and their KV-cache memory, even though those decodes will occupy the same GPUs during the forecasted 'usable' idle intervals. This is a functional-form issue, not a coefficient-fitting issue: no fitted η_F can represent the missing decode workload. The w/o-profile ablation in §VI-F only removes the fitted coefficients and cannot detect this misspecification. Because Equations (3) and (4) feed directly into node allocation and SLO-based deprioritization, the claimed 2.12x SLO attainment and part of the 3.53x throughput gain may not persist under workloads with large output-to-prompt ratios, which are common in chat and code-generation traces. I recommend extending the planning model to include decode occupancy and re-running the evaluation on traces with realistic output lengths.
  2. [§VI-A, Table I, §IV-B] The throughput metric is defined as 'the average number of completed tasks per second,' but Table I defines an inference task as forward-only and §IV-B measures response time to forward-pass completion. If throughput counts prefills rather than full generated responses, then the reported throughput is not end-to-end serving throughput, and the 3.53x figure overstates serving capacity. If throughput counts full request completions, then the scheduler is optimizing a prefill-only model for a decode-inclusive metric, which is internally inconsistent. Please disambiguate the metric and report end-to-end token-generation throughput, as well as goodput under the SLO, rather than the current task/s numbers.
  3. [§VI-A 'Metrics' and §V-B] The serving-quality claim is based on average inference decoding loss, which is a proxy for end-user quality, and the SEPARATE baseline's loss is computed under a periodic weight-synchronization policy ('e.g., every 100 training tasks') that is never varied in the experiments. The reported 0.61x loss reduction may therefore be dominated by an arbitrarily infrequent sync interval rather than by co-location or by LeMix's length-consistency (LC) heuristic. There is also no ablation that removes LC from LeMix, so its contribution to the loss improvement is not isolated. Please report loss under multiple sync intervals, add an end-to-end quality metric such as win-rate or downstream-task accuracy, and ablate LC.
  4. [§VI-A, Figures 10-12] The synthetic workloads are generated by stochastic sampling (Poisson arrivals, dataset subsets), but the paper reports no confidence intervals, standard deviations, or number of seeds. The headline improvements are 'up to' maxima over selected operating points, and some SLO ratios at low load are small (1.05-1.15x), making single-run differences unreliable. Because the central contribution is empirical, the paper should report means or medians with variance over multiple seeds and should identify the operating points where gains are typical rather than maximal.
minor comments (5)
  1. [Throughout] The paper switches between 'NAIVE MIX', 'MIX-RR', and 'Mix-RR' for the same baseline; please unify the terminology.
  2. [§IV-A] The claim that stage latency scales quadratically with query length is supported only by two empirical curves (Figure 2) without a fitted line, goodness-of-fit, or residual information; please include the fitted model and its accuracy.
  3. [§VI-C] The description of the LMSYS trace construction says timestamps are 're-scaled to a time window' but does not specify whether output lengths and token-level timing are taken from the trace or synthesized; please clarify.
  4. [§IV-C] Equation (2) defines LC as a Gaussian density evaluated at the incoming length ℓ; as written it is not a normalized similarity score and its scale depends on σ_a, so the interpretation of 'higher LC scores' in the priority score needs a short explanation.
  5. [Abstract and Evaluation] No code or data release is mentioned. For a systems paper whose central claims are empirical, releasing the profiler, scheduler, and trace preprocessing scripts would significantly strengthen reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: end-to-end evaluation against external baselines; only a minor self-citation for the LC heuristic.

full rationale

The scheduler's predictions are produced by the offline latency model Δ = η·C·ℓ² and Algorithm 1's trace simulation, but LeMix's headline claims (throughput up to 3.53x, loss down to 0.61x, SLO attainment up to 2.12x) are measured end-to-end against SEPARATE, MIX-RR, and MIX-LUF baselines, not by checking the model's own forecasts. The 'w/o profile' ablation in §VI-F even shows that the system remains competitive when the fitted coefficients are removed, so the fitted latency model is an internal planning input rather than a self-reported prediction renamed as a result. The only self-citation with any weight is the claim in §IV-C that 'Higher LC scores align tasks with a node's workload profile, which has been shown benefiting training convergence [43, 44]'; those references share authors with this paper, and no w/o-LC ablation isolates the heuristic, but the measured loss reduction does not reduce to that citation because co-location alone drives most of the improvement. No equation is defined in terms of its own output, and no fitted parameter is reported as a predicted benchmark result. The missing decode-phase term in the prefill-only latency model is a correctness and robustness risk, not a circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 2 invented entities

The system's predicted behavior rests on fitted latency coefficients, empirical thresholds, and several simplifying assumptions about task independence and workload distributions. No code is released, so these parameters and assumptions are not independently auditable.

free parameters (7)
  • eta_F, eta_B (forward/backward latency coefficients per hardware) = not reported
    Measured offline by averaging stage runtimes across batch sizes and query lengths, normalized by C*l^2 (Section IV-A); used in all execution predictions.
  • kappa (memory safety factor) = not reported
    Chosen empirically to set memory threshold M_threshold = kappa * M_peak (Section IV-A).
  • T_max (maximum wait time before offload) = not reported
    Selected from offline profiler by varying max wait time and finding the latency degradation cutoff (Section IV-A).
  • tau (idleness tolerance threshold) = not reported
    Parameter in the idleness profit formula, Eq. (1); tuned via parameter study (Section VI-G).
  • lambda_1, lambda_2 (priority weights) = not reported
    Weights in the node priority score, Eq. (3); tuned via parameter study (Section VI-G).
  • tau_R (response SLO goal) = 5x inference latency
    Used in the deprioritization condition, Eq. (4); example set to 5x inference latency (Section IV-C).
  • T_w (batching max wait time) = 0.5x inference latency
    Default in the continuous batching algorithm, Algorithm 3 (Section V-D).
assumptions (5)
  • domain assumption Tasks are independent and have no inter-task communication.
    Stated in Section II-A system model; excludes data-parallel gradient synchronization and inter-node dependencies, limiting generality.
  • domain assumption Stage forward and backward execution time scales as coefficient times batch size times query length squared.
    Section IV-A; assumes attention quadratic complexity dominates and batch scaling is linear; fitted coefficients may not capture memory-bound or communication-bound regimes.
  • domain assumption Request arrivals follow a Poisson process for synthetic workloads.
    Section VI-A; used to generate synthetic traces; real production arrival processes may differ.
  • ad hoc to paper Decoding loss on reference pairs measures serving quality.
    Section VI-A; the paper uses average inference decoding loss as the quality metric; this proxy is not standard for user-perceived quality.
  • domain assumption Cluster-level scheduler handles less than 150 rps, with front-end load balancers for higher rates.
    Section V-C; the evaluation is limited to this cluster-level scale, so scaling claims to millions of requests depend on an unvalidated hierarchical architecture.
invented entities (2)
  • Idleness Increased (II)
    purpose: Measures the amount of previously usable idle time a new task would make unusable on a node; used in priority scoring.
    Defined in Section IV-B and Algorithm 1; internal heuristic with no external validation.
  • Length Consistency (LC)
    purpose: Gaussian score favoring task placement on nodes whose historical query-length distribution matches the incoming task.
    Defined in Eq. (2); rationale cites the authors' prior work for convergence benefits, but there is no independent test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems." pith.science (2026). https://pith.science/paper/LJZ7BEFO

@misc{pith2026250721276,
  author       = {Pith},
  title        = {Pith review of: LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LJZ7BEFO}},
  note         = {Machine review of arXiv:2507.21276}
}
read the original abstract

Modern deployment of large language models (LLMs) frequently involves both inference serving and continuous retraining to stay aligned with evolving data and user feedback. Common practices separate these workloads onto distinct servers in isolated phases, causing substantial inefficiencies (e.g., GPU idleness) and delayed adaptation to new data in distributed settings. Our empirical analysis reveals that these inefficiencies stem from dynamic request arrivals during serving and workload heterogeneity in pipeline-parallel training. To address these challenges, we propose LeMix, a system for co-locating and managing concurrent LLM serving and training workloads. LeMix integrates offline profiling, execution prediction mechanisms, and runtime scheduling to dynamically adapt resource allocation based on workload characteristics and system conditions. By understanding task-specific behaviors and co-execution interference across shared nodes, LeMix improves utilization and serving quality without compromising serving responsiveness. Our evaluation shows that LeMix improves throughput by up to 3.53x, reduces inference loss by up to 0.61x, and delivers up to 2.12x higher response time SLO attainment over traditional separate setups. To our knowledge, this is the first work to uncover and exploit the opportunities of joint LLM inference and training, paving the way for more resource-efficient deployment of LLMs in production environments.

Figures

Figures reproduced from arXiv: 2507.21276 by the authors.

Figure 1
Figure 1. GPU utilization of three SEPARATE setups, NAIVEMIX, and LEMIX when deploying Llama-8B on eight A100 GPUs under LMSYS workloads. SEPARATE (2-2) and SEPARATE (1-3) dedicate 2 (1) nodes to inference and 2 (3) nodes to training, while SEPARATE (dynamic) alternates between these configurations based on request rates. NAIVEMIX and LEMIX co-locate both workloads across all four nodes. LLM training and inference workloads i… view at source ↗
Figure 2
Figure 2. Left: The length distribution (w/ standard deviation) of two datasets and Right: the forward (F) and backward (B) latency running GPT models on a single RTX A6000 GPU. inefficiencies of partitioning regardless of node allocation. These inefficiencies stem from two key challenges: serving idleness due to dynamic request arrivals and training pipeline idleness caused by workload heterogeneity. C. Two Main Sources of I… view at source ↗
Figure 4
Figure 4. Comparison of (a) SEPARATE and (b) NAIVEMIX under Left: low request rates and Right: high request rates on a two-node cluster. For simplicity, we present one micro-batch for each mini-batch. Enhanced serving quality. NAIVEMIX continuously im￾proves inference accuracy by co-locating both workloads, which allows model instances to be updated in near real-time, benefiting serving quality on shared nodes. For example, u… view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Impact of (a) workload heterogeneity on utilization under [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: LEMIX’s components and their interactions. is needed to monitor resource usage, defer or offload memory￾blocking workloads to ensure SLO compliance. IV. DESIGN OF LEMIX From §III, we discuss that LLM systems with concurrent workloads benefit from joint training-inferen…
Figure 8
Figure 8. Figure 8: Number of allocated nodes (out of 4) for L [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: An illustration of how Bottom: LEMIX consolidates workloads from Top: NAIVEMIX into one node to optimize utilization while maintaining SLOs via deprioritization. The node priority score integrates the three objectives f = IP + λ2 · LC λ1 · R , (3) where λ1 and λ2 balan…
Figure 10
Figure 10. Figure 10: Average inference loss at various training rates. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Throughput (task/s) across various request rates. [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 13
Figure 13. Figure 13: Breakdown E2E latencies under training rates of 10% (light), [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 17
Figure 17. Figure 17: Ablation analysis of LEMIX’s components. workloads across nodes (§IV-C) to prevent resource bottle￾necks and cut TTFT latency, and leverages runtime scheduling (§IV-D) to fill idle GPU slots and use batching opportunities to minimize TBT during decoding. This adaptabi…
Figure 16
Figure 16. Figure 16: Left: prefill and Right: decode latency of inference serving under various request rates for GPT-2.5B. in real-world workloads, we simulate a range of length hetero￾geneities by sampling subsets from the two datasets. We run GPT-2.5B model on each subset under a reque…
Figure 18
Figure 18. Figure 18: Trade-offs in LEMIX’s multi-objective allocation. 55% at 100 rps, as shown in [PITH_FULL_IMAGE:figures/full_fig_p012_18.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure

    cs.DC 2026-06 conditional novelty 6.0 of 10

    Matched-random merge placement is the right control for DiLoCo scheduling claims; only calibrated overlap plus EWMA forecasting beats it in real vLLM sidecar replay (6.54%→5.09% SLO violations).

Reference graph

Works this paper leans on

74 extracted references · 49 canonical work pages · cited by 1 Pith paper

  1. [1]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  2. [2]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  3. [3]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, 2024

    work page 2024

  4. [4]

    Test- time training with self-supervision for generalization under distribution shifts,

    Y . Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt, “Test- time training with self-supervision for generalization under distribution shifts,” in International conference on machine learning. PMLR, 2020, pp. 9229–9248

    work page 2020

  5. [5]

    The surprising effectiveness of test-time training for abstract reasoning,

    E. Aky ¨urek, M. Damani, L. Qiu, H. Guo, Y . Kim, and J. Andreas, “The surprising effectiveness of test-time training for abstract reasoning,” arXiv preprint arXiv:2411.07279 , 2024

    arXiv 2024

  6. [6]

    Machine learning model training over time,

    Langton, “Machine learning model training over time,” May 2018. [Online]. Available: https://www.langton.cloud/ machine-learning-model-training-over-time/

    work page 2018

  7. [7]

    Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning

    S. Choi, S. Lee, Y . Kim, J. Park, Y . Kwon, and J. Huh, “Multi-model machine learning inference serving with gpu spatial partitioning,” arXiv preprint arXiv:2109.01611, 2021

    work page Pith review arXiv 2021

  8. [8]

    Optimized training and inference of hugging face models on azure,

    A. Vilcek, “Optimized training and inference of hugging face models on azure,” September 2022. [Online]. Available: https: //techcommunity.microsoft.com/t5/microsoft-developer-community/ optimized-training-and-inference-of-hugging-face-models-on-azure/ ba-p/3631401

    work page arXiv 2022

Show all 74 references
  1. [9]

    Train a model with amazon sagemaker,

    Amazon Web Services, Inc., “Train a model with amazon sagemaker,” https://docs.aws.amazon.com/sagemaker/latest/dg/ how-it-works-training.html, Amazon Web Services, Inc., 2024, accessed: 2024-05-12

    2024

  2. [10]

    Serving heterogeneous machine learning models on Multi-GPU servers with Spatio-Temporal sharing,

    S. Choi, S. Lee, Y . Kim, J. Park, Y . Kwon, and J. Huh, “Serving heterogeneous machine learning models on Multi-GPU servers with Spatio-Temporal sharing,” in 2022 USENIX Annual Technical Confer- ence (USENIX ATC 22) . Carlsbad, CA: USENIX Association, Jul. 2022, pp. 199–216

    2022

  3. [11]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th Symposium on Operating Systems Principles , 2023, pp. 611–626

    2023

  4. [12]

    Llumnix: Dynamic scheduling for large language model serving,

    B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y . Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implemen- tation (OSDI 24) , 2024

    2024

  5. [13]

    {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “ {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 193–210

    2024

  6. [14]

    Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve},

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, “Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve},” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , 2024, pp. 117–134

    2024

  7. [15]

    {dLoRA}: Dynamically orchestrating requests and adapters for {LoRA}{LLM} serving,

    B. Wu, R. Zhu, Z. Zhang, P. Sun, X. Liu, and X. Jin, “ {dLoRA}: Dynamically orchestrating requests and adapters for {LoRA}{LLM} serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , 2024, pp. 911–927

    2024

  8. [16]

    {ServerlessLLM}:{Low-Latency} serverless inference for large language models,

    Y . Fu, L. Xue, Y . Huang, A.-O. Brabete, D. Ustiugov, Y . Patel, and L. Mai, “ {ServerlessLLM}:{Low-Latency} serverless inference for large language models,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , 2024, pp. 135–153

    2024

  9. [17]

    Orca: A distributed serving system for Transformer-Based generative models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for Transformer-Based generative models,” in 16th USENIX Symposium on Operating Systems Design and Implemen- tation (OSDI 22) . Carlsbad, CA: USENIX Association, Jul. 2022, pp. 521–538

    2022

  10. [18]

    Ampnet: Asynchronous model-parallel training for dynamic neural networks,

    A. L. Gaunt, M. A. Johnson, M. Riechert, D. Tarlow, R. Tomioka, D. Vytiniotis, and S. Webster, “Ampnet: Asynchronous model-parallel training for dynamic neural networks,” arXiv preprint arXiv:1705.09786, 2017

    2017 arXiv

  11. [19]

    Pipedream: generalized pipeline parallelism for dnn training,

    D. Narayanan, A. Harlap, A. Phanishayee, V . Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: generalized pipeline parallelism for dnn training,” in Proceedings of the 27th ACM symposium on operating systems principles , 2019, pp. 1–15

    2019

  12. [20]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053 , 2019

    1909 arXiv

  13. [21]

    Varuna: scalable, low-cost training of massive deep learning models,

    S. Athlur, N. Saran, M. Sivathanu, R. Ramjee, and N. Kwatra, “Varuna: scalable, low-cost training of massive deep learning models,” in Pro- ceedings of the Seventeenth European Conference on Computer Systems, 2022, pp. 472–487

    2022

  14. [22]

    {EnvPipe}: Performance-preserving {DNN} training framework for saving energy,

    S. Choi, I. Koo, J. Ahn, M. Jeon, and Y . Kwon, “ {EnvPipe}: Performance-preserving {DNN} training framework for saving energy,” in 2023 USENIX Annual Technical Conference (USENIX ATC 23), 2023, pp. 851–864

    2023

  15. [23]

    {AlpaServe}: Statistical multiplexing with model parallelism for deep learning serving,

    Z. Li, L. Zheng, Y . Zhong, V . Liu, Y . Sheng, X. Jin, Y . Huang, Z. Chen, H. Zhang, J. E. Gonzalez et al., “{AlpaServe}: Statistical multiplexing with model parallelism for deep learning serving,” in 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI ...

    2023

  16. [24]

    Merak: An efficient distributed dnn training framework with automated 3d parallelism for giant foundation models,

    Z. Lai, S. Li, X. Tang, K. Ge, W. Liu, Y . Duan, L. Qiao, and D. Li, “Merak: An efficient distributed dnn training framework with automated 3d parallelism for giant foundation models,” IEEE Transactions on Parallel and Distributed Systems , vol. 34, no. 5, pp. 1466–1478, 2023

    2023

  17. [25]

    Gpipe: Efficient training of giant neu- ral networks using pipeline parallelism,

    Y . Huang, Y . Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V . Le, Y . Wuet al., “Gpipe: Efficient training of giant neu- ral networks using pipeline parallelism,” Advances in neural information processing systems, vol. 32, 2019

    2019

  18. [26]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  19. [27]

    Lmsys-chat-1m: A large-scale real-world llm conversation dataset,

    L. Zheng, W.-L. Chiang, Y . Sheng, T. Li, S. Zhuang, Z. Wu, Y . Zhuang, Z. Li, Z. Lin, E. Xing et al., “Lmsys-chat-1m: A large-scale real-world llm conversation dataset,” in The Twelfth International Conference on Learning Representations, 2023

    2023

  20. [28]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  21. [29]

    Fairness in serving large language models,

    Y . Sheng, S. Cao, D. Li, B. Zhu, Z. Li, D. Zhuo, J. E. Gonzalez, and I. Stoica, “Fairness in serving large language models,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 965–988

    2024

  22. [30]

    Large language models empowered autonomous edge ai for connected intelligence,

    Y . Shen, J. Shao, X. Zhang, Z. Lin, H. Pan, D. Li, J. Zhang, and K. B. Letaief, “Large language models empowered autonomous edge ai for connected intelligence,” IEEE Communications Magazine , 2024

    2024

  23. [31]

    Deep reinforcement learning from human preferences,

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” Advances in neural information processing systems , vol. 30, 2017

    2017

  24. [32]

    Dr genre: Reinforcement learning from decoupled llm feedback for generic text rewriting,

    Y . Li, J. Nham, G. Jawahar, L. Shu, D. Uthus, Y .-H. Sung, C. Yang, I. Rolnick, Y . Qiao, and C. Liu, “Dr genre: Reinforcement learning from decoupled llm feedback for generic text rewriting,” arXiv preprint arXiv:2503.06781, 2025

    2025 arXiv

  25. [33]

    Safe rlhf: Safe reinforcement learning from human feedback,

    J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y . Wang, and Y . Yang, “Safe rlhf: Safe reinforcement learning from human feedback,” in The Twelfth International Conference on Learning Representations , 2023

    2023

  26. [34]

    Safety alignment in nlp tasks: Weakly aligned summarization as an in-context attack,

    Y . Fu, Y . Li, W. Xiao, C. Liu, and Y . Dong, “Safety alignment in nlp tasks: Weakly aligned summarization as an in-context attack,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2024, pp. 8483– 8502

    2024

  27. [35]

    Beyond data and model parallelism for deep neural networks

    Z. Jia, M. Zaharia, and A. Aiken, “Beyond data and model parallelism for deep neural networks.” Proceedings of Machine Learning and Systems, vol. 1, pp. 1–13, 2019

    2019

  28. [36]

    How many servers are needed to run chatgpt?

    GPT AI Team, “How many servers are needed to run chatgpt?” Aug. 2024, accessed: 2024-12-05. [Online]. Available: https://gptai.tn/ how-many-servers-to-run-chatgpt/

    2024

  29. [37]

    Training a helpful and harmless assistant with reinforcement learning from human feedback,

    Y . Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan et al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862 , 2022

    2022 arXiv

  30. [38]

    Understanding dataset difficulty with V-usable information,

    K. Ethayarajh, Y . Choi, and S. Swayamdipta, “Understanding dataset difficulty with V-usable information,” in Proceedings of the 39th Inter- national Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 17–23 Jul 2022, pp. 5988–6008

    2022

  31. [39]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  32. [40]

    Transparent {GPU} sharing in container clouds for deep learning workloads,

    B. Wu, Z. Zhang, Z. Bai, X. Liu, and X. Jin, “Transparent {GPU} sharing in container clouds for deep learning workloads,” in 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 2023, pp. 69–85

    2023

  33. [41]

    Efficiently scaling transformer inference,

    R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” Proceedings of Machine Learning and Systems , vol. 5, pp. 606–624, 2023

    2023

  34. [42]

    Methods and infrastructure in the era of accelerator-centric architectures,

    B. Reagen, Y . S. Shao, S. L. Xi, G.-Y . Wei, and D. Brooks, “Methods and infrastructure in the era of accelerator-centric architectures,” in 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS). IEEE, 2017, pp. 902–905

    2017

  35. [43]

    Rt-lm: Uncertainty-aware resource management for real-time inference of language models,

    Y . Li, Z. Li, W. Yang, and C. Liu, “Rt-lm: Uncertainty-aware resource management for real-time inference of language models,” in 2023 IEEE Real-Time Systems Symposium (RTSS) . IEEE, 2023, pp. 158–171

    2023

  36. [44]

    Mixtraining: A better trade-off between compute and performance,

    Z. Li, J. Zhang, Y . Li, Y . Zhu, and C. Liu, “Mixtraining: A better trade-off between compute and performance,” arXiv preprint arXiv:2502.19513, 2025

    2025 arXiv

  37. [45]

    Sglang: Efficient execution of structured language model programs,

    L. Zheng, L. Yin, Z. Xie, C. L. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalezet al., “Sglang: Efficient execution of structured language model programs,” Advances in neural information processing systems, vol. 37, pp. 62 557–62 583, 2024

    2024

  38. [46]

    {Check-N-Run}: A checkpointing system for training deep learning recommendation mod- els,

    A. Eisenman, K. K. Matam, S. Ingram, D. Mudigere, R. Krishnamoorthi, K. Nair, M. Smelyanskiy, and M. Annavaram, “ {Check-N-Run}: A checkpointing system for training deep learning recommendation mod- els,” in 19th USENIX Symposium on Networked Systems Design and Implementation ...

    2022

  39. [47]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,

    R. Y . Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley et al., “Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,” in SC22: International Conference for High Performance Com- ...

    2022

  40. [48]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” Advances in neural information processing systems , vol. 35, pp. 16 344–16 359, 2022

    2022

  41. [49]

    Dialogpt: Large-scale generative pre- training for conversational response generation,

    Y . Zhang, S. Sun, M. Galley, Y .-C. Chen, C. Brockett, X. Gao, J. Gao, J. Liu, and W. B. Dolan, “Dialogpt: Large-scale generative pre- training for conversational response generation,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics:...

    2020

  42. [50]

    Utilitiy accrual scheduling with real-time java,

    S. Feizabadi, W. Beebee Jr, B. Ravindran, P. Li, and M. Rinard, “Utilitiy accrual scheduling with real-time java,” in OTM Confederated Inter- national Conferences” On the Move to Meaningful Internet Systems” . Springer, 2003, pp. 550–563

    2003

  43. [51]

    Horovod: fast and easy distributed deep learning in tensorflow,

    A. Sergeev and M. Del Balso, “Horovod: fast and easy distributed deep learning in tensorflow,” arXiv preprint arXiv:1802.05799 , 2018

    2018 arXiv

  44. [52]

    A unified architecture for accelerating distributed {DNN} training in heteroge- neous {GPU/CPU} clusters,

    Y . Jiang, Y . Zhu, C. Lan, B. Yi, Y . Cui, and C. Guo, “A unified architecture for accelerating distributed {DNN} training in heteroge- neous {GPU/CPU} clusters,” in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) , 2020, pp. 463–479

    2020

  45. [53]

    Large scale distributed deep networks,

    J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ran- zato, A. Senior, P. Tucker, K. Yang et al., “Large scale distributed deep networks,” Advances in neural information processing systems , vol. 25, 2012

    2012

  46. [54]

    Chimera: efficiently training large-scale neural net- works with bidirectional pipelines,

    S. Li and T. Hoefler, “Chimera: efficiently training large-scale neural net- works with bidirectional pipelines,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2021, pp. 1–14

    2021

  47. [55]

    Pipefisher: Efficient training of large language models using pipelining and fisher information matrices,

    K. Osawa, S. Li, and T. Hoefler, “Pipefisher: Efficient training of large language models using pipelining and fisher information matrices,” Proceedings of Machine Learning and Systems , vol. 5, 2023

    2023

  48. [56]

    Torch- serve: Serve, optimize and scale pytorch models in production,

    L. Ning, H. Shojanazeri, K. Wen, and the PyTorch Foundation, “Torch- serve: Serve, optimize and scale pytorch models in production,” PyTorch Foundation, 2023, https://pytorch.org/serve/

    2023

  49. [57]

    Triton inference server: An optimized cloud and edge inferencing solution,

    NVIDIA Corporation, “Triton inference server: An optimized cloud and edge inferencing solution,” 2019, https://developer.nvidia.com/ nvidia-triton-inference-server

    2019

  50. [58]

    White-box multi-objective adversarial attack on dialogue generation,

    Y . Li, Z. Li, Y . Gao, and C. Liu, “White-box multi-objective adversarial attack on dialogue generation,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 1778–1792

    2023

  51. [59]

    Dycl: Dynamic neural network compilation via program rewriting and graph optimization,

    S. Chen, S. Wei, C. Liu, and W. Yang, “Dycl: Dynamic neural network compilation via program rewriting and graph optimization,” in Proceed- ings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2023, pp. 614–626

    2023

  52. [60]

    Learning to reverse dnns from ai programs automatically,

    S. Chen, H. Khanpour, C. Liu, and W. Yang, “Learning to reverse dnns from ai programs automatically,” arXiv preprint arXiv:2205.10364, 2022

    2022 arXiv

  53. [61]

    Efficient large-scale language model training on gpu clusters using megatron-lm,

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V . Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro et al. , “Efficient large-scale language model training on gpu clusters using megatron-lm,” in Proceedings of the International Conference fo...

    2021

  54. [62]

    Integrated optimization of large language models: Synergizing data utilization and compression techniques,

    X. Li, Y . Ma, Y . Huang, X. Wang, Y . Lin, and C. Zhang, “Integrated optimization of large language models: Synergizing data utilization and compression techniques,” 2024

    2024

  55. [63]

    Fast distributed inference serving for large language models,

    B. Wu, Y . Zhong, Z. Zhang, S. Liu, F. Liu, Y . Sun, G. Huang, X. Liu, and X. Jin, “Fast distributed inference serving for large language models,” arXiv preprint arXiv:2305.05920 , 2023

    2023 arXiv

  56. [64]

    Splitwise: Efficient generative llm inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, ´I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 118–132

    2024

  57. [65]

    D´ej`avu: KV-cache streaming for fast, fault-tolerant generative LLM serving,

    F. Strati, S. Mcallister, A. Phanishayee, J. Tarnawski, and A. Klimovic, “D´ej`avu: KV-cache streaming for fast, fault-tolerant generative LLM serving,” in Proceedings of the 41st International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, R. ...

    2024

  58. [66]

    Estimating predictive uncertainty under program data distribution shift,

    Y . Li, S. Chen, and W. Yang, “Estimating predictive uncertainty under program data distribution shift,” arXiv preprint arXiv:2107.10989, 2021

    2021 arXiv

  59. [67]

    Uncertainty awareness of large language models under code distribution shifts: A benchmark study,

    Y . Li, S. Chen, Y . Guo, W. Yang, Y . Dong, and C. Liu, “Uncertainty awareness of large language models under code distribution shifts: A benchmark study,” arXiv preprint arXiv:2402.05939 , 2024

    2024 arXiv

  60. [68]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in NeurIPS Deep Learning and Representation Learning Workshop, 2015

    2015

  61. [69]

    An empirical investigation of catastrophic forgetting in gradient-based neural networks,

    I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y . Bengio, “An empirical investigation of catastrophic forgetting in gradient-based neural networks,” arXiv preprint arXiv:1312.6211 , 2013

    2013 arXiv

  62. [70]

    Uncertainty-aware bootstrap learning for joint extraction on distantly-supervised data,

    Y . Li, X. Yu, Y . Liu, H. Chen, and C. Liu, “Uncertainty-aware bootstrap learning for joint extraction on distantly-supervised data,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , 2023, pp. 1349–1358

    2023

  63. [71]

    Distantly- supervised joint extraction with noise-robust learning,

    Y . Li, X. Yu, Y . Guo, Y . Liu, H. Chen, and C. Liu, “Distantly- supervised joint extraction with noise-robust learning,” inFindings of the Association for Computational Linguistics ACL 2024 , 2024, pp. 10 202– 10 217

    2024

  64. [72]

    Ekya: Continuous learning of video analytics models on edge compute servers,

    R. Bhardwaj, Z. Xia, G. Ananthanarayanan, J. Jiang, Y . Shu, N. Kar- ianakis, K. Hsieh, P. Bahl, and I. Stoica, “Ekya: Continuous learning of video analytics models on edge compute servers,” in 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), 202...

    2022

  65. [73]

    Adainf: Data drift adaptive scheduling for accurate and slo-guaranteed multiple-model inference serving at edge servers,

    S. S. Shubha and H. Shen, “Adainf: Data drift adaptive scheduling for accurate and slo-guaranteed multiple-model inference serving at edge servers,” in Proceedings of the ACM SIGCOMM 2023 Conference, 2023, pp. 473–485

    2023

  66. [74]

    Lyra: Elastic scheduling for deep learning clusters,

    J. Li, H. Xu, Y . Zhu, Z. Liu, C. Guo, and C. Wang, “Lyra: Elastic scheduling for deep learning clusters,” in Proceedings of the Eighteenth European Conference on Computer Systems , 2023, pp. 835–850

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.