SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services

Sheng Zhou; Yaodan Xu; Zhisheng Niu

REVIEW 3 major objections 4 minor 50 references

SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that dynamic batching for online services with size-dependent service times can be formulated as an infinite-state average-cost semi-Markov decision process and solved to near optimality by a finite-state approximation.

desk verdict The SMDP formulation is worth engaging, but the paper's 'optimally solve' claim overreaches: the finite-state approximation gap is unproven and the abstract-cost speedup is tuned on one experiment. read the letter →

arxiv 2501.02181 v1 pith:2SPFINA5 submitted 2025-01-04 cs.DC cs.LGcs.SYeess.SY

Yaodan Xu , Sheng Zhou , Zhisheng Niu This is my paper · ORCID

classification cs.DCcs.LGcs.SYeess.SY MSC 90C4060K25

keywords dynamicbatchingsemi-Markovdecisionprocessbatchservicequeuesize-dependenttimeslatency-energytradeoffGPUinferenceservingaverage-costMDPrelativevalueiteration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles a practical tradeoff: servers with parallel processors work more efficiently on larger batches, but waiting to form a batch delays online requests. The authors claim that choosing batch sizes over time can be cast as a semi-Markov decision process (SMDP), a continuous-time sequential decision problem where the state is the number of waiting requests and the action is the batch size to serve. Their objective is a weighted sum of average response time and average power consumption. Because the exact problem has infinitely many states, unbounded costs, and an average-cost criterion, they solve a truncated version with an added overflow cost and show numerically that the resulting policies beat static and greedy batching across loads, while forming a latency-power Pareto frontier. A sympathetic reader would take the central claim to be that dynamic batching can be solved rigorously, not just tuned by heuristics.

What carries the argument

The load-bearing machinery is the semi-Markov decision process model with state $s$ equal to the number of requests in the system, action $a \in \{0\} \cup \mathcal{B}$ equal to either waiting or serving a batch of size $a$, sojourn-time distributions $\Gamma_{s,a}(\cdot)$, and expected cost until the next decision epoch $c(s,a) = w_2\zeta(a) + w_1\big(s/(\lambda\mu[a]) + \mathbb{E}[G_a^2]/2\big)$. The paper converts the intractable infinite-state, average-cost, unbounded-cost problem into a finite computation through three linked components: tail-state aggregation, where all states above $s_{\max}$ collapse into one overflow state $S_o$ carrying an abstract cost $c_o y(s,a)$ that estimates the extra holding cost of the tail; a discretization that turns the finite SMDP into an equivalent discrete-time MDP; and relative value iteration, a span-contraction algorithm that produces an $\epsilon$-optimal stationary policy for the finite model.

What would settle it

Run the SMDP-derived policy on real hardware with a different workload, such as bursty traffic traces or a different inference model, while logging actual response time and power, and compare each measured (latency, power) pair against static and greedy batching under the same load; if any SMDP pair lies above and to the right of a benchmark's pair, or if the measured weighted cost is higher than a simple threshold policy on the actually fitted $l(b)$ and $\zeta(b)$, the central claim fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the dynamic batching problem for a single parallel server with size-dependent batch service times and energy consumption admits an approximately optimal stationary policy obtained from an SMDP formulation. Requests arrive as a Poisson process, the state at each decision epoch is the number of requests in the system, and the action is either to wait or to serve a batch of size b; service times may follow any distribution with finite second moment. The paper proves existence of an optimal stationary deterministic policy, derives the average-cost optimality equations, and gives a three-step numerical procedure: truncate the infinite state space with an aggregated overflow state carrying an abstract cost, transform the finite SMDP into an equivalent discrete-time MDP by a discretization, and run relative value iteration. In special cases where the service time is independent of batch size and energy is linear, the optimal policy is shown to be a control-limit Q-policy, matching a known closed-form threshold. In the general size-dependent case, the computed policies agree with these special-case thresholds where applicable but can take more complex forms; numerical experiments report that the SMDP policies achieve the lowest weighted average cost among greedy, static, and maximum batching benchmarks and dominate them in the latency-power plane.

Load-bearing premise

The entire optimality claim hangs on the fitted affine latency and energy curves ($l(b)=0.3051b+1.0524$ ms and $\zeta(b)=19.899b+19.603$ mJ) being a faithful picture of the real server, with Poisson arrivals and well-profiled service times; if real traffic, interference, or hardware behavior deviates from those curves, the computed policy is only as good as that model.

Editorial extensions

If this is right

If the SMDP-derived policy is correct, an inference server can precompute a batching policy offline and apply it online with no extra scheduling module, selecting the weight $w_2$ to hit latency or power targets.
Under the model, for any fixed average response time the SMDP policy consumes no more power than greedy, static $b=8$, static $b=16$, or maximum batching, and the reverse holds for fixed power.
For size-independent service times and linear energy with $B_{\min}=1$, the optimal policy reduces to a single threshold $Q$ that can be computed in closed form, so lightweight deployments can use the threshold directly.
The same weighted-cost objective can replace energy with monetary cost, so the framework transfers to cost-aware batch serving.
Tail latency improves as well: numerical CDFs show SMDP solutions have lighter tails than static $b=8$ at the 90th and 95th percentiles, which matters for service-level-objective compliance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper does not develop is phase-switching traffic: the authors note that Markov-modulated Poisson arrivals require adding phase to the state, and one could test whether the abstract-cost truncation still controls error under such traffic.
The affine fits for latency and energy imply a constant per-request marginal cost; if real hardware shows superlinear latency or energy at large batches, the optimal policy would likely shift toward smaller batches than the paper's charts suggest.
The reported complexity reductions of 63.5% in space and 98% in time depend on choosing the abstract cost coefficient near 100, so a practical deployment would need to re-tune this coefficient for each hardware profile.
The observed tail-latency improvement suggests that optimizing average cost can also improve percentile-based service objectives, but that connection is empirical in this paper rather than a proven bound.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things worth knowing before you read it. The SMDP formulation is the real deal: batch service queue with size-dependent service times, average-cost objective trading latency against energy, and the size-dependent case is genuinely open in the literature they cite. But the headline claim — "first to rigorously formulate and optimally solve" — overshoots. What is proven is optimality for a finite-state surrogate; the gap back to the original infinite-state problem is closed by an acceptance heuristic and an empirically tuned constant, not by a bound.

Credit where it is due. The model is clean and the derivations check out: transition probabilities via Laplace transforms, the cost-rate decomposition, and the unichain argument are standard and correctly handled. The special-case theory (Q-policy structure) is properly attributed to Deb-Serfozo and Aalto, and the computed policies reproduce those known results where they should. The inputs are external NVIDIA profiling data with excellent fits, and the paper tests robustness to service-time variability (CoV 0 to 2). The percentile/CDF analysis is a genuine bonus. Most importantly, the paper is honest: Section VII-D openly shows that c_o = 0 and 10 produce degenerate always-wait policies with tiny Delta-pi, which is an admission that the acceptance criterion is not a certificate of proximity to the true optimum.

Soft spots, in proportion. The main one is the just-mentioned gap: nothing bounds the difference between the policy computed on the truncated chain and the true optimal average cost of model P. Since the paper discloses this internally, the overreach lives in the abstract and contribution list, which promise more than the body proves. Second, c_o is chosen from a single experiment (rho = 0.9, weights [1,1]); the advertised 63.5%/98% complexity reduction is a property of that tuning, not of the method. A sweep across rho and w2 is needed before anyone should treat the speedup as reliable. Third, "never consume more energy than any other benchmark" in contribution 4 overreaches simulated points; "in the evaluated settings" would be accurate. Fourth, idle server power is not in the objective; because the idle fraction is policy-dependent, that biases the tradeoff, though weight tuning absorbs some of it. Minor: no code or data shipped, so the numerical results are not independently reproducible without reimplementation.

This is a paper for the inference-serving and queueing-control community. It deserves a serious referee: the formulation is worth publishing, but the claims need softening and a c_o sensitivity study added. Send it to review, expect heavy revision.

Referee Report

3 major / 4 minor

Summary. The paper studies dynamic batching for a single-server queue with batch-size-dependent service times and energy consumption, motivated by ML inference serving on GPUs. It formulates the batch-size decision problem as an average-cost semi-Markov decision process (SMDP) with an infinite state space, unbounded costs, and a weighted objective of mean response time and mean power consumption. The proposed solution procedure truncates the state space with an overflow state, adds an abstract cost coefficient c_o, transforms the finite-state SMDP into a discrete-time MDP, and applies relative value iteration. Numerical experiments compare the computed policies against greedy and static batching, and special-case theoretical results (control-limit optimality) are used as validation.

Significance. If the finite-state approximation were proven to converge to the original infinite-state SMDP optimum, the paper would be a solid contribution to the queueing and ML-serving literature: the SMDP formulation is clean, the cost derivations are detailed, and the numerical study is extensive, including latency tail percentiles and multiple service-time distributions. The special-case validation against known control-limit results is a useful sanity check. The paper also ships a complete derivation of the Bellman equations and a self-contained solution algorithm. However, the main advertised claim of 'rigorously formulate and optimally solve' is not supported by any error bound between the truncated-model policy and the original SMDP optimum; the abstract cost c_o is a hand-tuned hyperparameter, and the reported complexity reduction depends on its empirical selection. The significance is therefore conditional on treating the method as a practical heuristic rather than a proven optimal solver.

major comments (3)

[Section V-A, Eq. (19), Eq. (22), Eq. (30), Section VII-D] The paper does not establish any bound on the difference between the average cost of the policy pi_epsilon defined in Eq. (30) on the original infinite-state SMDP P and the optimal cost g*. The acceptance criterion Delta_pi < delta (Eq. 22) only checks the stationary cost mass of the aggregated overflow state in the truncated chain, and Section VII-D itself shows that this criterion is unreliable: with c_o=10 and c_o=0, the computed policy degenerates to 'always wait' for moderate smax while Delta_pi is reported as roughly 10^-14, and Table II lists iterations=10000 for these cases, which equals the imposed maximum iteration cap, so the RVI may not have even satisfied the epsilon termination criterion. The abstract cost coefficient c_o in Eq. (19) is a free parameter selected by empirical search (Fig. 10 and Table II), and the advertised 63.5%/98% complexity reduction is the difference between smax=192 at c_o=0 and smax=70 at c_o=100. Without a proof that this surrogate objective preserves near-optimality for the original problem, the claim in Section I of 'rigorously formulate and optimally solve' is not supported, and the comparisons in Section VII-B are comparisons of a policy derived from a surrogate, not necessarily a solution of the original SMDP. Please either provide a convergence guarantee (e.g., conditions under which the cost of pi_epsilon tends to g* as smax grows, with c_o chosen in a specified way) or substantially moderate the optimality claims and present the method as a heuristic approximation validated numerically.
[Section III, Eq. (3), Section VII-B] The 'average power consumption' P in the objective excludes idle server power; it is defined as the long-run average batch-processing energy per unit time (the sum of zeta(b(t_i)) over service rounds divided by T). The text in Section III calls this 'the power consumption of the server', but a server's total power typically includes a substantial idle component, especially at low load where the server is idle for long periods. This omission may shift the latency-energy tradeoff and the reported Pareto comparisons. Please define the metric precisely (e.g., 'active batch-processing power' rather than 'server power') and discuss how including idle power would affect the optimal policy and the comparison with the benchmark policies.
[Table II, Section VII-D] In Table II, the rows for c_o=10 and c_o=0 report exactly 10000 iterations, which is the maximum iteration count set in Section VII-D. This means the RVI may not have terminated by the span criterion, so the resulting policies are not certified as epsilon-optimal even for the finite-state MDP. Using these rows to support the comparison of complexity and accuracy is problematic: the comparison mixes a converged run (c_o=100, 1483 iterations) with potentially non-converged runs. The authors should state explicitly whether the iteration limit was reached for these cases and, if so, re-run with a larger cap or a looser epsilon, or else exclude these cases from the efficiency comparison.

minor comments (4)

[Appendix A] There is a typo near the end of the proof: 'exisits' should be 'exists'.
[Section II and Figure 2] The vendor name 'NIVIDIA' appears in the text and in the capitation of Figure 2; it should be 'NVIDIA'.
[Section V-A and Section VII-D] The abstract cost c_o is introduced without any guideline for choosing it in general settings; the paper selects c_o=100 based on a single experiment. It would be helpful to add a short sensitivity analysis of the actual simulated average cost (not just the truncated-model cost) as c_o varies, to show that the final policy's performance is robust to this choice.
[General] The notation Delta_pi and hat_g_pi is used in Section VII-D with a subscript 'pi' that is sometimes dropped in the text (e.g., 'hat_g_pi' vs 'hat_g'); please standardize the notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SMDP formulation and RVI solution chain are self-contained; the finite-state approximation gap is an unproven step, not a circular reduction.

full rationale

The paper's derivation chain is P (infinite-state SMDP) -> finite-state aggregation hat P -> discrete-time MDP tilde P -> relative value iteration. Each transformation is either explicitly constructed in the paper or cited to an external textbook (Puterman), not to the paper's own conclusions. The fitted functions l(b) and zeta(b) are empirical inputs to the model, not outputs of the solver; no equation redefines the cost in terms of the policy it produces. The abstract cost c_o y(s,a) in Eq. (19) is a hand-chosen term added only to the overflow state, and c_o is selected by numerical search in Section VII-D, but the paper's optimality claims are for the truncated model with that cost, and the benchmark comparisons evaluate the same objective; this is hyperparameter selection, not a fitted prediction. The paper itself flags the approximation gap: 'When considering the performance of pi_epsilon in the original infinite state SMDP problem (as presented in Section IV), it is closely tied to the impact of finite state approximation,' and no bound on that gap is shown. That is a correctness or approximation risk, not circularity. The self-citation [1] in Section II ('In our prior work [1], we proposed an SMDP-based dynamic batching scheme...') is descriptive and not load-bearing. No equation reduces to its own input, and no prediction is constructed from a fit of the quantity it predicts.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard queueing assumptions (Poisson arrivals, infinite buffer, stability) and on the empirical fitted latency and energy functions. The abstract cost co is a numerical hyperparameter, not a physical entity, and is selected by search.

free parameters (3)

l(b) (mean batch service time function) = 0.3051*b + 1.0524 ms (TESLA P4 GoogLeNet)
Fitted to NVIDIA profiling data; the SMDP model and optimal policy depend on this function.
ζ(b) (batch energy consumption function) = 19.899*b + 19.603 mJ (TESLA P4 GoogLeNet)
Fitted to NVIDIA profiling data; defines the energy cost in the objective.
co (abstract cost coefficient) = 100 (selected in Section VII-D)
Chosen by empirical search over {0,10,100,1000,10000} to minimize required smax and RVI iterations. The claimed 63.5% space and 98% time complexity reductions are based on this selected value.

assumptions (5)

domain assumption Request arrivals follow a Poisson process with rate λ.
Used to derive transition probabilities p[b]_k in Eq. (4) and to define decision epochs. Real traffic may be burstier; the paper notes MMPP as a possible extension but does not solve it.
domain assumption Batch service time distributions G_b have finite second moment, mean l(b), and l(b) and θ(b)=b/l(b) are non-decreasing in b.
Justifies the model and the assumption that larger batches improve throughput. Real parallel hardware may show non-monotonic behavior due to memory limits.
domain assumption Energy consumption function ζ(b) yields non-decreasing energy efficiency η(b)=b/ζ(b).
Used in the objective; large batches are assumed to be more energy-efficient, which is supported by the NVIDIA data but may not hold universally.
domain assumption Buffer capacity is infinite.
The queueing model requires an unbounded state space; real servers have finite memory, and overflow loss is not modeled.
domain assumption Stability condition λ < Bmax*μ[Bmax] (i.e., ρ<1) holds.
Necessary for the existence of a stabilizing policy and for finite average costs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services." pith.science (2026). https://pith.science/paper/2SPFINA5

@misc{pith2026250102181,
  author       = {Pith},
  title        = {Pith review of: SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2SPFINA5}},
  note         = {Machine review of arXiv:2501.02181}
}

read the original abstract

For servers incorporating parallel computing resources, batching is a pivotal technique for providing efficient and economical services at scale. Parallel computing resources exhibit heightened computational and energy efficiency when operating with larger batch sizes. However, in the realm of online services, the adoption of a larger batch size may lead to longer response times. This paper aims to provide a dynamic batching scheme that delicately balances latency and efficiency. The system is modeled as a batch service queue with size-dependent service times. Then, the design of dynamic batching is formulated as a semi-Markov decision process (SMDP) problem, with the objective of minimizing the weighted sum of average response time and average power consumption. A method is proposed to derive an approximate optimal SMDP solution, representing the chosen dynamic batching policy. By introducing an abstract cost to reflect the impact of "tail" states, the space complexity and the time complexity of the procedure can decrease by 63.5% and 98%, respectively. Numerical results showcase the superiority of SMDP-based batching policies across various parameter setups. Additionally, the proposed scheme exhibits noteworthy flexibility in balancing power consumption and latency.

Figures

Figures reproduced from arXiv: 2501.02181 by the authors.

**Figure 2.** Inference latency and energy consumption for batch processing [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** The converged SMDP solutions under various parameter settings. The maximum batch size is chosen as [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Comparison of different policies on the average cost per unit time [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

**Figure 5.** Figure 5: The latency-energy tradeoff curves for different policies under various load conditions. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 6.** Figure 6: The latency distribution and percentile analysis under different policies, with [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

**Figure 7.** Figure 7: Illustration and comparison of results with batch-size independent service time. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: Illustration and comparison of results with a logarithmic energy consumption function. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

**Figure 9.** Figure 9: Comparisons of CDFs and latency-power consumption pairs under different service time distributions. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

**Figure 10.** Figure 10: The evolution of gˆ π (the average cost per unit time) and ∆π (the average cost contributed by So per unit time) regarding smax under different co, where smax and co are the parameters in finite state approximation. time distribution becomes heavier as the CoV increas…

**Figure 11.** Figure 11: The converged SMDP solutions under cases with characteristics such as a minimum batch size greater than 1, a nonlinear energy consumption [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 49 canonical work pages

[1]

SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms,

Y . Xu, J. Sun, S. Zhou, and Z. Niu, “SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms,” in Proc. IEEE Int. Conf. Commun. (ICC) , Rome, Italy, May 2023

work page 2023
[2]

GPU Implementation of Neural Networks,

K.-S. Oh and K. Jung, “GPU Implementation of Neural Networks,” Pattern Recognition, vol. 37, no. 6, pp. 1311–1314, Jun. 2004

work page 2004
[3]

MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving,

C. Zhang, M. Yu, W. Wang, and F. Yan, “MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving,” in Proc. USENIX Annu. Tech. Conf. (ATC), Renton, W A, USA, Jul. 2019

work page 2019
[4]

Clipper: A Low-Latency Online Prediction Serving System,

D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, “Clipper: A Low-Latency Online Prediction Serving System,” in Proc. USENIX Symp. Netw. Syst. Des. Implement. (NSDI) , Boston, MA, USA, Mar. 2017

work page 2017
[5]

McDRAM: Low Latency and Energy-Efficient Matrix Computations in DRAM,

H. Shin, D. Kim, E. Park, S. Park, Y . Park, and S. Yoo, “McDRAM: Low Latency and Energy-Efficient Matrix Computations in DRAM,” IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. , vol. 37, no. 11, pp. 2613–2622, Nov. 2018

work page 2018
[6]

Benchmarking TPU, GPU, and CPU Platforms for Deep Learning,

Y . E. Wang, G.-Y . Wei, and D. Brooks, “Benchmarking TPU, GPU, and CPU Platforms for Deep Learning,” [Online]. Available: https://arxiv. org/abs/1907.10701, 2019

arXiv 1907
[7]

NVIDIA AI inference platform technical overview,

NVIDIA, “NVIDIA AI inference platform technical overview,” [On- line]. Available: https://www.nvidia.com/en-us/data-center/resources/ inference-technical-overview/, 2018, (accessed 23-Nov-2019)

work page 2018
[8]

EAIS: Energy-Aware Adaptive Scheduling for CNN Inference on High-Performance GPUs,

C. Yao, W. Liu, W. Tang, and S. Hu, “EAIS: Energy-Aware Adaptive Scheduling for CNN Inference on High-Performance GPUs,” Future Gener. Comp. Syst., vol. 130, pp. 253–268, May 2022

work page 2022

Show all 50 references

[9]

Energy-Efficient Online Scheduling of Transformer Inference Services on GPU Servers,

Y . Wang, Q. Wang, and X. Chu, “Energy-Efficient Online Scheduling of Transformer Inference Services on GPU Servers,” IEEE Trans. Green Commun. Netw., vol. 6, no. 3, pp. 1649–1659, Sep. 2022

2022
[10]

Packrat: Automatic Reconfiguration for Latency Minimization in CPU- Based DNN Serving,

A. Bhardwaj, A. Phanishayee, D. Narayanan, M. Tarta, and R. Stutsman, “Packrat: Automatic Reconfiguration for Latency Minimization in CPU- Based DNN Serving,” [Online]. Available: http://arxiv.org/abs/2311. 18174, 2023

2023
[11]

Coordinated Batching and DVFS for DNN Inference on GPU Accelerators,

S. M. Nabavinejad, S. Reda, and M. Ebrahimi, “Coordinated Batching and DVFS for DNN Inference on GPU Accelerators,” IEEE Trans. Parallel Distrib. Syst., vol. 33, no. 10, pp. 2496–2508, Oct. 2022

2022
[12]

BATCH: Machine Learning Inference Serving on Serverless Platforms with Adaptive Batching,

A. Ali, R. Pinciroli, F. Yan, and E. Smirni, “BATCH: Machine Learning Inference Serving on Serverless Platforms with Adaptive Batching,” in Proc. Int. Conf. High Perform. Comput. Netw. Storage Anal. (SC) , Atlanta, GA, USA, Nov. 2020

2020
[13]

CoFB: Latency-Constrained Co- Scheduling of Flows and Batches for Deep Learning Inference Service on the CPU–GPU System,

Q. Zhang, Y . Liu, T. Liu, and D. Qian, “CoFB: Latency-Constrained Co- Scheduling of Flows and Batches for Deep Learning Inference Service on the CPU–GPU System,” J. Supercomput., vol. 79, pp. 14 172–14 199, Apr. 2023

2023
[14]

Lazy Batching: An SLA-Aware Batching System for Cloud Machine Learning Inference,

Y . Choi, Y . Kim, and M. Rhu, “Lazy Batching: An SLA-Aware Batching System for Cloud Machine Learning Inference,” in Proc. IEEE Int. Symp. High Perform. Comput. Archit. (HPCA) , Seoul, Korea (South), Feb. 2021

2021
[15]

E2Bird: Enhanced Elastic Batch for Improving Responsiveness and Throughput of Deep Learning Services,

W. Cui, Q. Chen, H. Zhao, M. Wei, X. Tang, and M. Guo, “E2Bird: Enhanced Elastic Batch for Improving Responsiveness and Throughput of Deep Learning Services,” IEEE Trans. Parallel Distrib. Syst., vol. 32, no. 6, pp. 1307–1321, Jun. 2021

2021
[16]

Google cloud prediction API documentation,

GoogleCloud, “Google cloud prediction API documentation,” [On- line]. Available: https://cloud.google.com/prediction/docs/, 2017, (ac- cessed 19-Oct-2022). 18

2017
[17]

Going Deeper with Convolutions,

C. Szegedy et al. , “Going Deeper with Convolutions,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , Boston, MA, USA, Jun. 2015

2015
[18]

Virtual Batching: Request Batching for Server Energy Conservation in Virtualized Data Centers,

Y . Wang and X. Wang, “Virtual Batching: Request Batching for Server Energy Conservation in Virtualized Data Centers,” IEEE Trans. Parallel Distrib. Syst., vol. 24, no. 8, pp. 1695–1705, Aug. 2012

2012
[19]

Adaptive Scheduling Parallel Jobs with Dynamic Batching in Spark Streaming,

D. Cheng, X. Zhou, Y . Wang, and C. Jiang, “Adaptive Scheduling Parallel Jobs with Dynamic Batching in Spark Streaming,” IEEE Trans. Parallel Distrib. Syst., vol. 29, no. 12, pp. 2672–2685, Dec. 2018

2018
[20]

DBS: Dynamic Batch Size for Distributed Deep Neural Network Training,

Q. Ye, Y . Zhou, M. Shi, Y . Sun, and J. Lv, “DBS: Dynamic Batch Size for Distributed Deep Neural Network Training,” [Online]. Available: https://arxiv.org/abs/2007.11831, 2020

2007 arXiv
[21]

Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training,

J. You, J.-W. Chung, and M. Chowdhury, “Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training,” in Proc. USENIX Symp. Netw. Syst. Des. Implement. (NSDI) , Boston, MA, USA, Apr. 2023

2023
[22]

Energy- Latency Tradeoff for Dynamic Computation Offloading in Vehicular Fog Computing,

R. Yadav, W. Zhang, O. Kaiwartya, H. Song, and S. Yu, “Energy- Latency Tradeoff for Dynamic Computation Offloading in Vehicular Fog Computing,” IEEE Trans. Veh. Technol. , vol. 69, no. 12, pp. 14 198– 14 211, Dec. 2020

2020
[23]

QoS and Fairness Oriented Dynamic Computation Offloading in the Internet of Vehicles based on Estimate Time of Arrival,

C. Ling, W. Zhang, H. He, R. Yadav, J. Wang, and D. Wang, “QoS and Fairness Oriented Dynamic Computation Offloading in the Internet of Vehicles based on Estimate Time of Arrival,”IEEE Trans. Veh. Technol., vol. 73, no. 7, pp. 10 554–10 571, Jul. 2024

2024
[24]

Smart Healthcare: RL-Based Task Offloading Scheme for Edge-Enable Sensor Networks,

R. Yadav et al., “Smart Healthcare: RL-Based Task Offloading Scheme for Edge-Enable Sensor Networks,” IEEE Sens. J. , vol. 21, no. 22, pp. 24 910–24 918, Nov. 2021

2021
[25]

SERF: Efficient Scheduling for Fast Deep Neural Network Serving via Judicious Parallelism,

F. Yan, O. Ruwase, Y . He, and E. Smirni, “SERF: Efficient Scheduling for Fast Deep Neural Network Serving via Judicious Parallelism,” in Proc. Int. Conf. High Perform. Comput. Netw. Storage Anal. (SC) , Salt Lake City, UT, USA, Nov. 2016

2016
[26]

AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving,

Z. Li et al., “AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving,” in Proc. USENIX Symp. Oper. Syst. Des. Implement. (OSDI), Boston, MA, USA, Jul. 2023

2023
[27]

Swift Machine Learning Model Serving Scheduling: A Region Based Re- inforcement Learning Approach,

H. Qin, S. Zawad, Y . Zhou, L. Yang, D. Zhao, and F. Yan, “Swift Machine Learning Model Serving Scheduling: A Region Based Re- inforcement Learning Approach,” in Proc. Int. Conf. High Perform. Comput. Netw. Storage Anal. (SC) , Denver, CO, USA, Nov. 2019

2019
[28]

The Markov-Modulated Poisson Process (MMPP) Cookbook,

W. Fischer and K. Meier-Hellstern, “The Markov-Modulated Poisson Process (MMPP) Cookbook,” Performance Evaluation, vol. 18, no. 2, pp. 149–171, Sep. 1993

1993
[29]

Queueing Analysis of GPU-Based Inference Servers with Dynamic Batching: A Closed-Form Characterization,

Y . Inoue, “Queueing Analysis of GPU-Based Inference Servers with Dynamic Batching: A Closed-Form Characterization,” Performance Evaluation, vol. 147, p. 102183, May 2021

2021
[30]

Control and Optimization of Batch Processes,

D. Bonvin, “Control and Optimization of Batch Processes,” IEEE Control Syst. Mag. , vol. 6, no. 26, pp. 34–45, Dec. 2006

2006
[31]

Bulk Service Queueing Models-A Survey,

S. Sasikala and K. Indhira, “Bulk Service Queueing Models-A Survey,” Int. J. Pure Appl. Math , vol. 106, no. 6, pp. 43–56, Apr. 2016

2016
[32]

A Survey of Scheduling with Parallel Batch (p-Batch) Processing,

J. W. Fowler and L. M ¨onch, “A Survey of Scheduling with Parallel Batch (p-Batch) Processing,” Eur. J. Oper. Res., vol. 298, no. 1, pp. 1–24, Apr. 2022

2022
[33]

Optimal Control of Batch Service Queues,

R. K. Deb and R. F. Serfozo, “Optimal Control of Batch Service Queues,” Adv. Appl. Probability, vol. 5, no. 2, pp. 340–361, Aug. 1973

1973
[34]

Optimal bulking threshold of batch service queues,

Y . Zeng and C. H. Xia, “Optimal bulking threshold of batch service queues,” J. Appl. Probability , vol. 54, no. 2, pp. 409–423, Jun. 2017

2017
[35]

Multiuser Co-Inference With Batch Processing Capable Edge Server,

W. Shi, S. Zhou, Z. Niu, M. Jiang, and L. Geng, “Multiuser Co-Inference With Batch Processing Capable Edge Server,” IEEE Trans. Wireless Commun., vol. 22, no. 1, pp. 286–300, Jul. 2022

2022
[36]

Latency and Throughput Characterization of Convo- lutional Neural Networks for Mobile Computer Vision,

J. Hanhirova, T. K ¨am¨ar¨ainen, S. Sepp ¨al¨a, M. Siekkinen, V . Hirvisalo, and A. Yl¨a-J¨a¨aski, “Latency and Throughput Characterization of Convo- lutional Neural Networks for Mobile Computer Vision,” in Proc. ACM Multimedia Syst. Conf. (MMSys) , Amsterdam, Netherlands, Jun. 2018

2018
[37]

A General Class of Bulk Queues with Poisson Input,

M. F. Neuts, “A General Class of Bulk Queues with Poisson Input,” Ann. Math. Statist. , vol. 38, no. 3, pp. 759–770, Jun. 1967

1967
[38]

Analysis and Optimal Control of a Queue with Infinite Buffer Under Batch-Size Dependent Versatile Bulk-Service Rule,

A. Maity and U. C. Gupta, “Analysis and Optimal Control of a Queue with Infinite Buffer Under Batch-Size Dependent Versatile Bulk-Service Rule,” OPSEARCH, vol. 52, no. 3, pp. 472–489, Sep. 2015

2015
[39]

On the Distribution of an Infinite-Buffer Queueing System with Versatile Bulk-Service Rule Under Batch-Size-Dependent Service Policy: M/G(a,y) n /1,

S. Pradhan, “On the Distribution of an Infinite-Buffer Queueing System with Versatile Bulk-Service Rule Under Batch-Size-Dependent Service Policy: M/G(a,y) n /1,” Int. J. Math. Oper. Res. , vol. 16, no. 3, p. 407, Apr. 2020

2020
[40]

Analysis of Infinite Buffer General Bulk Service Queue with State Dependent Balking,

G. K. Gupta and A. Banerjee, “Analysis of Infinite Buffer General Bulk Service Queue with State Dependent Balking,” Int. J. Oper. Res., vol. 40, no. 2, pp. 137–161, Mar. 2021

2021
[41]

Exploiting Structure in Adaptive Dynamic Programming Algorithms for a Stochastic Batch Service Problem,

K. P. Papadaki and W. B. Powell, “Exploiting Structure in Adaptive Dynamic Programming Algorithms for a Stochastic Batch Service Problem,” Eur. J. Oper. Res., vol. 142, no. 1, pp. 108–127, Oct. 2002

2002
[42]

A Load-Balancing Problem for Distributed Bulk-Service Queues with Size-Dependent Batch Processing Times,

Y . Inoue, “A Load-Balancing Problem for Distributed Bulk-Service Queues with Size-Dependent Batch Processing Times,” Queueing Sys- tems, vol. 100, no. 3-4, pp. 449–451, Apr. 2022

2022
[43]

M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dy- namic Programming. John Wiley & Sons, 1994

1994
[44]

Finite State Approximation Algorithms for Average Cost Denumerable State Markov Decision Processes,

L. C. Thomas and D. Stengos, “Finite State Approximation Algorithms for Average Cost Denumerable State Markov Decision Processes,” Oper. Res. Spectr., vol. 7, no. 1, pp. 27–37, Mar. 1985

1985
[45]

Finite State Approximations for Denumerable State Infinite Horizon Discounted Markov Decision Processes with Unbounded Re- wards,

D. White, “Finite State Approximations for Denumerable State Infinite Horizon Discounted Markov Decision Processes with Unbounded Re- wards,” J. Math. Anal. Appl. , vol. 86, no. 1, pp. 292–306, Mar. 1982

1982
[46]

A Proof for the Queuing Formula: L = λW,

J. D. C. Little, “A Proof for the Queuing Formula: L = λW,” Operations Research, vol. 9, no. 3, pp. 383–387, Jun. 1961

1961
[47]

Optimal Control of Batch Service Queues with Finite Service Capacity and Linear Holding Costs,

S. Aalto, “Optimal Control of Batch Service Queues with Finite Service Capacity and Linear Holding Costs,” Math. Method Oper. Res., vol. 51, no. 2, pp. 263–285, Apr. 2000

2000
[48]

The Computation of Optimal Control Limits for a Queue with Batch Services,

H. J. Weiss, “The Computation of Optimal Control Limits for a Queue with Batch Services,” Management Science, vol. 25, no. 4, pp. 320–328, Apr. 1979

1979
[49]

Optimal Dispatching of an Infinite-Capacity Shuttle: Control at a Single Terminal,

E. Ignall and P. Kolesar, “Optimal Dispatching of an Infinite-Capacity Shuttle: Control at a Single Terminal,” Operations Research, vol. 22, no. 5, pp. 1008–1024, Oct. 1974

1974
[50]

Average Cost Semi-Markov Decision Processes and the Control of Queueing Systems,

L. I. Sennott, “Average Cost Semi-Markov Decision Processes and the Control of Queueing Systems,” Probability Eng. Inf. Sci. , vol. 3, no. 2, pp. 247–272, Apr. 1989

1989

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms,

Y . Xu, J. Sun, S. Zhou, and Z. Niu, “SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms,” in Proc. IEEE Int. Conf. Commun. (ICC) , Rome, Italy, May 2023

work page 2023

[2] [2]

GPU Implementation of Neural Networks,

K.-S. Oh and K. Jung, “GPU Implementation of Neural Networks,” Pattern Recognition, vol. 37, no. 6, pp. 1311–1314, Jun. 2004

work page 2004

[3] [3]

MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving,

C. Zhang, M. Yu, W. Wang, and F. Yan, “MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving,” in Proc. USENIX Annu. Tech. Conf. (ATC), Renton, W A, USA, Jul. 2019

work page 2019

[4] [4]

Clipper: A Low-Latency Online Prediction Serving System,

D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, “Clipper: A Low-Latency Online Prediction Serving System,” in Proc. USENIX Symp. Netw. Syst. Des. Implement. (NSDI) , Boston, MA, USA, Mar. 2017

work page 2017

[5] [5]

McDRAM: Low Latency and Energy-Efficient Matrix Computations in DRAM,

H. Shin, D. Kim, E. Park, S. Park, Y . Park, and S. Yoo, “McDRAM: Low Latency and Energy-Efficient Matrix Computations in DRAM,” IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. , vol. 37, no. 11, pp. 2613–2622, Nov. 2018

work page 2018

[6] [6]

Benchmarking TPU, GPU, and CPU Platforms for Deep Learning,

Y . E. Wang, G.-Y . Wei, and D. Brooks, “Benchmarking TPU, GPU, and CPU Platforms for Deep Learning,” [Online]. Available: https://arxiv. org/abs/1907.10701, 2019

arXiv 1907

[7] [7]

NVIDIA AI inference platform technical overview,

NVIDIA, “NVIDIA AI inference platform technical overview,” [On- line]. Available: https://www.nvidia.com/en-us/data-center/resources/ inference-technical-overview/, 2018, (accessed 23-Nov-2019)

work page 2018

[8] [8]

EAIS: Energy-Aware Adaptive Scheduling for CNN Inference on High-Performance GPUs,

C. Yao, W. Liu, W. Tang, and S. Hu, “EAIS: Energy-Aware Adaptive Scheduling for CNN Inference on High-Performance GPUs,” Future Gener. Comp. Syst., vol. 130, pp. 253–268, May 2022

work page 2022