Pith. sign in

REVIEW 4 minor 1 cited by

Mixture-of-Experts Serving

T0 review · 0 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces a formal model for Mixture-of-Experts serving and proves that a polynomial-time online algorithm can keep total cost within $O(\sqrt{\log k})$ of the optimal hindsight schedule, where $k$ is the number of GPUs beyond…

desk verdict Clean theoretical paper: O(sqrt(log k)) online algorithm and O(1) offline approximation for a stylized but well-defined MoE serving model; proofs check out, main caveat is external validity of the latency model. read the letter →

arxiv 2607.17880 v1 pith:CRNJ6PUB submitted 2026-07-20 cs.DS

Zhiyi Huang , Qinpei Lou , Tao Xiao This is my paper · ORCID
classification cs.DS MSC 68W2768W2568Q17
keywords Mixture-of-ExpertsonlinealgorithmcompetitiveratioGPUallocationconvexoptimizationprimal-dualreconfigurationcostNP-hardness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper formalizes the problem of dynamically assigning GPUs to experts in a Mixture-of-Experts language model, balancing per-step latency against the cost of moving GPUs between experts. It establishes that this online problem admits a polynomial-time algorithm whose total cost is within $O(\sqrt{\log k})$ of the best schedule chosen with full knowledge of the future, where $k$ is the number of spare GPUs beyond one per expert. For the offline version, it gives a constant-factor approximation and proves that no fully polynomial-time approximation scheme exists unless the Exponential Time Hypothesis fails. A sympathetic reader would care because this turns an engineering heuristic (periodic rebalancing in deployed MoE systems) into a problem with provable guarantees and a clear performance barrier.

What carries the argument

The load-bearing object is the latency cost function $f_t(x_t) = \max_i r_{t,i}/(1 + x_{t,i})$, which expresses the time for the busiest expert to finish its workload. The argument runs through its signed Fenchel conjugate $\hat f_t(\gamma_t) = \inf_{x_t \ge 0} (f_t(x_t) + \langle x_t, \gamma_t\rangle)$, whose sublinearity $\hat f_t(\theta\gamma_t) \ge \sqrt{\theta}\,\hat f_t(\gamma_t)$ is what improves the competitive ratio from the standard logarithmic bound to $O(\sqrt{\log k})$. The fractional algorithm is Regularized Greedy, which at each step minimizes the current latency cost plus a Kullback-Leibler divergence from the previous configuration; the rounding procedure, Lazy Threshold Rounding, uses per-coordinate random thresholds to preserve both expected allocation and reconfiguration cost up to constant factors.

What would settle it

Measure the end-to-end latency of a single expert in a deployed MoE system with $1,2,4,\ldots$ GPUs and fixed workload: if the latency does not decrease roughly as $1/(1+x)$ (e.g., it plateaus due to communication or bandwidth limits), then the cost model that the theorem optimizes does not describe the real system, and the competitive guarantee may not transfer.

Watch

Extended reading notes

Core claim

The central discovery is the pair of algorithmic results for the Dynamic MoE Serving problem: there is a fractional online algorithm, Regularized Greedy, that is $O(\sqrt{\log k})$-competitive, and it can be converted into a feasible integral schedule by Lazy Threshold Rounding while losing only a constant factor in both latency and reconfiguration cost. The competitive ratio is proved through a primal-dual analysis using an offset Fenchel dual, exploiting two structural properties of the latency cost function: a smoothness condition (adding GPUs can only reduce the cost in a controlled way) and a sublinearity condition on its conjugate. The paper also proves an $\Omega(\sqrt{\log k})$ lower bound for the online dual problem, showing that the analysis is tight for the primal-dual approach, and for the static (no-reconfiguration) version it gives a 2-approximation and shows that the problem is NP-hard and admits no FPTAS under ETH.

Load-bearing premise

The strongest assumption is that adding one more GPU to an expert reduces that expert's latency exactly according to $r_{t,i}/(1+x_{t,i})$, with no overhead from memory bandwidth, communication, or other interaction effects.

Editorial extensions

If this is right

  • If the central theorem is correct, MoE serving systems can dynamically reallocate GPUs with a provable worst-case guarantee that grows only as the square root of the logarithm of the number of spare GPUs, rather than accepting heuristic performance.
  • The $O(1)$-approximation for the offline problem gives a principled baseline for capacity planning and placement that does not depend on forecasting future workload popularity.
  • The $\Omega(\sqrt{\log k})$ lower bound for the online dual problem means that any primal-dual algorithm using this dual formulation cannot beat the stated competitive ratio, so improved ratios require a different analytical approach.
  • The NP-hardness of the static version implies that finding the exact fixed configuration is intractable in general, but the 2-approximation provides a simple practical fallback.
  • The model's dependence on only two cost components, latency and reconfiguration, suggests that adding more realistic constraints (e.g., communication overhead or shared experts) would change the guarantees but the framework may extend.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The smoothness and sublinearity properties are likely to hold for other latency models where doubling resources at best halves cost, so the $O(\sqrt{\log k})$ technique may transfer to resource allocation problems beyond MoE serving.
  • Editorial inference: The lower bound for the dual suggests that the $\sqrt{\log k}$ factor is not an artifact of the proof but an intrinsic price of not knowing the stopping time; whether a different relaxation could evade it remains open.
  • Editorial inference: A testable consequence is that in real deployments, the benefit of frequent rebalancing should diminish as $k$ grows, so systems with many spare GPUs can rely on simpler static or rarely-updated placements.
  • Editorial inference: The model ignores shared experts and communication costs; incorporating those would likely make the problem harder but the constant-factor approximation could still serve as a first-order placement strategy.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 4 minor

Summary. The paper formalizes a dynamic resource-allocation problem for Mixture-of-Experts serving. There are m experts and n GPUs, with k = n - m spare GPUs; at each step a workload vector r_t arrives, the system chooses an integer allocation x_t with total k, and pays a latency cost max_i r_{t,i}/(1+x_{t,i}) plus an L1 reconfiguration cost. The main result (Theorem 4.1) is a polynomial-time online algorithm that is O(sqrt(log k))-competitive, obtained by combining a KL-regularized fractional greedy update with a new Lazy Threshold Rounding scheme, together with a polynomial-time O(1)-approximation for the offline dynamic problem. Section 5 proves an Omega(sqrt(log k)) lower bound for the online dual problem that underlies the primal-dual analysis. Section 6 analyzes the static variant, giving a 2-approximation, an integrality gap of 2 - 1/m, and NP-hardness plus ETH-based FPTAS inapproximability via a reduction from Densest k-Subgraph. The paper is a theory paper motivated by systems such as DeepSeek-V3, and it explicitly abstracts away low-level deployment details.

Significance. This is a solid theoretical contribution. The positive result is non-trivial: by exploiting smoothness of the latency function and sublinearity of its conjugate, the authors improve the standard logarithmic competitive ratio of regularized-greedy online primal-dual algorithms to O(sqrt(log k)). The Lazy Threshold Rounding scheme is simple, polynomial-time, and preserves both latency and reconfiguration costs up to constant factors. The lower bound for the online dual is an honest information-theoretic barrier for the primal-dual framework, and the offline results including the ETH-based FPTAS hardness are clean. All main proofs are present and internally consistent; the regularization parameter eta is chosen analytically rather than tuned, so the bounds are parameter-free in that sense. The principal caveat is external validity: the entire analysis is for the exact latency model max_i r_{t,i}/(1+x_{t,i}), which asserts linear speedup with no memory-bandwidth, communication, batching, or shared-expert effects. The paper partly acknowledges this in Section 1.1 and footnote 1, but the caveat deserves more prominence because it delimits the practical reading of the main theorems.

minor comments (4)
  1. [Section 4.1, proof of Lemma 4.2] The bound Primal <= 2 sqrt(eta) (OPT + m + k) bounds the convex-program value Primal = sum_t f_t(x_t) + sum_t y_t, whereas the online algorithm's actual cost is sum_t f_t(x_t) + 2 sum_t y_t. The missing factor of 3 (via Lemma 4.3) is harmless for the O(sqrt(log k)) claim, but the transition from the relaxation value to the true objective should be stated explicitly so the reader can verify the constant-factor bookkeeping.
  2. [Section 5, first paragraph] The one-sentence derandomization assertion before Theorem 5.1 is terse for a theorem that claims to rule out all online algorithms, including randomized ones. Because the adversary is adaptive, the deterministic lower bound can be lifted to randomized algorithms by fixing the random seed and applying the adversary's deterministic strategy to the resulting deterministic execution; please spell out this argument in full.
  3. [Section 2 and Section 6.3] The notation in Section 2 has rendering issues (for example, 'Z >0' and 'R m>=0' appear as separate tokens rather than as subscripted sets), and the symbol k is reused in Section 6.3 for the Densest k-Subgraph parameter after being defined as the number of spare GPUs. Please clean up the notation and add a remark when k is reused.
  4. [Section 1.1 and footnote 1] Please add a sentence in the introduction or conclusion stating explicitly that the results are for the idealized latency model r/(1+x) with no memory-bandwidth, communication, batching, or shared-expert effects. Footnote 1 currently mentions only shared experts and placement optimizations; a more explicit statement about the latency model would set reader expectations about external validity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all main bounds are derived from the stated model primitives, with no fitted parameters or load-bearing self-citation.

full rationale

The paper's central results—the O(sqrt(log k))-competitive online algorithm, the O(1) offline approximation, and the Omega(sqrt(log k)) dual lower bound—are derived from the model primitives defined in Section 2 rather than being postulated or fitted. In particular, the latency cost f_t(x) = max_i r_{t,i}/(1+x_{t,i}) and the L1 reconfiguration cost are fixed at the outset; Lemmas 3.1, 3.3, and 3.4 are proven from these definitions, and Lemma 4.2 derives the competitive ratio by comparing the regularized greedy primal solution to the Fenchel dual, with the regularization parameter eta chosen analytically as eta = max{1, ln(k+1)} to optimize the analysis, not tuned to data. The rounding step (Lemma 4.4) is proven directly from the smoothness property of f_t and the threshold construction. The lower bound in Theorem 5.1 is stated and proved for the online dual problem as an independent online optimization problem; its objective and constraints are defined from the same conjugate function, and the adversary argument is self-contained, so it is not a re-derivation of the algorithm's assumptions. Self-citations appear only as pointers to standard online primal-dual and regularization techniques (e.g., references [14,15,20,21,28]) and are not used to import an unverified uniqueness theorem or ansatz; the paper supplies its own proofs for every load-bearing step it reuses. The external hardness result (Lemma 6.3) is cited to Manurangsi [40], an independent published result, and the reduction from Densest k-Subgraph is carried out in the paper. The acknowledged limitation that the linear-speedup latency model may not capture real MoE serving behavior is an external-validity concern about the model assumptions, not a circularity in the internal derivation. Thus no step reduces, by definition or by self-citation, to its own inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims rest on standard convex analysis, classical inequalities, and the explicitly stated cost model. The regularization parameter η is an algorithmic design choice, not a data-fitted constant; no new entities are postulated.

free parameters (1)
  • η (KL regularization strength) = max{1, ln(k+1)}
    Chosen by analysis in Section 4.1 to balance dual feasibility (η must be at least ln(k+1) to keep β in [0,1]) against the competitive ratio (minimizing 2√η). It is a deterministic design parameter of the algorithm, not fitted to data.
assumptions (4)
  • standard math Fenchel duality and the offset conjugate convention apply to the convex latency function extended by infinity outside the nonnegative orthant.
    Used to formulate the primal-dual pair in Section 3 and in the online analysis; relies on standard convex analysis.
  • standard math Hardy's inequality in the form Σ γ_t ≥ (1/4) Σ ( (Σ_{t'≤t} √γ_{t'}) / t )^2.
    Invoked in Lemma 5.2 to translate prefix-sum lower bounds into a lower bound on total dual mass.
  • domain assumption Manurangsi's theorem that, under ETH, DkS is hard to distinguish between clique and δ-sparse instances with δ = m^{-1/(log log m)^c}.
    Used in Theorem 6.4 to rule out an FPTAS for Static MoE Serving; the hardness result is external and depends on ETH.
  • domain assumption The latency cost of an expert is inversely proportional to its assigned GPU count, and system latency is the maximum over experts.
    Defines the model in Section 2. All guarantees are measured against this cost, so the results are meaningful only to the extent that this abstraction is faithful.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mixture-of-Experts Serving." pith.science (2026). https://pith.science/paper/CRNJ6PUB

@misc{pith2026260717880,
  author       = {Pith},
  title        = {Pith review of: Mixture-of-Experts Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CRNJ6PUB}},
  note         = {Machine review of arXiv:2607.17880}
}
abstract

Mixture-of-Experts (MoE) models route each token to only a few expert networks, distributing the serving load across experts whose popularity shifts over time. A serving system must therefore dynamically decide how many GPUs to assign to each expert, trading off service latency against the cost of reconfiguring the assignment. We introduce a formal model of MoE Serving and initiate a principled study of online and offline algorithms for it. Our main result is a polynomial-time $O(\sqrt{\log k})$-competitive online algorithm, where $k$ is the number of GPUs beyond one per expert. We complement it with a matching $\Omega(\sqrt{\log k})$ barrier for the online dual problem underlying our analysis. In the offline setting, we give a constant-factor approximation, show that MoE Serving is NP-hard, and rule out an FPTAS assuming ETH.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes

    cs.DC 2026-08 accept novelty 7.0 of 10

    Expert cost in MoE serving follows a two-regime max-affine law, and a makespan-aware dispatcher over this model tracks the best fixed policy and wins where regimes mix.

Reference graph

Works this paper leans on

48 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    arXiv 2025

  2. [2]

    Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gula- vani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In18th USENIX symposium on operating systems design and implementation (OSDI 24), pages 117–134, 2024

    work page 2024

  3. [3]

    A tale of two metrics: Simultaneous bounds on competitiveness and regret

    Lachlan Andrew, Siddharth Barman, Katrina Ligett, Minghong Lin, Adam Meyerson, Alan Roytman, and Adam Wierman. A tale of two metrics: Simultaneous bounds on competitiveness and regret. InConference on Learning Theory, pages 741–763. PMLR, 2013. 19

    work page 2013

  4. [4]

    Dimension-free bounds for chasing convex functions

    CJ Argue, Anupam Gupta, and Guru Guruganesh. Dimension-free bounds for chasing convex functions. InConference on Learning Theory, pages 219–241. PMLR, 2020

    2020

  5. [5]

    Online algo- rithms for covering and packing problems with convex objectives

    Yossi Azar, Niv Buchbinder, TH Hubert Chan, Shahar Chen, Ilan Reuven Cohen, Anupam Gupta, Zhiyi Huang, Ning Kang, Viswanath Nagarajan, Joseph Naor, et al. Online algo- rithms for covering and packing problems with convex objectives. In2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pages 148–157. IEEE, 2016

    work page 2016

  6. [6]

    Randomized competitive algorithms for generalized caching.SIAM Journal on Computing, 41(2):391–414, 2012

    Nikhil Bansal, Niv Buchbinder, and Joseph Naor. Randomized competitive algorithms for generalized caching.SIAM Journal on Computing, 41(2):391–414, 2012

    work page 2012

  7. [7]

    A 2-competitive algorithm for online convex optimization with switching costs

    Nikhil Bansal, Anupam Gupta, Ravishankar Krishnaswamy, Kirk Pruhs, Kevin Schewior, and Cliff Stein. A 2-competitive algorithm for online convex optimization with switching costs. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques (APPROX/RANDOM 2015), pages 96–109. Schloss Dagstuhl–Leibniz-Zentrum f¨ ur Informatik, 2015

    work page 2015

  8. [8]

    A polylog (n)-competitive algo- rithm for metrical task systems

    Yair Bartal, Avrim Blum, Carl Burch, and Andrew Tomkins. A polylog (n)-competitive algo- rithm for metrical task systems. InProceedings of the twenty-ninth annual ACM symposium on Theory of computing, pages 711–719, 1997

    work page 1997

Show all 48 references
  1. [9]

    Nonclairvoyant scheduling to minimize the total flow time on single and parallel machines.Journal of the ACM (JACM), 51(4):517–539, 2004

    Luca Becchetti and Stefano Leonardi. Nonclairvoyant scheduling to minimize the total flow time on single and parallel machines.Journal of the ACM (JACM), 51(4):517–539, 2004

    2004

  2. [10]

    Finely-competitive paging

    Avrim Blum, Carl Burch, and Adam Kalai. Finely-competitive paging. In40th Annual Sym- posium on Foundations of Computer Science (Cat. No. 99CB37039), pages 450–457. IEEE, 1999

    1999

  3. [11]

    An optimal on-line algorithm for metrical task system.Journal of the ACM (JACM), 39(4):745–763, 1992

    Allan Borodin, Nathan Linial, and Michael E Saks. An optimal on-line algorithm for metrical task system.Journal of the ACM (JACM), 39(4):745–763, 1992

    1992

  4. [12]

    K- server via multiscale entropic regularization

    S´ ebastien Bubeck, Michael B Cohen, Yin Tat Lee, James R Lee, and Aleksander Madry. K- server via multiscale entropic regularization. InProceedings of the 50th annual ACM SIGACT symposium on theory of computing, pages 3–16, 2018

    2018

  5. [13]

    Competitively chasing convex bodies

    S´ ebastien Bubeck, Yin Tat Lee, Yuanzhi Li, and Mark Sellke. Competitively chasing convex bodies. InProceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, pages 861–868, 2019

    2019

  6. [14]

    Competitive analysis via regularization

    Niv Buchbinder, Shahar Chen, and Joseph Naor. Competitive analysis via regularization. In Proceedings of the twenty-fifth annual ACM-SIAM symposium on Discrete algorithms, pages 436–444. SIAM, 2014

    2014

  7. [15]

    k-servers with a smile: Online algorithms via projections

    Niv Buchbinder, Anupam Gupta, Marco Molinaro, and Joseph Naor. k-servers with a smile: Online algorithms via projections. InProceedings of the Thirtieth Annual ACM-SIAM Sym- posium on Discrete Algorithms, pages 98–116. SIAM, 2019

    2019

  8. [16]

    Minimizing total completion time on batch processing machines.The International Journal Of Production Research, 31(9):2097–2121, 1993

    Vijaya Chandru, C-Y Lee, and Reha Uzsoy. Minimizing total completion time on batch processing machines.The International Journal Of Production Research, 31(9):2097–2121, 1993. 20

    1993

  9. [17]

    Smoothed online convex optimization in high dimensions via online balanced descent

    Niangjun Chen, Gautam Goel, and Adam Wierman. Smoothed online convex optimization in high dimensions via online balanced descent. InConference On Learning Theory, pages 1574–1594. PMLR, 2018

    2018

  10. [18]

    Pure entropic regularization for metrical task systems

    Christian Coester and James R Lee. Pure entropic regularization for metrical task systems. InConference on Learning Theory, pages 835–848. PMLR, 2019

    2019

  11. [19]

    Deepseekmoe: Towards ultimate expert specializa- tion in mixture-of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specializa- tion in mixture-of-experts language models. InProceedings of the 62nd Annual Meeting of the Association f...

    2024

  12. [20]

    Primal dual gives almost optimal energy efficient online algorithms

    Nikhil R Devanur and Zhiyi Huang. Primal dual gives almost optimal energy efficient online algorithms. InProceedings of the twenty-fifth annual ACM-SIAM symposium on Discrete algorithms, pages 1123–1140, 2014

    2014

  13. [21]

    Online matching with concave returns

    Nikhil R Devanur and Kamal Jain. Online matching with concave returns. InProceedings of the forty-fourth annual ACM symposium on Theory of computing, pages 137–144, 2012

    2012

  14. [22]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  15. [23]

    Competitive paging algorithms.Journal of Algorithms, 12(4):685–699, 1991

    Amos Fiat, Richard M Karp, Michael Luby, Lyle A McGeoch, Daniel D Sleator, and Neal E Young. Competitive paging algorithms.Journal of Algorithms, 12(4):685–699, 1991

    1991

  16. [24]

    On convex body chasing.Discrete & Computational Ge- ometry, 9(3):293–321, 1993

    Joel Friedman and Nathan Linial. On convex body chasing.Discrete & Computational Ge- ometry, 9(3):293–321, 1993

    1993

  17. [25]

    Moetuner: Optimized mixture of expert serving with balanced expert placement and token routing.arXiv preprint arXiv:2502.06643, 2025

    Seokjin Go and Divya Mahajan. Moetuner: Optimized mixture of expert serving with balanced expert placement and token routing.arXiv preprint arXiv:2502.06643, 2025

    2025 arXiv

  18. [26]

    Gemini 3 Pro model card, November 2025

    Google DeepMind. Gemini 3 Pro model card, November 2025. Model card; accessed July 2026

    2025

  19. [27]

    Scheduling semiconductor burn-in operations to minimize total flowtime.Operations research, 45(6):874–885, 1997

    Dorit S Hochbaum and Dan Landy. Scheduling semiconductor burn-in operations to minimize total flowtime.Operations research, 45(6):874–885, 1997

    1997

  20. [28]

    Welfare maximization with production costs: A primal dual approach

    Zhiyi Huang and Anthony Kim. Welfare maximization with production costs: A primal dual approach. InProceedings of the Twenty-Sixth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 59–72. SIAM, 2014

    2014

  21. [29]

    Adaptive mixtures of local experts.Neural computation, 3(1):79–87, 1991

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts.Neural computation, 3(1):79–87, 1991

    1991

  22. [30]

    Online scheduling for llm inference with kv cache constraints.arXiv preprint arXiv:2502.07115, 2025

    Patrick Jaillet, Jiashuo Jiang, Konstantina Mellou, Marco Molinaro, Chara Podimata, and Zijie Zhou. Online scheduling for llm inference with kv cache constraints.arXiv preprint arXiv:2502.07115, 2025

    2025

  23. [31]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  24. [32]

    Hierarchical mixtures of experts and the em algorithm

    Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994. 21

    1994

  25. [33]

    Speed is as powerful as clairvoyance.Journal of the ACM (JACM), 47(4):617–643, 2000

    Bala Kalyanasundaram and Kirk Pruhs. Speed is as powerful as clairvoyance.Journal of the ACM (JACM), 47(4):617–643, 2000

    2000

  26. [34]

    Geometry-aware online scheduling for llm serving: From theoretical bound to system practice.arXiv preprint arXiv:2606.22327, 2026

    Li Kong, Qi Qi, Yinyu Ye, and Zijie Zhou. Geometry-aware online scheduling for llm serving: From theoretical bound to system practice.arXiv preprint arXiv:2606.22327, 2026

    2026 arXiv

  27. [35]

    Papadimitriou

    Elias Koutsoupias and Christos H. Papadimitriou. On thek-server conjecture.Journal of the ACM, 42(5):971–983, 1995

    1995

  28. [36]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th symposium on operating systems principles, pa...

    2023

  29. [37]

    Accelerating distributed MoE training and inference with lina

    Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. Accelerating distributed MoE training and inference with lina. In2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 945–959, 2023

    2023

  30. [38]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  31. [39]

    Manasse, Lyle A

    Mark S. Manasse, Lyle A. McGeoch, and Daniel D. Sleator. Competitive algorithms for server problems.Journal of Algorithms, 11(2):208–230, 1990

    1990

  32. [40]

    Almost-polynomial ratio eth-hardness of approximating densest k- subgraph

    Pasin Manurangsi. Almost-polynomial ratio eth-hardness of approximating densest k- subgraph. InProceedings of the 49th Annual ACM SIGACT Symposium on Theory of Com- puting, pages 954–961, 2017

    2017

  33. [41]

    Nonclairvoyant scheduling.Theoretical computer science, 130(1):17–47, 1994

    Rajeev Motwani, Steven Phillips, and Eric Torng. Nonclairvoyant scheduling.Theoretical computer science, 130(1):17–47, 1994

    1994

  34. [42]

    Chasing convex bodies optimally

    Mark Sellke. Chasing convex bodies optimally. InProceedings of the Fourteenth Annual ACM- SIAM Symposium on Discrete Algorithms, pages 1509–1518. SIAM, 2020

    2020

  35. [43]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations, 2017

    2017

  36. [44]

    Amortized efficiency of list update and paging rules

    Daniel D Sleator and Robert E Tarjan. Amortized efficiency of list update and paging rules. Communications of the ACM, 28(2):202–208, 1985

    1985

  37. [45]

    Scheduling a single batch processing machine with non-identical job sizes.The International Journal of Production Research, 32(7):1615–1635, 1994

    Reha Uzsoy. Scheduling a single batch processing machine with non-identical job sizes.The International Journal of Production Research, 32(7):1615–1635, 1994

    1994

  38. [46]

    Llm serving optimization with variable prefill and decode lengths.arXiv preprint arXiv:2508.06133, 2025

    Meixuan Wang, Yinyu Ye, and Zijie Zhou. Llm serving optimization with variable prefill and decode lengths.arXiv preprint arXiv:2508.06133, 2025

    2025 arXiv

  39. [47]

    FastServe:Iteration-Level preemptive scheduling for large language model inference

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. FastServe:Iteration-Level preemptive scheduling for large language model inference. In23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI...

    2026

  40. [48]

    Orca: A distributed serving system for Transformer-Based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In16th USENIX symposium on operating systems design and implementation (OSDI 22), pages 521–538, 2022. 23

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.