Pith. sign in

REVIEW 5 cited by

LLM Inference Serving: Survey of Recent Advances and Opportunities

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.12391 v1 pith:7X2XVFCQ submitted 2024-07-17 cs.DC cs.AI

Baolin Li , Yankai Jiang , Vijay Gadepally , Devesh Tiwari This is my paper · ORCID
classification cs.DCcs.AI
keywords surveyrecentservingabreastadvancementsadvancesalteringcomprehensive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This survey offers a comprehensive overview of recent advancements in Large Language Model (LLM) serving systems, focusing on research since the year 2023. We specifically examine system-level enhancements that improve performance and efficiency without altering the core LLM decoding mechanisms. By selecting and reviewing high-quality papers from prestigious ML and system venues, we highlight key innovations and practical considerations for deploying and scaling LLMs in real-world production environments. This survey serves as a valuable resource for LLM practitioners seeking to stay abreast of the latest developments in this rapidly evolving field.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Adaptive LLM Routing under Budget Constraints

    cs.LG 2025-08 conditional novelty 6.0 of 10

    LLM routing is framed as a budget-constrained contextual bandit, solved by a preference-prior initialized LinUCB variant with an online multi-choice knapsack cost policy.

  2. InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata

    cs.DC 2026-07 conditional novelty 5.0 of 10

    InstantInfer refactors vLLM's cold start into a concurrent state-machine pipeline, speeding up startup by up to 7.2×.

  3. Universal Model Routing for Efficient LLM Inference

    cs.CL 2025-02 conditional novelty 5.0 of 10

    UniRoute represents each language model by its error rates on a few prompt clusters, letting a router choose among models it has never seen during training.

  4. Joint Partitioning and Placement of Foundation Models for Real-Time Edge AI

    cs.DC 2025-11 reject novelty 4.0 of 10

    A framework for runtime re-splitting and re-placement of foundation model layers across edge nodes is proposed, but its claimed latency gains are inherited from prior work rather than measured.

  5. Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques

    cs.LG 2025-06 unverdicted novelty 4.0 of 10

    A survey of LLM routing and hierarchical inference techniques that proposes an unvalidated unified evaluation metric called the Inference Efficiency Score.

Pith tools