Pith. sign in

REVIEW 8 cited by

BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.17644 v5 pith:V5HYVXB7 submitted 2024-01-31 cs.DC cs.PF

Yuxin Wang , Yuhan Chen , Zeyu Li , Xueze Kang , Yuchu Fang , Yeju Zhou , Yang Zheng , Zhenheng Tang
show 6 more authors
Xin He Rui Guo Xin Wang Qiang Wang Amelie Chi Zhou Xiaowen Chu
This is my paper
classification cs.DCcs.PF
keywords servingburstgptservicessystemsworkloadazuremodeloptimizations
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Serving systems for Large Language Models (LLMs) are often optimized to improve quality of service (QoS) and throughput. However, due to the lack of open-source LLM serving workloads, these systems are frequently evaluated under unrealistic workload assumptions. Consequently, performance may degrade when systems are deployed in real-world scenarios. This work presents BurstGPT, an LLM serving workload with 10.31 million traces from regional Azure OpenAI GPT services over 213 days. BurstGPT captures LLM serving characteristics from user, model and system perspectives: (1) User request concurrency: burstiness variations of requests in Azure OpenAI GPT services, revealing diversified concurrency patterns in different services and model types. (2) User conversation patterns: counts and intervals within conversations for service optimizations. (3) Model response lengths: auto-regressive serving processes of GPT models, showing statistical relations between requests and their responses. (4) System response failures: failures of conversation and API services, showing intensive resource needs and limited availability of LLM services in Azure. The details of the characteristics can serve multiple purposes in LLM serving optimizations, such as system evaluation and trace provisioning. In our demo evaluation with BurstGPT, frequent variations in BurstGPT reveal declines in efficiency, stability, or reliability in realistic LLM serving. We identify that the generalization of KV cache management, scheduling and disaggregation optimizations can be improved under realistic workload evaluations. BurstGPT is publicly available now at https://github.com/HPMLL/BurstGPT and is widely used to develop prototypes of LLM serving frameworks in the industry.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving

    cs.CL 2026-07 conditional novelty 8.0 of 10

    Coding-agent prompts can be re-tokenized incrementally or on a GPU without changing token IDs, cutting front-end tokenization from O(full context) to O(append).

  2. CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving

    cs.DC 2026-07 conditional novelty 6.5 of 10

    Layer-wise data-parallel replication of hot Transformer layers onto reclaimed idle GPUs reduces LLM serving cold-start latency 97.9–99.3% and average latency 20.7–28.1% while attaining 100% SLO on production traces.

  3. Trusted Floors Under Untrusted Learners: A Runtime Assured-SLO Guard for ML Serving

    cs.DC 2026-07 conditional novelty 6.0 of 10

    Reservation plus assured-first priority holds admitted assured-class miss at 0.0 on real 2xV100 under every miscalibration of a learned admitter, while GAIE Flow Control fails under label swap.

  4. Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism

    cs.DC 2025-09 conditional novelty 6.0 of 10

    Hetis dynamically splits LLM attention across heterogeneous GPUs at the granularity of individual heads while keeping dense computation on a selected subset, and reports up to 2.25x throughput gains over Splitwise and Hexgen.

  5. Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics

    cs.DB 2025-08 conditional novelty 6.0 of 10

    SISO caches representative centroid embeddings of similar queries and adjusts its similarity threshold under load, improving semantic cache hit ratios for LLM serving by up to 1.71x with a small accuracy cost.

  6. Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts

    cs.NI 2025-08 conditional novelty 6.0 of 10

    A DRL router using graph attention state abstraction and QoS-aware rewards improves average QoS by up to 35.78% over four baselines in simulated edge LLM routing.

  7. Adaptive Inference Batching using Policy Gradients

    cs.LG 2026-07 conditional novelty 4.0 of 10

    A REINFORCE agent trained on a custom simulator learns workload-segregation routing that outperforms Shortest-Queue by 48% in heterogeneous multi-GPU inference serving, while offering negligible gains over static batc...

  8. Past-Future Scheduler for LLM Serving under SLA Guarantees

    cs.DC 2025-07 conditional novelty 4.0 of 10

    A past-future scheduler predicts output lengths from historical requests and only admits new requests when the predicted future memory peak of the running batch fits in GPU memory.

Pith tools