REVIEW 16 cited by
BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Serving systems for Large Language Models (LLMs) are often optimized to improve quality of service (QoS) and throughput. However, due to the lack of open-source LLM serving workloads, these systems are frequently evaluated under unrealistic workload assumptions. Consequently, performance may degrade when systems are deployed in real-world scenarios. This work presents BurstGPT, an LLM serving workload with 10.31 million traces from regional Azure OpenAI GPT services over 213 days. BurstGPT captures LLM serving characteristics from user, model and system perspectives: (1) User request concurrency: burstiness variations of requests in Azure OpenAI GPT services, revealing diversified concurrency patterns in different services and model types. (2) User conversation patterns: counts and intervals within conversations for service optimizations. (3) Model response lengths: auto-regressive serving processes of GPT models, showing statistical relations between requests and their responses. (4) System response failures: failures of conversation and API services, showing intensive resource needs and limited availability of LLM services in Azure. The details of the characteristics can serve multiple purposes in LLM serving optimizations, such as system evaluation and trace provisioning. In our demo evaluation with BurstGPT, frequent variations in BurstGPT reveal declines in efficiency, stability, or reliability in realistic LLM serving. We identify that the generalization of KV cache management, scheduling and disaggregation optimizations can be improved under realistic workload evaluations. BurstGPT is publicly available now at https://github.com/HPMLL/BurstGPT and is widely used to develop prototypes of LLM serving frameworks in the industry.
Forward citations
Cited by 16 Pith papers
-
TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving
Coding-agent prompts can be re-tokenized incrementally or on a GPU without changing token IDs, cutting front-end tokenization from O(full context) to O(append).
-
CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving
Layer-wise data-parallel replication of hot Transformer layers onto reclaimed idle GPUs reduces LLM serving cold-start latency 97.9–99.3% and average latency 20.7–28.1% while attaining 100% SLO on production traces.
-
Trusted Floors Under Untrusted Learners: A Runtime Assured-SLO Guard for ML Serving
Reservation plus assured-first priority holds admitted assured-class miss at 0.0 on real 2xV100 under every miscalibration of a learned admitter, while GAIE Flow Control fails under label swap.
-
Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism
Hetis dynamically splits LLM attention across heterogeneous GPUs at the granularity of individual heads while keeping dense computation on a selected subset, and reports up to 2.25x throughput gains over Splitwise and Hexgen.
-
Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics
SISO caches representative centroid embeddings of similar queries and adjusts its similarity threshold under load, improving semantic cache hit ratios for LLM serving by up to 1.71x with a small accuracy cost.
-
Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts
A DRL router using graph attention state abstraction and QoS-aware rewards improves average QoS by up to 35.78% over four baselines in simulated edge LLM routing.
-
MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systems
A two-step soft-prompt backdoor attack called Robo-Troj (listed as MuTRAP on arXiv) makes LLM-based robot planners emit malicious plans when hidden trigger words are present, with near-perfect attack success.
-
SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference
SeaLLM shares GPU resources across multiple LLM services with a preemptive, service-characteristic-aware scheduler, search-based placement, adaptive replacement, and a merged-block unified KV cache, cutting normalized...
-
An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models
Public LLM services show distinct failure patterns: ChatGPT outages last longer but happen less often than Claude, failures follow weekly and monthly rhythms, and OpenAI isolates failures across services better than A...
-
Energy Considerations of Large Language Model Inference and Efficiency Optimizations
Measuring LLM inference across workloads, frameworks, and GPUs shows that efficiency optimizations like vLLM and CUDA graphs can cut energy use by up to 73% versus an unoptimized PyTorch baseline.
-
RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU
RAGDoll pipelines retrieval and generation, jointly manages memory across disk, RAM, and GPU, and adaptively sizes batches to cut average RAG latency by up to 3.6x on a single GPU.
-
Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
A MILP scheduler that jointly optimizes GPU composition, deployment configuration, and workload assignment reports 20-41% cost-efficiency gains over homogeneous GPU clusters for LLM serving.
-
Adaptive Inference Batching using Policy Gradients
A REINFORCE agent trained on a custom simulator learns workload-segregation routing that outperforms Shortest-Queue by 48% in heterogeneous multi-GPU inference serving, while offering negligible gains over static batc...
-
Past-Future Scheduler for LLM Serving under SLA Guarantees
A past-future scheduler predicts output lengths from historical requests and only admits new requests when the predicted future memory peak of the running batch fits in GPU memory.
-
SecPE: Secure Prompt Ensembling for Private and Robust Large Language Models
SecPE uses a log-depth max tree to compute encrypted argmax, cutting the cost of private prompt ensembling by up to 35x with negligible accuracy loss on benchmarks.
-
Topology-aware Preemptive Scheduling for Co-located LLM Workloads
A topology-aware preemption scheduler for co-located LLM workloads raises topology affinity hit rate from 44.5% to 100% in simulation, but the 55% performance improvement is inferred, not directly measured.
Discussion (0). Continue with ORCID to comment.