REVIEW 6 cited by
Parrot: Efficient Serving of LLM-based Applications with Semantic Variable
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The rise of large language models (LLMs) has enabled LLM-based applications (a.k.a. AI agents or co-pilots), a new software paradigm that combines the strength of LLM and conventional software. Diverse LLM applications from different tenants could design complex workflows using multiple LLM requests to accomplish one task. However, they have to use the over-simplified request-level API provided by today's public LLM services, losing essential application-level information. Public LLM services have to blindly optimize individual LLM requests, leading to sub-optimal end-to-end performance of LLM applications. This paper introduces Parrot, an LLM service system that focuses on the end-to-end experience of LLM-based applications. Parrot proposes Semantic Variable, a unified abstraction to expose application-level knowledge to public LLM services. A Semantic Variable annotates an input/output variable in the prompt of a request, and creates the data pipeline when connecting multiple LLM requests, providing a natural way to program LLM applications. Exposing Semantic Variables to the public LLM service allows it to perform conventional data flow analysis to uncover the correlation across multiple LLM requests. This correlation opens a brand-new optimization space for the end-to-end performance of LLM-based applications. Extensive evaluations demonstrate that Parrot can achieve up to an order-of-magnitude improvement for popular and practical use cases of LLM applications.
Forward citations
Cited by 6 Pith papers
-
A Workflow-Aware Serving Layer for Agentic Applications
A workflow-aware serving layer compiles per-node model-verifier-backend plans with an ILP and adapts only uncommitted work via pre-solved pressure rungs and residual re-solves.
-
BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching
BlendServe combines resource-aware batching with prefix sharing using a resource-aware prefix tree and dual scanner, achieving up to 1.44x throughput vs vLLM/SGLang in offline LLM inference.
-
Semantic Scheduling for LLM Inference
A semantic scheduler for LLM inference uses urgency labels and estimated remaining compute to cut waiting times for urgent requests, tested on emergency medical data.
-
GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines
The paper proposes offloading AI pipeline data processing tasks to SmartNICs and sketches designs for normalization, bilinear interpolation, and tokenization, without implementing them.
-
CHASE: A Native Relational Database for Hybrid Queries on Structured and Unstructured Data
CHASE natively integrates ANN vector search into a compiled relational engine, cuts redundant similarity computation in hybrid queries, and reports up to 7,500x speedups.
-
Taming the Titans: A Survey of Efficient LLM Inference Serving
A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.
Discussion (0). Continue with ORCID to comment.