REVIEW 36 cited by
Autellix: An Efficient Serving Engine for LLM Agents as General Programs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Autellix: An Efficient Serving Engine for LLM Agents as General Programs
read the original abstract
Large language model (LLM) applications are evolving beyond simple chatbots into dynamic, general-purpose agentic programs, which scale LLM calls and output tokens to help AI agents reason, explore, and solve complex tasks. However, existing LLM serving systems ignore dependencies between programs and calls, missing significant opportunities for optimization. Our analysis reveals that programs submitted to LLM serving engines experience long cumulative wait times, primarily due to head-of-line blocking at both the individual LLM request and the program. To address this, we introduce Autellix, an LLM serving system that treats programs as first-class citizens to minimize their end-to-end latencies. Autellix intercepts LLM calls submitted by programs, enriching schedulers with program-level context. We propose two scheduling algorithms-for single-threaded and distributed programs-that preempt and prioritize LLM calls based on their programs' previously completed calls. Our evaluation demonstrates that across diverse LLMs and agentic workloads, Autellix improves throughput of programs by 4-15x at the same latency compared to state-of-the-art systems, such as vLLM.
Forward citations
Cited by 36 Pith papers
-
VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?
VibeServe demonstrates that AI agents can synthesize bespoke LLM serving systems end-to-end, remaining competitive with vLLM in standard settings while outperforming it in six non-standard scenarios involving unusual ...
-
Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation
Tessera performs kernel-granularity disaggregation on heterogeneous GPUs, achieving up to 2.3x throughput and 1.6x cost efficiency gains for large model inference while generalizing beyond prior methods.
-
Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
Across 13.5M production GitHub Copilot sessions, coding-agent work is dominated by autonomous LLM–tool loops, with KV-cache reuse at ~90% inside a turn but 55% at turn boundaries and 8% after model switches.
-
SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering
SmoothAgent introduces lookahead context engineering to eliminate transformation overhead in LLM agents, reducing TTFT by up to 11.9x through proactive KV cache preparation.
-
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Presents a distribution-aware scheduling framework for LLM inference that reduces P99 TTLT by 35-50% and TTFT by 34-47% versus SRPT with perfect length knowledge using statistical signals instead of predictions.
-
AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving
AGENTSERVESIM is a new simulator that models multi-turn LLM agent serving at program granularity and reproduces real-system performance within 6% error on commodity CPUs.
-
A Policy-Driven Runtime Layer for Agentic LLM Serving
Introduces a three-tier architecture with an agent runtime layer and four primitives for agent-aware policies in LLM serving, validated on KV caching via CacheSage showing 13-37pp hit-rate gains on five workloads.
-
FATE: Future-State-Aware Scheduling for Heterogeneous LLM Workflows
FATE reduces normalized makespan and P95 latency in real LLM workflow DAGs to 0.675 and 0.677 by jointly preserving multiple future execution states, outperforming RoundRobin by 32.5% and the strongest baseline by 8.9%.
-
SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters
SAGA introduces workflow-atomic scheduling for compound AI agents, achieving 1.64x lower task completion time and 1.22x better memory utilization than vLLM on a 64-GPU cluster at the cost of 30% lower peak throughput.
-
SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters
SAGA reduces AI agent task completion time by 1.64x on 64-GPU clusters by scheduling at the full workflow level with execution graphs, affinity batching, and completion-time fairness.
-
Low-Scaling Many-Body Green's Function Calculations for Molecular Systems via Interacting-Bath Dynamical Embedding Theory
ibDET assembles the full molecular Green's function from multiple small interacting-bath embedding calculations, delivering spectral properties with ~0.1 eV accuracy at far lower cost than full-system methods.
-
Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflows
This work delivers the first measurements of performance-energy trade-offs across four multi-request LLM workflow patterns on A100 GPUs using vLLM and Parrot.
-
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
Balancing only first-turn agent requests while keeping follow-ups cache-aware raises LLM cluster TPS 10-16% (colocation) and prefill TPS 2-34% (disaggregation) over SOTA schedulers on production agent traces.
-
A Workflow-Aware Serving Layer for Agentic Applications
A workflow-aware serving layer compiles per-node model-verifier-backend plans with an ILP and adapts only uncommitted work via pre-solved pressure rungs and residual re-solves.
-
Coordinated Networking for On-Device Agent-Augmented Real-Time Communication
HAFS coordinates video and AI-agent context flows in real-time communication, raising video quality 1.5x and cutting agent response time 31% in testbed trials.
-
TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI
Task-consistent sticky routing with a contextual bandit improves accuracy-latency trade-offs on agentic benchmarks by learning from delayed task-level reward.
-
Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI
MORI improves throughput 20-71% and TTFT 18-43% over baselines by ranking programs on a continuous idleness spectrum and shifting the GPU-CPU boundary to match capacity in agentic LLM serving.
-
A Policy-Driven Runtime Layer for Agentic LLM Serving
A four-primitive agent runtime layer between agent frameworks and LLM serving engines raises cross-session KV-cache hit rate by 13–37 percentage points in the paper's experiments.
-
KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving
KAIROS reduces power by 27% on average (up to 39.8%) for agentic AI inference by using long-lived context to jointly manage GPU frequency, concurrency, and request routing across instances.
-
Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines
Scepsy schedules arbitrary multi-LLM agentic workflows on GPU clusters by constructing Aggregate LLM Pipelines from stable per-LLM execution time shares, then searching fractional GPU allocations, tensor parallelism, ...
-
MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems
MARS coordinates heterogeneous GPU-CPU resources for agentic LLM workloads via decoupled admission control and agent-centric KV cache management, delivering up to 5.94x lower latency and 1.87x faster task completion.
-
Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation
R-DSGD and R-DSGD-M under (δ,c)-robust aggregation have tight Byzantine error floors under (B,ζ)-bounded dissimilarity; local momentum eliminates the stochastic-noise term but not the heterogeneity term.
-
ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache
ForkKV uses copy-on-write disaggregated KV cache with DualRadixTree and ResidualAttention kernels to deliver up to 3x throughput over prior multi-LoRA serving systems with negligible quality loss.
-
TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
TokenDance scales multi-agent LLM serving to 2.7x more concurrent agents by collective KV cache reuse and block-sparse diff encoding that achieves 11-17x compression.
-
Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
Pattern-aware speculative tool execution cuts agent end-to-end latency by roughly half and observed tool latency by about 1.8× by overlapping predicted tools with LLM generation.
-
ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
A program-aware scheduler and tool-lifecycle manager for LLM agent workflows raises serving throughput by 1.5–3.6x and RL rollout throughput by 1.8–3.9x over vLLM/Continuum baselines.
-
Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
Continuum applies a time-to-live mechanism to KV cache retention during tool calls in multi-turn LLM agents, reporting over 8x faster average job completion times on benchmarks including SWE-Bench with models up to 35...
-
Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
TTL-based KV-cache pinning that uses predicted tool-call durations and queueing-delay costs cuts agent job completion time by up to 8x.
-
TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
TokenCake introduces agent-aware temporal and spatial schedulers for KV cache management in LLM multi-agent serving, claiming over 47% lower end-to-end latency and up to 16.9% better GPU memory utilization than vLLM o...
-
Scalable LLM Agent Tool Access in the Cloud
A cloud-scale MCP gateway with hybrid dense-sparse retrieval lets LLM agents work with 3,000+ tools at 98% Top-15 recall, cutting tool-selection time 8.9× and token use 23.8×.
-
Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
AsymCache combines Multi-Segment Attention, position-aware eviction, and adaptive chunking to cut TTFT by up to 2.03x and TPOT by up to 1.71x versus recent baselines in LLM serving.
-
An Executable Benchmarking Suite for Tool-Using Agents
The paper delivers a unified executable benchmarking suite for tool-using agents that enforces a shared evidence-admission contract across web, code, and micro-task environments.
-
Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management
PBKV predicts agent invocations in dynamic LLM workflows to manage KV-cache reuse, delivering up to 1.85x speedup over LRU and 1.26x over KVFlow.
-
AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agent
AgentOpt introduces a framework-agnostic package that uses algorithms like UCB-E to find cost-effective model assignments in multi-step LLM agent pipelines, cutting evaluation budgets by 62-76% while maintaining near-...
-
Low-Scaling Many-Body Green's Function Calculations for Molecular Systems via Interacting-Bath Dynamical Embedding Theory
Molecular ibDET assembles full-system GW/EOM-CCSD Green's functions from small atom-centered impurity problems and recovers IPs/EAs within about 0.1 eV of full-system results.
-
An Executable Benchmarking Suite for Tool-Using Agents
Introduces a benchmarking suite with common workload adapters, event schemas, and an evidence gate connecting WebArena Verified, SWE-Gym, and MiniWoB++ for tool-using agents.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.