Pith. sign in

REVIEW 5 major objections 5 minor 5 cited by

KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read KVFlow proposes removing cache-miss stalls in multi-agent LLM serving by making eviction and prefetching follow each agent's expected next execution step.

desk verdict KVFlow's workflow-aware cache management is a genuinely new and sensible mechanism, but the abstract's 'eliminates cache misses' claim is unsupported by the current evaluation, which lacks dynamic-branching tests and a shipped artifact. read the letter →

arxiv 2507.07400 v1 pith:H35TVOAN submitted 2025-07-10 cs.DC cs.MA

Zaifeng Pan , Ajjkumar Patel , Zhengding Hu , Yipeng Shen , Yue Guan , Wan-Lu Li , Lianhui Qin , Yida Wang
show 1 more author
Yufei Ding
This is my paper
classification cs.DCcs.MA
keywords KVcacheprefixcachingmulti-agentworkflowsLLMservingevictionpolicyprefetchingAgentStepGraphradix
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

KVFlow argues that the standard LRU eviction policy used by LLM serving systems is the wrong tool for multi-agent workloads, because an agent's future reuse depends on workflow structure, not recency. The paper proposes to abstract each workflow as an Agent Step Graph, compute a steps-to-execution value for every agent, and use those values to decide which KV cache nodes to keep, evict, or reload. On top of this, KVFlow prefetches the next agents' cached KV tensors from CPU to GPU while the current agent is still generating, and skips not-yet-loaded requests when scheduling. The claimed result is that cache misses and CPU-GPU transfer stalls disappear, yielding up to 1.83x speedup for single workflows with long prompts and up to 2.19x under high concurrency compared with SGLang's hierarchical radix cache.

What carries the argument

The carrying mechanism is the Agent Step Graph, a graph whose nodes are agent invocations and whose edges carry step aggregation functions that compute a steps-to-execution value for each agent: how many workflow steps until the agent is expected to run. These values are mapped onto the radix cache tree at the last node of each agent's fixed prompt and propagated upward with a conservative minimum priority, so shared prefixes survive as long as any near-future agent needs them. The second carrying mechanism is overlapped prefetching: CPU memory acts as a second cache, background threads load the next step's KV tensors over PCIe while the GPU generates the current agent's output, and the scheduler skips nodes marked as loading until the transfer completes. Together the two mechanisms replace reactive load-on-miss with scheduled load-before-use.

What would settle it

Run KVFlow on a workflow whose next agent is chosen by the current agent's LLM output, with fixed prompt ends unmarked, and compare its speedup against a variant given an oracle of the next step; if cache-miss stalls remain, the claim that prefetching fully overlaps CPU-GPU transfer does not hold in data-dependent workflows.

Watch

Extended reading notes

Core claim

The paper's central claim is that if an LLM serving system knows the execution schedule of an agentic workflow, it can make prefix caching nearly optimal: eviction should be guided by expected next-use distance rather than by last access, and cache reloads should be initiated proactively and overlapped with GPU computation. KVFlow embodies this by assigning each agent a steps-to-execution value derived from the Agent Step Graph, propagating priorities through the radix cache tree so shared prefix nodes are kept as long as any near-future agent needs them, and running background threads that prefetch the fixed-prompt KV tensors of all agents that may run next. Combining prefetch with status-aware scheduling, which skips requests whose KV tensors are still loading and runs other ready requests in the meantime, is what lets the system hide PCIe transfer latency completely. If correct, the cache-miss stalls that dominate latency in prompt-heavy agentic workflows are eliminated, and the claimed speedups follow as a direct consequence.

Load-bearing premise

The whole benefit rests on the server being told, correctly and in advance, which agents may run next and which tokens of each prompt are fixed; if workflow execution branches on LLM output, creates agents dynamically, or changes prompts between invocations, the predicted steps-to-execution and prefetch decisions can be wrong, and the claimed elimination of cache misses does not follow.

Editorial extensions

If this is right

  • For agentic workflows with fixed, long agent prompts and limited GPU memory, KVFlow should reduce end-to-end latency below both GPU-only LRU caching and SGLang's hierarchical radix cache with reactive loading.
  • Larger fixed prompts make the relative gain larger, since the cost of a cache miss grows with prefix length; as output length grows, decode time dominates and the benefit shrinks.
  • Under high concurrency, the status-aware scheduler keeps the GPU busy with ready requests while other requests' KV tensors are loading, recovering up to 2.19x over LRU-based hierarchical caching in the paper's measurements.
  • Because KVFlow only changes cache management and scheduling, not model weights, prompts, or decoding logic, semantic correctness of the generated outputs is preserved by construction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same steps-to-execution machinery could extend to other workloads with predictable reuse, such as retrieval-augmented generation with fixed knowledge-base prefixes or tool-calling loops, not just named-agent workflows.
  • The prefetch benefit assumes PCIe transfer is cheaper than recomputation; on interconnects with much higher transfer cost, the optimal policy might shift from load-before-use to recompute-on-use, a regime the paper does not explore.
  • A stress test worth running is data-dependent branching: when the next agent is chosen by LLM output, comparing KVFlow against an oracle that knows the true next step would quantify the remaining gap caused by imperfect prediction.
  • If workflow frameworks exposed branch probabilities, the prefetch limit could be prioritized rather than treating all possible next agents equally, cutting transfer waste on highly branching workflows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. KVFlow proposes a workflow-aware KV cache management layer for LLM serving in multi-agent workflows. It abstracts the agent execution schedule as an Agent Step Graph, computes a steps-to-execution value for each agent using dependency-specific aggregation functions, and uses these values to assign eviction priorities at the radix-tree node level. It also introduces a proactive prefetching mechanism that loads KV tensors from CPU to GPU ahead of scheduled agent invocations, together with status-aware scheduling that skips requests whose caches are still loading. The paper reports speedups over SGLang and SGLang with hierarchical radix cache of up to 1.83x for single-workflow latency and up to 2.19x for high-concurrency settings, based on a static 10-agent sequential workflow, synthetic concurrent workloads, and PEER-style workflows. The central claim is that this combination eliminates cache misses and fully overlaps GPU computation with prefetching.

Significance. If the claims are established, KVFlow would be a useful contribution: the observation that LRU eviction fails in cyclic multi-agent workflows is well motivated, and the steps-to-execution abstraction is a clean way to inject workflow knowledge into cache policy. The node-level priority propagation over shared prefixes is also sensible. However, the significance is currently tempered by the absence of reproducible artifacts, the reliance on point estimates without variance, the synthetic nature of the workloads, and the lack of any evaluation of the dynamic branching and dynamic agent creation cases that the paper's general claims require. The paper's strengths are the clarity of the abstraction and the consistency of the speedups on the static schedules tested; these strengths do not yet support the stronger 'eliminates cache misses' and 'fully overlaps' statements.

major comments (5)
  1. [§3.2, §4.1] The central claim that KVFlow 'effectively eliminates cache misses and fully overlaps GPU computation with prefetching' is not justified by the described mechanism. The paper itself acknowledges that when the current agent's execution time is shorter than the prefetch duration, generation may still be blocked by incomplete KV loading; status-aware scheduling can only hide this delay if there is some other ready request to run. In the single-workflow experiments of §4.1, there is no other request, so full overlap cannot hold in general. This claim should be replaced with a measurable, bounded statement, or the evaluation should report stall time and demonstrate that prefetching completes before the next agent is scheduled in the tested configurations.
  2. [§3.1, §3.2, §4] The mechanism presumes that the Agent Step Graph and the steps-to-execution metadata embedded in each request identify the agents that may run next and the last node of each fixed prompt. For conditional branching determined by LLM output, the paper only says that the system 'conservatively prefetches all agents that may be executed next based on the Step Graph, within a predefined limit'; it never specifies how the possible-next set is obtained when the next agent is selected by the model at runtime, nor how agents created dynamically, which are absent from the original graph, receive an eviction priority or a prefetch entry. None of the experiments in Figures 5-8 include data-dependent branching or dynamic agent creation, so the general claim that cache misses are eliminated is unsupported outside statically known schedules.
  3. [§4.2] The high-concurrency evaluation is restricted by design to 'a proper concurrency that the GPU can accommodate without exhausting memory for prefix caching,' and the paper explicitly excludes regimes where all memory is consumed by active requests. This is exactly the memory-pressure regime that motivates workflow-aware eviction, and it is also the regime where the HiCache baseline collapses (e.g., 0.57x of SGLang at 1024 fixed tokens with 64 workflows). The reported up-to-2.19x speedup is therefore conditional on favorable memory conditions; the paper should either report performance across a sweep of concurrency and memory-pressure levels or clearly state that the contribution applies only when reusable prefix caches can be retained.
  4. [§4.1, §4.2] The experimental evidence consists of point estimates without error bars, repetitions, or seeds, despite random synthetic prompt generation and nondeterministic scheduling under concurrency; no direct measurement of cache-miss rate, prefetch hit rate, or stall time is reported. As a result, the paper does not verify the causal mechanism behind the speedups. I would ask for multiple runs with error bars, a direct miss-rate and stall-time breakdown, and, ideally, release of the implementation and the exact values of the concurrent prefetch limit and memory trigger so that the experiments can be reproduced.
  5. [§3.3] Identification of the fixed-prompt prefix is a load-bearing precondition: eviction priorities are assigned to the last KV node of the agent's fixed prompt, and prefetching targets only the fixed part. The two proposed methods are a user-supplied marker and a cache-hit-history heuristic. The heuristic can misclassify a repeated dynamic suffix as part of the fixed prefix, and the marker is optional, so in the default mode the scheme's correctness depends on an unvalidated heuristic. The paper does not measure the accuracy of this boundary detection or the sensitivity of the speedups to misclassification. This should be evaluated, or the scheme should be presented as requiring the explicit marker.
minor comments (5)
  1. [Figure 2(b)] The y-axis labels 'Prefill Latency' and 'KV Cache Transmission Time' are presented without a clear legend or separate panels; please clarify which series corresponds to which quantity.
  2. [Figure 6] The axis labels '512/20-T ask' and '1024/10-T ask' appear to be rendering artifacts of 'Task'; please fix the labels.
  3. [§4.1] The paper says the benchmark uses a 10-agent sequential workflow, but Figure 5 reports aggregate fixed/dynamic/output lengths; please state how these lengths are distributed across the agents and whether every agent has the same prompt composition.
  4. [§3.3] The paper states that embedding workflow metadata into HTTP requests has negligible overhead, but no measurement of the metadata size or per-request overhead is given; please add this information.
  5. [§5] The related-work section cites Autellix and ParrotServe as complementary but does not compare against them experimentally or analytically; a brief positioning discussion of how their scheduling decisions interact with KVFlow's cache management would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: KVFlow's mechanisms are implemented system optimizations evaluated against external baselines, and no claimed speedup reduces to a fitted parameter or a self-citation chain.

full rationale

The paper's derivation chain is not circular. The steps-to-execution values are computed from the workflow graph using stated aggregation functions (max/min plus one) and are independent of the measured latency or speedup; they are not fitted to the reported results. The claimed benefits of workflow-aware eviction and proactive prefetching are evaluated by comparing against fixed baselines (SGLang and SGLang with HiCache) on synthetic and PEER-style workflows, so the 1.83x and 2.19x speedups are empirical outcomes rather than quantities defined into existence. The paper contains no author self-citations invoked as load-bearing evidence; references to prior systems such as SGLang, vLLM, and PEER are external and independent. The main limitation—that the mechanism assumes accurate step-graph information and fixed identifiable prompt prefixes—is a robustness or generalization concern about dynamic branching and runtime-created agents, not a circularity. The 'conservatively prefetches all agents that may be executed next' design is an explicitly stated heuristic, and the cache-hit-history heuristic for identifying fixed prefixes is presented as an implementation alternative rather than as the source of the claimed speedups. Therefore, no step reduces by construction to its own inputs, and the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central mechanism relies on the availability of accurate workflow schedule information and on the assumption that cached fixed prompts dominate cost; these are reasonable for the targeted agent frameworks but unverified beyond synthetic workloads. No new physical entities are introduced; the Agent Step Graph and status states are software abstractions, not entities with independent evidence.

free parameters (3)
  • concurrent prefetch limit = unspecified
    In §3.2, 'within a predefined limit on the number of concurrent prefetches'; the limit is neither given nor swept, and it bounds PCIe parallelism and queueing, so it shapes the reported overlap gains.
  • prefetch/eviction memory trigger = unspecified
    In §3.3, 'trigger prefetching if the evictable GPU memory is large enough'; the threshold is not defined, so the behavior under memory pressure is under-specified.
  • fixed-prefix detection window = unspecified
    In §3.3, the heuristic treats a 'consistently hit prefix' as fixed; the consistency window and hit threshold are not specified, which affects where eviction priorities are placed on the cache tree.
assumptions (4)
  • domain assumption Agent prompts contain a stable fixed part that is reused across invocations
    Used throughout §2 and §3 to justify caching and priority assignment; true for many agent templates but not guaranteed for agents whose instructions are rewritten each call.
  • domain assumption Steps-to-execution values computed from the Agent Step Graph reflect future invocation order
    §3.1 derives priorities from these values; correctness of eviction and prefetch decisions depends on this premise.
  • domain assumption CPU-GPU KV transfer is faster than recomputation and overlaps with GPU forward passes
    Supported by Figure 2(b) on two testbeds, but the claimed full overlap also requires that prefetch traffic does not contend with generation, which is only partially true on shared PCIe.
  • ad hoc to paper Each sgl.function maps to one agent and workflow metadata can be injected into HTTP requests with negligible overhead
    §3.3 describes a just-in-time substitution; this binds the design to SGLang's frontend and assumes the metadata path does not add latency or break existing workflows.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows." pith.science (2026). https://pith.science/paper/H35TVOAN

@misc{pith2026250707400,
  author       = {Pith},
  title        = {Pith review of: KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H35TVOAN}},
  note         = {Machine review of arXiv:2507.07400}
}
abstract

Large language model (LLM) based agentic workflows have become a popular paradigm for coordinating multiple specialized agents to solve complex tasks. To improve serving efficiency, existing LLM systems employ prefix caching to reuse key-value (KV) tensors corresponding to agents' fixed prompts, thereby avoiding redundant computation across repeated invocations. However, current systems typically evict KV caches using a Least Recently Used (LRU) policy, which fails to anticipate future agent usage and often discards KV caches shortly before their reuse. This leads to frequent cache misses and substantial recomputation or swapping overhead. We present KVFlow, a workflow-aware KV cache management framework tailored for agentic workloads. KVFlow abstracts the agent execution schedule as an Agent Step Graph and assigns each agent a steps-to-execution value that estimates its temporal proximity to future activation. These values guide a fine-grained eviction policy at the KV node level, allowing KVFlow to preserve entries likely to be reused and efficiently manage shared prefixes in tree-structured caches. Moreover, KVFlow introduces a fully overlapped KV prefetching mechanism, which proactively loads required tensors from CPU to GPU in background threads for agents scheduled in the next step, thereby avoiding cache miss stalls during generation. Compared to SGLang with hierarchical radix cache, KVFlow achieves up to 1.83$\times$ speedup for single workflows with large prompts, and up to 2.19$\times$ speedup for scenarios with many concurrent workflows.

Figures

Figures reproduced from arXiv: 2507.07400 by the authors.

Figure 1
Figure 1. A cyclic agentic workflow abstraction consisting of four agents, Planner, Executor, Ex [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. KV cache characteristics with varying context lengths. (a) KV cache size grows linearly [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the workflow-aware eviction policy. (a) Each agentic workflow is abstracted [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Illustration of overlapped KV prefetching. Compared to reactive loading, KVFlow combines [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Speedup over SGLang (GPU-only cache) for a 10-agent sequential workflow. Horizontal [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: High-concurrency workflow performance comparison under different fixed-prompt/concurrency settings on an H100. 0 200 400 600 Token Number 0.000 0.001 0.002 0.003 0.004 0.005 Density Fixed Dynamic Output [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework

    cs.DC 2026-07 conditional novelty 7.0 of 10

    Agent serving faces non-inference bottlenecks (up to 48% of latency), a 4.4x serving-capacity loss from long context, and 4.9x cost amplification from snapshot-based sandbox suspension.

  2. Streaming Communication in Multi-Agent Reasoning

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    StreamMA introduces streaming communication in multi-agent reasoning to reduce latency via pipelining and improve effectiveness by leveraging reliable early steps, with closed-form analysis and a step-level scaling law.

  3. VineLM: Trie-Based Fine-Grained Control for Agentic Workflows

    cs.DC 2026-04 conditional novelty 6.0 of 10

    VineLM uses an annotated execution trie plus cascade profiling and online re-rooting to select models per stage invocation in agentic workflows, improving the cost-latency-accuracy frontier by up to 18% accuracy at fi...

  4. Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving

    cs.DC 2026-03 conditional novelty 6.0 of 10

    Pattern-aware speculative tool execution cuts agent end-to-end latency by roughly half and observed tool latency by about 1.8× by overlapping predicted tools with LLM generation.

  5. Workload-Aware Caching for Multi-Agent Systems

    cs.AI 2026-06 conditional novelty 5.0 of 10

    A workload-aware score combining recomputation cost, DAG dependency count, and agent invocation frequency outperforms standard eviction policies for multi-agent task-result caching.

Reference graph

Works this paper leans on

40 extracted references · 12 canonical work pages · cited by 5 Pith papers

  1. [1]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  2. [2]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  3. [3]

    Metagpt: Meta programming for multi-agent collaborative framework

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 3(4):6, 2023

    arXiv 2023

  4. [4]

    Camel: Communicative agents for" mind" exploration of large language model society

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991–52008, 2023

    2023

  5. [5]

    PEER: Expertizing Domain-Specific Tasks with a Multi-Agent Framework and Tuning Methods

    Yiying Wang, Xiaojing Li, Binzhu Wang, Yueyang Zhou, Yingru Lin, Han Ji, Hong Chen, Jinshi Zhang, Fei Yu, Zewei Zhao, et al. Peer: Expertizing domain-specific tasks with a multi-agent framework and tuning methods. arXiv preprint arXiv:2407.06985, 2024

    work page Pith review arXiv 2024

  6. [6]

    Autogen: Enabling next-gen llm applications via multi-agent conversation

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023

    arXiv 2023

  7. [7]

    Gptswarm: Language agents as optimizable graphs

    Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning, 2024

    work page 2024

  8. [8]

    Aflow: Automating agentic workflow generation

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762, 2024

    arXiv 2024

Show all 40 references
  1. [9]

    Very large-scale multi-agent simulation in agentscope

    Xuchen Pan, Dawei Gao, Yuexiang Xie, Yushuo Chen, Zhewei Wei, Yaliang Li, Bolin Ding, Ji-Rong Wen, and Jingren Zhou. Very large-scale multi-agent simulation in agentscope. arXiv preprint arXiv:2407.17789, 2024

    2024 arXiv

  2. [10]

    Cognify: Supercharging gen-ai workflows with hierarchical autotuning

    Zijian He, Reyna Abhyankar, Vikranth Srivatsa, and Yiying Zhang. Cognify: Supercharging gen-ai workflows with hierarchical autotuning. arXiv preprint arXiv:2502.08056, 2025

    2025 arXiv

  3. [11]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  4. [12]

    Gonzalez, Clark W

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. Advances in Neural Information ...

    2024

  5. [13]

    TensorRT-LLM

    NVIDIA. TensorRT-LLM. https://github.com/NVIDIA/TensorRT-LLM, 2025

    2025

  6. [14]

    Automatic Prefix Caching

    vLLM Team. Automatic Prefix Caching. https://docs.vllm.ai/en/latest/features/ automatic_prefix_caching.html, 2025. 10

    2025

  7. [15]

    Ragcache: Efficient knowledge caching for retrieval-augmented generation

    Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. Ragcache: Efficient knowledge caching for retrieval-augmented generation. arXiv preprint arXiv:2404.12457, 2024

    2024 arXiv

  8. [16]

    {Cost-Efficient} large language model serving for multi- turn conversations with {CachedAttention}

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. {Cost-Efficient} large language model serving for multi- turn conversations with {CachedAttention}. In 2024 USENIX Annual Technical Conference (USENIX ATC 24),...

    2024

  9. [17]

    Generative agents: Interactive simulacra of human behavior

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed- ings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023

    2023

  10. [18]

    A survey on large language model based autonomous agents

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024

    2024

  11. [19]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023

    2023 arXiv

  12. [20]

    Code generation with alphacodium: From prompt engineering to flow engineering

    Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500, 2024

    2024 arXiv

  13. [21]

    Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi- persona self-collaboration

    Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi- persona self-collaboration. arXiv preprint arXiv:2307.05300, 2023

    2023 arXiv

  14. [22]

    Mage: A multi- agent engine for automated rtl code generation

    Yujie Zhao, Hejia Zhang, Hanxian Huang, Zhongming Yu, and Jishen Zhao. Mage: A multi- agent engine for automated rtl code generation. arXiv preprint arXiv:2412.07822, 2024

    2024 arXiv

  15. [23]

    Chatdev: Communicative agents for software development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023

    2023 arXiv

  16. [24]

    Improv- ing factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2023

    2023

  17. [25]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    2023 arXiv

  18. [26]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR, 2023

    2023

  19. [27]

    Prompt lookup decoding, November 2023

    Apoorv Saxena. Prompt lookup decoding, November 2023

    2023

  20. [28]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv, 2023

    2023

  21. [29]

    Efficiently serving llm reasoning programs with certaindex

    Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang. Efficiently serving llm reasoning programs with certaindex. arXiv preprint arXiv:2412.20993, 2024

    2024 arXiv

  22. [30]

    Orca: A distributed serving system for {Transformer-Based} generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, 2022

    2022

  23. [31]

    Fast distributed inference serving for large language models

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920, 2023. 11

    2023 arXiv

  24. [32]

    Andes: Defining and enhancing quality-of-experience in llm-based text streaming services

    Jiachen Liu, Jae-Won Chung, Zhiyu Wu, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury. Andes: Defining and enhancing quality-of-experience in llm-based text streaming services. arXiv preprint arXiv:2404.16283, 2024

    2024 arXiv

  25. [33]

    Stateful large language model serving with pensieve

    Lingfan Yu, Jinkun Lin, and Jinyang Li. Stateful large language model serving with pensieve. In Proceedings of the Twentieth European Conference on Computer Systems, pages 144–158, 2025

    2025

  26. [34]

    Infercept: Efficient intercept support for augmented large language model inference

    Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang. Infercept: Efficient intercept support for augmented large language model inference. arXiv preprint arXiv:2402.01869, 2024

    2024 arXiv

  27. [35]

    Autellix: An efficient serving engine for llm agents as general programs

    Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E Gonzalez, et al. Autellix: An efficient serving engine for llm agents as general programs. arXiv preprint arXiv:2502.13965, 2025

    2025 arXiv

  28. [36]

    Parrot: Efficient serving of {LLM-based} applications with semantic variable

    Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. Parrot: Efficient serving of {LLM-based} applications with semantic variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 929–945, 2024

    2024

  29. [37]

    LangGraph

    LangChain. LangGraph. https://github.com/langchain-ai/langgraph, 2025

    2025

  30. [38]

    Building effective agents

    Anthropic. Building effective agents. https://www.anthropic.com/engineering/ building-effective-agents, 2024

    2024

  31. [39]

    Agentscope: A flexible yet robust multi-agent platform

    Dawei Gao, Zitao Li, Xuchen Pan, Weirui Kuang, Zhijian Ma, Bingchen Qian, Fei Wei, Wenhao Zhang, Yuexiang Xie, Daoyuan Chen, et al. Agentscope: A flexible yet robust multi-agent platform. arXiv preprint arXiv:2402.14034, 2024

    2024 arXiv

  32. [40]

    Cut the crap: An economical communication pipeline for llm-based multi-agent systems

    Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for llm-based multi-agent systems. arXiv preprint arXiv:2410.02506, 2024. 12

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.