Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that by moving newly appended prefix tokens to the end of the prompt, EFIM lets LLM servers reuse cached states for both prefix and suffix in multi-round infilling, cutting latency by 52% and raising throughput by 98%.

desk verdict EFIM's prompt transformation is a genuinely new and useful idea with real measured gains, but the headline 52%/98% serving numbers are only proven for prefix-append workloads; the paper's own routing logic does not deliver the claimed suffix-head benefit. read the letter →

arxiv 2505.21889 v2 pith:7JBIPLEP submitted 2025-05-28 cs.CL

Tianyu Guo , Hande Dong , Yichong Leng , Feng Liu , Cheater Lin , Nong Xiao , Xianwei Zhang This is my paper · ORCID
classification cs.CL
keywords fill-in-the-middleKVcachereusesubtokengenerationfragmenttokenizationLLMservinginfillinglatencyoptimizationcodecompletion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard fill-in-the-middle prompt used for infilling tasks is the reason LLM serving is slow: when a user appends tokens to the prefix across interaction rounds, the suffix's cached attention states must be recomputed. The proposed fix, EFIM, moves each round's appended tokens to the very end of the prompt while keeping the original prefix and suffix fixed, so servers can reuse cached key-value states for both. Because this makes the model continue generation inside a partial word, the paper adds a fragment tokenization training method that splits training text into many pieces before tokenization, teaching models to complete partial words anywhere. On two code models, the combined scheme is reported to cut average latency by 52% and raise throughput by 98% while matching standard FIM infilling quality on adapted benchmarks. If the gains hold in real deployment, infilling services can roughly double per-GPU request throughput without sacrificing code completion quality.

What carries the argument

The load-bearing object is the prompt reordering from "<P>prefix+inc<S>suffix<M>" to "<P>prefix<S>suffix<M>inc". This keeps both the prefix and suffix sequences identical to the previous turn, so a per-user session pool can point new requests at cached KV entries for both parts and only compute attention for the increment and the generated middle. The second mechanism is fragment tokenization: training documents are randomly split into multiple segments, with segment length uniform in [1,200], and each segment is tokenized separately before FIM assembly. That scatters subtoken boundaries through the training corpus rather than only next to FIM special tokens, which is what gives the model the ability to complete a partial token at the end of an EFIM prompt.

What would settle it

Log the edit operations in a real multi-turn infilling service, classify each turn as prefix-tail growth, suffix-head growth, both, or an edit inside the prefix, middle, or suffix, and count the fraction of turns where EFIM's session invariant holds. If the fraction is far below what is needed to sustain the paper's low-concurrency reuse rate of about 80%, the 52%/98% serving gains will not reproduce. A direct second check is running the paper's serving experiment on a suffix-growth workload and observing EFIM's reuse rate and latency converge to PSM-level values rather than the reported EFIM numbers.

Watch

Extended reading notes

Core claim

The paper claims that the standard fill-in-the-middle prompt format itself blocks cross-request KV cache reuse: when a user appends tokens to the prefix, every token of the suffix changes its preceding context and its cached attention states become invalid. EFIM instead keeps the original prefix and suffix in fixed positions and appends the increment at the very end, forming a prompt like "<P>prefix<S>suffix<M>inc", so both prefix and suffix caches survive across turns. Because the position after the middle token is normally a token boundary, this exposes a hidden failure mode: models cannot complete a partially tokenized word there. The paper's fragment tokenization training, which randomly splits documents into multiple segments and tokenizes each segment separately, produces subtoken boundaries throughout the training data and gives the model a universal subtoken-completion ability. The result is an infilling model that matches the original fill-in-the-middle capability on the adapted benchmarks while serving with far less prefill computation.

Load-bearing premise

The reported gains rest on the untested workload assumption that most real infilling edits append to the tail of an unchanged prefix or the head of an unchanged suffix; if users instead edit the middle or replace the suffix, EFIM's prefix and suffix caches are invalidated and it degrades to standard fill-in-the-middle serving.

Editorial extensions

If this is right

  • Any serving engine that already reuses prefix KV cache can adopt EFIM by swapping the prompt format and adding a per-user session pool; no model architecture change is needed.
  • Continued pretraining with fragment tokenization for about a week on 64 GPUs is enough to restore subtoken generation, and the extra cost disappears if the method is used from the start of pretraining.
  • A 98% throughput gain implies serving cost roughly halves; the paper estimates this can repay the training overhead within a day when serving load is comparable to training cost.
  • KV cache capacity, not prompt format, becomes the binding constraint at high concurrency: reuse rate falls once completed sessions exceed GPU memory, so EFIM's latency advantage narrows as user count grows.
  • EFIM's quality is workload-dependent in a specific way: benchmarks where the appended prefix ends on a complete token show no loss, and fragment tokenization restores the random-span subtoken cases to FIM-level quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If real editing logs contain many middle rewrites or suffix replacements, the measured 52%/98% gains are upper bounds for a workload dominated by prefix-tail appends; an adaptive policy that detects edit direction per session is the obvious next step.
  • Fragment tokenization is a general capability: any system needing token healing, insertions, or byte-level edits could adopt it, not just infilling servers.
  • Combining EFIM with cheaper secondary KV cache storage should extend its reuse advantage to larger concurrency, where the paper shows reuse rate drops because completed sessions exceed GPU memory.
  • A direct test of the workload assumption would run the serving benchmark with suffix-head growth instead of prefix-tail growth; the prediction is that EFIM falls back to ordinary PSM-level reuse and the reported gains disappear.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes EFIM, a prompt-format transformation for FIM-based LLM infilling that moves an incremental prefix extension to the end of the prompt so that the KV cache of both prefix and suffix can be reused across requests. Because this creates subtoken-completion demands, the paper introduces a fragment-tokenization continued-pretraining method. Experiments on Deepseek-coder-6.7B and Llama3.1-8B evaluate infilling quality on adapted HumanEval and CCEval and report a 52% latency reduction and 98% throughput improvement versus FIM with reuse in a small serving simulation.

Significance. If validated, the method is useful: it is architecture-agnostic, requires only a prompt-level change, and the fragment-tokenization training addresses a real weakness of EFIM. The public code release and the capability-recovery results on both models are strengths. The main significance is as a systems optimization for infilling workloads; however, the headline efficiency gain is conditional on an unquantified workload assumption and a single synthetic serving setup.

major comments (3)
  1. [§2.2 and §3.1] The paper's motivating claim that 'most of modifying behaviors involve appending tokens to the tail of the prefix or the head of the suffix' is presented as 'according to our statistics' but no statistics, methodology, or reference are provided. This assumption is load-bearing: the EFIM transformation in §3.1 is applied only when the prefix grows (step ❹), while suffix-head growth is routed to PSM (step ❺). The reported 52%/98% serving gains therefore apply only to the prefix-growth branch, not to the full set of modifications the paper says dominates real workloads. Please supply the underlying telemetry or, failing that, explicitly state the efficiency results as conditional on prefix-tail growth.
  2. [§3.1 and Figure 3] Figure 3 claims EFIM achieves the most KV cache reuse in both prefix-tail and suffix-head growth scenes, but the routing logic in §3.1 does not implement EFIM for suffix-head growth: step ❺ sends such requests in PSM format, reusing only the prefix. Moreover, it is not shown that an EFIM-style prompt can be constructed for suffix-head growth without moving the suffix increment in front of the autoregressive generation position, which would change what the model is predicting. As a result, the 'both scenes' claim is unsupported and the design currently delivers the advertised advantage for only one edit pattern.
  3. [§4.2 and Figures 11–12] The serving evaluation is a single small simulation (two models, 16 users, 5 rounds) that only exercises the prefix-append scenario. There are no repeated runs, error bars, or confidence intervals, and the workload does not reflect the mixed edit types claimed in §2.2. Since the latency and throughput gains are driven by the KV reuse rate, which depends on the edit distribution, the paper should report variance across runs and at least a sensitivity analysis over edit mixes (e.g., varying the fraction of suffix-head/middle edits) before the 52%/98% headline is presented as a general result.
minor comments (5)
  1. [§5.2] The abstract and §5.2 should state explicitly that the 52% latency reduction and 98% throughput increase are relative to FIM (PSM with reuse), not to the no-reuse Baseline.
  2. [§3.2] The fragment segment length range [1,200] and the uniform distribution are hyperparameters of the training method; a sensitivity study would help establish robustness.
  3. [Table 1] The pass@1 and CCEval metrics are reported without variance or sample counts; given that one sample is generated per problem, reporting standard errors or multiple seeds would strengthen the capability claim.
  4. [§5.3] The cost comparison mixes H100 and H800 GPU hours without normalization; the reader should be told what conversion or assumption is being used.
  5. [Throughout] Fix typos such as 'sutokens' in §2.3 and inconsistent casing of 'DeepSeek-Coder'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: EFIM's serving gains are structurally derived and measured on held-out benchmarks; the workload assumption in Section 3.1 is an external-validity limitation, not circular reasoning.

full rationale

The paper's central derivation is not circular. EFIM's KV-cache advantage follows by construction from the prompt transformation: replacing '<P>prefix+inc<S>suffix<M>' with '<P>prefix<S>suffix<M>inc' keeps both prefix and suffix byte-identical across turns, so the session pool can reuse both KV caches; this is a structural property, not a fitted quantity. The subtoken-generation failure is identified in Section 2.3, and fragment tokenization is a training-data remedy evaluated on held-out benchmarks in Section 4.1, not a post-hoc fit to the reported speedup. The 52% latency reduction and 98% throughput increase are measured in the Section 4.2 serving experiment, comparing Baseline, FIM, and EFIM under the stated prefix-extension scenario. The only notable weakness is that Section 3.1 routes suffix-head growth back to PSM, and the workload assertion 'most of modifying behaviors involve appending tokens to the tail of the prefix or the head of the suffix' is unquantified and uncited, so the headline gains may not generalize to mixed workloads. That is a generalization and external-validity limitation, not circularity: the derivation chain does not reduce to its own inputs or to self-citations. No load-bearing self-citation chain, imported uniqueness theorem, ansatz-smuggling citation, or fitted-input-called-prediction pattern is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The key dependencies are domain assumptions about the workload pattern and about how a transformer model can generalize from the fragment tokenization training distribution to the EFIM prompt format. One hand-chosen hyperparameter, the segment length range [1,200], is a free parameter of the training method.

free parameters (1)
  • fragment segment length range = [1,200] tokens
    The random segment length in fragment tokenization follows a uniform distribution [1,200] (Section 3.2, Figure 8 caption). This is a hand-chosen hyperparameter that controls the density of subtoken boundaries in the training data and likely affects the degree of subtoken generation capability learned.
assumptions (4)
  • domain assumption Cross-request KV cache reuse in the serving engine (vLLM) preserves exact attention semantics when prompt prefixes match.
    Standard property of vLLM prefix caching used in Section 4.2; the paper relies on it to attribute the latency and throughput gains.
  • domain assumption The EFIM prompt transformation (moving the incremental prefix after the <M> token) preserves the semantic content of the infilling task for a model with subtoken generation ability.
    This is the central design premise of Section 3.1. The paper does not prove that the model can treat input after <M> as a prefix continuation; it is only validated empirically on the adapted benchmarks.
  • domain assumption Real-world infilling interactions are dominated by appending tokens to the prefix tail or suffix head.
    Stated in Section 3.1 without supporting statistics or a citation. This assumption determines which requests can use EFIM and underpins the reported 52%/98% efficiency gains as being representative of production workloads.
  • domain assumption Fragment tokenization continued pretraining on StarCoderData does not degrade general infilling ability relative to standard FIM training.
    Supported by the eLLM w/FIM vs oLLM w/FIM comparisons in Table 1 on the two code benchmarks, but the paper does not test other model capabilities, so the generality of this assumption is unestablished.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse." pith.science (2026). https://pith.science/paper/7JBIPLEP

@misc{pith2026250521889,
  author       = {Pith},
  title        = {Pith review of: EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7JBIPLEP}},
  note         = {Machine review of arXiv:2505.21889}
}
read the original abstract

Large language models (LLMs) are often used for infilling tasks, which involve predicting or generating missing information in a given text. These tasks typically require multiple interactions with similar context. To reduce the computation of repeated historical tokens, cross-request key-value (KV) cache reuse, a technique that stores and reuses intermediate computations, has become a crucial method in multi-round interactive services. However, in infilling tasks, the KV cache reuse is often hindered by the structure of the prompt format, which typically consists of a prefix and suffix relative to the insertion point. Specifically, the KV cache of the prefix or suffix part is frequently invalidated as the other part (suffix or prefix) is incrementally generated. To address the issue, we propose EFIM, a transformed prompt format of FIM to unleash the performance potential of KV cache reuse. Although the transformed prompt can solve the inefficiency, it exposes subtoken generation problems in current LLMs, where they have difficulty generating partial words accurately. Therefore, we introduce a fragment tokenization training method which splits text into multiple fragments before tokenization during data processing. Experiments on two representative LLMs show that LLM serving with EFIM can lower the latency by 52% and improve the throughput by 98% while maintaining the original infilling capability. EFIM's source code is publicly available at https://github.com/gty111/EFIM.

Figures

Figures reproduced from arXiv: 2505.21889 by the authors.

Figure 1
Figure 1. A python code snippet where a programmer wants to insert code inside [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of PSM and SPM. <P> follows prefix part, <S> follows suffix part, <M> follows middle part and <E> marks the end of infilling span. Current decoder-based autoregressive (AR) language models [2, 4, 7, 14] are capable of generating text from left to right. However, they struggle with infilling tasks, where the model is required to generate text at a specific location within a snippet, conditioned on both a p… view at source ↗
Figure 3
Figure 3. Reusable part between PSM, SPM and EFIM when the growth (inc) happens either at the prefix tail or at the suffix head. The reusable part includes the content before inc. multi-round interactions with LLMs, especially when dealing with long contexts. According to our statistics from online infilling services, most of modifying be￾haviors involve appending tokens to the tail of the prefix or the head of the suffix [P… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Elapsed time breakdown of prefill and decode stage for infilling serving [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Subtoken generation ability between different prompt formats considering [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Overall diagram of design with EFIM. Our proposed design with EFIM consists of two key parts as illustrated in Fig￾ure 6. The first part operates between the user and the LLM to seamlessly and automatically convert the prompt format from FIM to EFIM. This transforma￾ti…
Figure 7
Figure 7. Figure 7: The prompt transformation process from FIM to [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Data processing diagram between FIM (left) and fragment tokenization [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Comparison between FIM (left) and fragment tokenization (right) data [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Prompt creation procedure and prompt format between FIM and [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Overall inference performance on average latency, request throughput, [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Variation of latency (above) and KV cache reuse rate (below) as the [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents

    cs.PL 2026-06 unverdicted novelty 6.0 of 10

    FCGraft synthesizes code policies for embodied agents by grafting KV caches from a library of validated functions, claiming 18.31% higher success rate and 2.3x faster synthesis than prompt-level caching.

Reference graph

Works this paper leans on

39 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    In: SC (2022).https://doi.org/10.1109/SC41404.2022.00051

    Aminabadi, R.Y., Rajbhandari, S., Awan, A.A., et al.: Deepspeed-inference: En- abling efficient inference of transformer models at unprecedented scale. In: SC (2022).https://doi.org/10.1109/SC41404.2022.00051

    arXiv 2022

  2. [3]

    arXiv (2022).https://doi.org/10.48550/ARXIV.2207.14255

    Bavarian, M., Jun, H., Tezak, N., et al.: Efficient training of language models to fill in the middle. arXiv (2022).https://doi.org/10.48550/ARXIV.2207.14255

    arXiv doi:10.48550/arxiv.2207.14255 2022

  3. [4]

    In: NeurIPS (2020)

    Brown, T.B., Mann, B., Ryder, N., et al.: Language models are few-shot learners. In: NeurIPS (2020)

    work page 2020

  4. [5]

    https://openai.com/index/introducing-canvas/

    OpenAI canvas. https://openai.com/index/introducing-canvas/

    work page

  5. [6]

    arXiv (2021)

    Chen, M., Tworek, J., Jun, H., et al.: Evaluating large language models trained on code. arXiv (2021)

    work page 2021

  6. [7]

    Chowdhery, A., Narang, S., Devlin, J., et al.: Palm: Scaling language modeling with pathways. J. Mach. Learn. Res. (2023)

    work page 2023

  7. [8]

    https://docs.aws.amazon.com/codewhisperer/

    Amazon CodeWhisper. https://docs.aws.amazon.com/codewhisperer/

    work page

  8. [9]

    https://github.com/features/copilot

    GitHub Copilot. https://github.com/features/copilot

    work page

Show all 39 references
  1. [10]

    In: ICLR (2024)

    Dao, T.: Flashattention-2: Faster attention with better parallelism and work par- titioning. In: ICLR (2024)

    2024

  2. [11]

    In: NeurIPS (2022)

    Dao, T., Fu, D.Y., Ermon, S., et al.: Flashattention: Fast and memory-efficient exact attention with io-awareness. In: NeurIPS (2022)

    2022

  3. [12]

    arXiv (2024).https://doi.org/10.48550/ARXIV.2412.19437

    DeepSeek-AI, Liu, A., Feng, B., Xue, B., et al.: Deepseek-v3 technical report. arXiv (2024).https://doi.org/10.48550/ARXIV.2412.19437

    2024 doi arXiv

  4. [13]

    https://zhuanlan.zhihu.com/p/27181462601

    DeepSeek V3 serving. https://zhuanlan.zhihu.com/p/27181462601

  5. [14]

    In: NAACL-HLT (2019)

    Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (2019). https://doi.org/10.18653/V1/N19-1423

    2019 doi

  6. [15]

    In: NeurIPS (2023)

    Ding, Y., Wang, Z., Ahmad, W.U., et al.: Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. In: NeurIPS (2023)

    2023

  7. [16]

    arXiv (2024)

    Dubey, A., Jauhri, A., Pandey, A., et al.: The llama 3 herd of models. arXiv (2024). https://doi.org/10.48550/ARXIV.2407.21783

    2024 doi arXiv

  8. [17]

    In: ICLR (2023)

    Fried, D., Aghajanyan, A., Lin, J., et al.: Incoder: A generative model for code infilling and synthesis. In: ICLR (2023)

    2023

  9. [18]

    In: ATC (2024)

    Gao, B., He, Z., Sharma, P., et al.: Cost-efficient large language model serving for multi-turn conversations with cachedattention. In: ATC (2024)

    2024

  10. [19]

    In: MLSys (2024) 14 T

    Gim, I., Chen, G., Lee, S., et al.: Prompt cache: Modular attention reuse for low- latency inference. In: MLSys (2024) 14 T. Guo et al

    2024

  11. [20]

    arXiv (2024).https://doi.org/ 10.48550/ARXIV.2401.14196

    Guo, D., Zhu, Q., Yang, D., et al.: Deepseek-coder: When the large language model meets programming - the rise of code intelligence. arXiv (2024).https://doi.org/ 10.48550/ARXIV.2401.14196

    2024 doi arXiv

  12. [21]

    arXiv (2024)

    Hui, B., Yang, J., Cui, Z., et al.: Qwen2.5-coder technical report. arXiv (2024). https://doi.org/10.48550/ARXIV.2409.12186

    2024 doi arXiv

  13. [22]

    In: SOSP (2023).https://doi.org/10

    Kwon, W., Li, Z., Zhuang, S., et al.: Efficient memory management for large lan- guage model serving with pagedattention. In: SOSP (2023).https://doi.org/10. 1145/3600006.3613165

    2023

  14. [23]

    In: OSDI (2024)

    Lee, W., Lee, J., Seo, J., Sim, J.: Infinigen: Efficient generative inference of large language models with dynamic KV cache management. In: OSDI (2024)

    2024

  15. [24]

    Li, R., Allal, L.B., Zi, Y., et al.: Starcoder: may the source be with you! TMLR (2023)

    2023

  16. [25]

    https://huggingface.co/meta-llama/Llama-3.1-8B

    Llama3.1 model card. https://huggingface.co/meta-llama/Llama-3.1-8B

  17. [26]

    arXiv (2024).https://doi.org/10.48550/ARXIV.2402.19173

    Lozhkov, A., Li, R., Allal, L.B., et al.: Starcoder 2 and the stack v2: The next generation. arXiv (2024).https://doi.org/10.48550/ARXIV.2402.19173

    2024 doi arXiv

  18. [27]

    arXiv (2023).https://doi.org/10.48550/ ARXIV.2305.02309

    Nijkamp, E., Hayashi, H., Xiong, C., et al.: Codegen2: Lessons for training llms on programming and natural languages. arXiv (2023).https://doi.org/10.48550/ ARXIV.2305.02309

    2023 doi arXiv

  19. [28]

    In: ICLR (2023).https://doi.org/ 10.48550/ARXIV.2312.11805

    Nijkamp, E., Pang, B., Hayashi, H., et al.: Codegen: An open large language model for code with multi-turn program synthesis. In: ICLR (2023).https://doi.org/ 10.48550/ARXIV.2312.11805

    2023 doi arXiv

  20. [29]

    In: MLSys (2023)

    Pope, R., Douglas, S., Chowdhery, A., et al.: Efficiently scaling transformer infer- ence. In: MLSys (2023)

    2023

  21. [30]

    arXiv (2023).https://doi.org/10.48550/ARXIV.2308.12950

    Rozière, B., Gehring, J., Gloeckle, F., et al.: Code llama: Open foundation models for code. arXiv (2023).https://doi.org/10.48550/ARXIV.2308.12950

    2023 doi arXiv

  22. [31]

    https://huggingface.co/datasets/bigcode/starcoderdata

    Starcoderdata. https://huggingface.co/datasets/bigcode/starcoderdata

  23. [32]

    In: NeurIPS (2017)

    Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: NeurIPS (2017)

    2017

  24. [33]

    In: ICLR (2024)

    Wang, X., Wang, Z., Liu, J., et al.: MINT: evaluating llms in multi-turn interaction with tools and language feedback. In: ICLR (2024)

    2024

  25. [34]

    In: EuroSys (2023).https://doi.org/10.1145/ 3552326.3587438

    Wang, Y., Chen, K., Tan, H., Guo, K.: Tabi: An efficient multi-level inference system for large language models. In: EuroSys (2023).https://doi.org/10.1145/ 3552326.3587438

    2023

  26. [35]

    In: ICLR (2024)

    Xiao, G., Tian, Y., Chen, B., et al.: Efficient streaming language models with attention sinks. In: ICLR (2024)

    2024

  27. [36]

    In: ACL (2024).https://doi

    Ye, L., Tao, Z., Huang, Y., Li, Y.: Chunkattention: Efficient self-attention with prefix-aware KV cache and two-phase partition. In: ACL (2024).https://doi. org/10.18653/V1/2024.ACL-LONG.623

    2024 doi

  28. [37]

    In: OSDI (2022)

    Yu, G., Jeong, J.S., Kim, G., et al.: Orca: A distributed serving system for transformer-based generative models. In: OSDI (2022)

    2022

  29. [38]

    In: EuroSys (2025).https://doi.org/10.1145/3689031.3696086

    Yu, L., Lin, J., Li, J.: Stateful large language model serving with pensieve. In: EuroSys (2025).https://doi.org/10.1145/3689031.3696086

    2025

  30. [39]

    In: NeurIPS (2023)

    Zhang, Z., Sheng, Y., Zhou, T., et al.: H2O: heavy-hitter oracle for efficient gener- ative inference of large language models. In: NeurIPS (2023)

    2023

  31. [40]

    In: NeurIPS (2024)

    Zheng, L., Yin, L., Xie, Z., et al.: Sglang: Efficient execution of structured language model programs. In: NeurIPS (2024)

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.