Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PrefillOnly claims to serve single-token LLM workloads at up to 4x higher query rates by keeping only one layer of KV cache and scheduling with exact completion-time estimates.

desk verdict Good systems idea -- hybrid prefilling is clean and the scheduler makes sense -- but the 4x QPS headline outruns the evidence because the no-batching design is only tested on long prompts. read the letter →

arxiv 2505.07203 v1 pith:ZZUFA66E submitted 2025-05-12 cs.DC

Kuntai Du , Bowen Wang , Chen Zhang , Yiming Cheng , Qing Lan , Hejian Sang , Yihua Cheng , Jiayi Yao
show 4 more authors
Xiaoxuan Liu Yifan Qiao Ion Stoica Junchen Jiang
This is my paper · ORCID
classification cs.DC
keywords LLMinferenceprefill-onlyworkloadKVcachehybridprefillingcontinuousJCTcalibrationshortestremainingjobfirstprefixcachingGPUmemory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PrefillOnly is an inference engine for a workload the paper identifies: LLM requests that generate exactly one output token, as in recommendation, credit checking, and data labeling. The paper argues that existing engines waste GPU memory by storing full multi-layer KV caches and waste scheduling opportunities by treating completion time as uncertain, and that both wastes can be eliminated when the output length is fixed at one. Its central claim, supported by evaluation across four hardware setups, is that PrefillOnly serves 1.4-4.0x higher query rates than four baselines without raising average or P99 latency, and extends maximum input length by up to 5x without parallelizing inference. If true, this would let a fixed GPU fleet serve far more single-token LLM decisions, or serve longer inputs without paying the throughput cost of chunking or cross-GPU communication.

What carries the argument

The two load-bearing mechanisms are hybrid prefilling and continuous JCT calibration. Hybrid prefilling forwards a request's non-attention (linear) layers chunk-by-chunk while running attention layers normally, shrinking the peak memory of intermediate tensors so the engine can keep only one layer's KV cache and discard suffix caches; it is implemented through a graph-compilation pass rather than kernel rewrites. Continuous JCT calibration re-estimates every waiting request's completion time at each scheduling step, using profiled JCT as a function of input length and prefix-cache-hit tokens, and schedules the shortest remaining job first, with a queue-time offset to prevent starvation. Together they let PrefillOnly process requests one at a time without batching, which the paper argues is the right choice for compute-bound prefill workloads.

What would settle it

Run PrefillOnly and a batching engine on a prefill-only workload whose prompts average a few hundred tokens, such as short data-labeling queries, and compare throughput and P99 latency at equal query rates; if the batching engine matches or beats PrefillOnly, the compute-bound long-input premise fails.

Watch

Extended reading notes

Core claim

The paper's discovery is that a prefill-only workload-one token out, no decoding-inverts the memory and scheduling assumptions of LLM serving. Because the KV cache will never be reused for long decoding, PrefillOnly keeps only the cache of one layer and discards suffix KV caches, and because the output length is fixed, it can compute each request's completion time before scheduling. The paper shows that the main obstacle to exploiting these properties is not the KV caches themselves but the large intermediate tensors of non-attention layers, and that chunking those linear layers while leaving attention intact reduces peak GPU memory enough to make the cache savings real. On that basis PrefillOnly claims to handle 1.4-4.0x the query-per-second of page-based attention, chunked prefill, pipeline-parallel, and tensor-parallel baselines at equal or better average and P99 latency.

Load-bearing premise

The design assumes prefill-only workloads have long inputs and are GPU-compute-bound, so that discarding suffix KV caches, processing requests one at a time, and predicting JCT are the right optimizations; if many requests are short, batching would likely outperform the one-by-one scheduler.

Editorial extensions

If this is right

  • A GPU that stores only one layer of KV cache can serve inputs several times longer than an engine that stores all layers, without chunking or parallelizing attention.
  • Continuous JCT calibration raises prefix-cache hit rates by prioritizing requests whose prefix is currently cached, lowering both average and P99 latency.
  • PrefillOnly's throughput advantage grows in compute-bound, long-input workloads and shrinks at low QPS, where tensor-parallel baselines can offer lower latency.
  • The same design can apply to prefill nodes in disaggregated prefill-decode serving, since those nodes also see one-token outputs.
  • Suffix KV cache discarding trades future prefix reuse for memory; the paper notes that offloading to CPU rather than discarding is a compatible extension.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the long-input premise fails, for example a labeling workload where prompts average a few hundred tokens, batching would likely beat one-at-a-time scheduling, so the reported speedup should not be expected there.
  • The JCT proxy of cache-miss tokens achieves a Pearson correlation of 0.987 on one A100 setup; the paper leaves open whether that proxy holds across other models, quantization levels, and interconnects, so re-profiling per deployment is a natural extension.
  • Because hybrid prefilling is implemented via graph compilation over linear layers, the same memory reduction could apply to any transformer-style model with MLP blocks, and possibly to encoder-only classification models.
  • The fairness parameter trades worst-case latency against average latency, and could be exposed as a per-application SLO knob in production.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PrefillOnly, an LLM inference engine targeting requests that generate a single output token (prefill-only workloads). It has two mechanisms: hybrid prefilling, which chunks linear (non-attention) layers to reduce intermediate-tensor memory while running attention normally, enabling longer inputs and suffix KV-cache discarding without parallelization; and continuous JCT calibration, which re-estimates job completion times from input length and prefix-cache hit length and schedules one request at a time via SRJF-style selection with a fairness offset. The evaluation, implemented on vLLM with torch.compile, compares against PagedAttention, chunked prefill, pipeline parallel, and tensor parallel across L4/A100/H100 setups, Llama-8B/DeepSeek-Qwen-32B/Llama-70B, and two synthetic datasets (post recommendation and credit verification), reporting 1.4-4.0x higher QPS at equal average/P99 latency and up to 5x MIL expansion.

Significance. If the claims hold, PrefillOnly addresses a real and emerging workload class with a clean memory argument: since no decoding follows, per-layer KV caches for the generated suffix are dead on arrival, and linear-layer chunking is mathematically exact because linear operators act tokenwise. The multi-hardware/multi-model evaluation and the use of Poisson arrivals are strengths. The central quantitative claim, however, is currently demonstrated only for long-input synthetic traces; the design decision to avoid batching and the scheduler benefit itself are not isolated from the memory optimizations, and no artifacts are released. With scope stated precisely, or with the missing short-input/batched experiments, the contribution would be solid; as written, the headline 'prefill-only workload' claim overreaches the evidence.

major comments (4)
  1. [§6.1, §7.1 (Table 1)] The central claim in the Abstract and §7 — 'can process up to 4x larger queries per second without inflating average and P99 latency' — is made for prefill-only workloads in general, but the evaluation uses only long-input traces: post-recommendation profiles of about 11-17k tokens and credit-verification profiles of 40-60k tokens. The premise behind the one-at-a-time scheduler in §6.1 is that prefill-only input is 'typically long' and compute-bound, which is exactly the regime in which not batching is plausible. No experiment uses short prompts (e.g., 256-2,000 tokens) and no batched prefill-only scheduler appears among the baselines, so the key design trade-off is untested where it is most questionable. Either the claims must be explicitly scoped to long-input prefill-only workloads, or the evaluation needs a short-input workload and a batching-enabled baseline.
  2. [§7.2, Figures 6-9] The evaluation does not isolate the scheduling contribution. The post-recommendation discussion credits continuous JCT calibration with avoiding prefix-cache throttling, but Figure 9 compares full PrefillOnly only against the four baselines; there is no ablation that keeps hybrid prefilling and suffix discarding fixed and compares SRJF with continuous calibration against, say, FIFO or one-shot JCT-based SRJF. Without such an ablation, the magnitude of the 1.4-4.0x QPS gain attributable to the scheduler, as opposed to the memory/MIL improvements, cannot be assessed.
  3. [§6.3] The JCT model is calibrated by regression on the same hardware and is described as enabling 'precisely determine the JCT' (§1, §2.6). The only evidence is a Pearson correlation of 0.987 on one model (Qwen 32B FP8), and the proxy is the number of cache-miss tokens. Since the scheduling policy's cache-hit benefit depends on JCT ordering rather than absolute accuracy, report prediction error or rank-correlation on the actual models/hardware used in §7.1, and state whether the same fitted model transfers across the Llama-8B and Llama-70B setups.
  4. [§7.1, datasets] The workload traces are synthetic and no artifacts or generation scripts are released. The only descriptions are distribution parameters (Table 1), so the central quantitative results cannot be reproduced or audited. At minimum, release the trace generators and seeds, and state whether result variability across seeds is small.
minor comments (5)
  1. [Abstract and §10] 'upto' should be 'up to' in both the Abstract and the Conclusion.
  2. [Algorithm 1, line 11] The update 'score_min ← jct' should be 'score_min ← score'; as written, the algorithm retains the raw JCT value rather than the fairness-adjusted score used for selection.
  3. [§7.2 and Figure 7 caption] The text says the evaluation was run on 'three different hardware setups', but Table 3 and Figure 6 show four setups (L4, A100, H100 w/o NVLink, H100 w/ NVLink); unify the counts.
  4. [Reference [2]] Reference [2] is cited for vLLM but its title is 'Extensions in arc: How to import, add, & open' and appears to be a different document; this citation should be corrected.
  5. [Throughout] The spellings 'p99' and 'P99' are used inconsistently; choose one convention.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's performance claims are direct measurements against external baselines, and the fitted JCT model is an internal scheduler heuristic, not a prediction derived from its own inputs.

full rationale

PrefillOnly's central performance claims (§7) are direct measurements of QPS-latency trade-offs against four external baselines (PagedAttention, chunked prefill, pipeline parallel, and tensor parallel) on two workloads and four hardware setups; these comparisons do not reduce to the paper's own definitions or fitted parameters. The two mechanisms are also non-circular: hybrid prefilling's maximum-input-length improvement is measured from GPU memory usage (§4, Figure 10), and continuous JCT calibration is an internal scheduler heuristic whose JCT estimate is obtained by profiling plus linear regression (§6.3) and used only to rank waiting requests, not to fabricate the reported end-to-end latency or throughput numbers. The observation that prefill-only requests have fixed output length and therefore predictable JCT is a definitional premise of the workload class, not a hidden reuse of the result. The paper does cite prior work by overlapping authors (e.g., CacheGen, CacheBlend), but only in Related Work as compatible extensions, not as load-bearing justification for PrefillOnly's claims. No equation or fitted parameter is renamed as a predicted outcome, and no uniqueness theorem is imported from the authors' own prior work. Thus no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

The central design rests on workload assumptions (long, compute-bound, single-token), a fitted JCT predictor, and a hand-chosen fairness parameter. The hybrid prefilling memory benefit is a real mechanism, but the quantitative throughput claims depend on these unverified or fitted quantities, and no artifact is released for independent checking.

free parameters (4)
  • Fairness parameter lambda = 500 (default)
    Scheduling prioritization offset; set by hand in §7.1 and varied in Figure 11; affects average vs P99 latency trade-off.
  • JCT linear model coefficients = not disclosed
    JCT is predicted from profiled (n_input, n_cached) pairs at 1000-token granularity using linear regression (§6.3); coefficients are hardware/model-specific and not reported.
  • JCT proxy: number of cache-miss tokens = n_input - n_cached
    The proxy is selected because it had Pearson correlation 0.987 with measured JCT on one model/hardware (§6.3); an empirical choice rather than a derived quantity.
  • Hybrid prefilling chunk size = not specified
    Chunk size for the non-attention layer chunks determines the memory-throughput trade-off; the paper does not report the value used.
assumptions (7)
  • domain assumption Single-token LLM requests require only the prefilling phase and do not reuse KV caches for decoding.
    Defines the prefill-only workload (§2.3); if requests sometimes produce longer outputs, the design loses its premise.
  • domain assumption Prefill-only workloads have long inputs and are GPU-compute-bound.
    §2.4 states this; it justifies KV cache discarding and one-by-one scheduling, but short-input workloads would violate it.
  • ad hoc to paper Batching prefill-only requests does not improve throughput.
    §6.1 asserts this with a memory-bandwidth argument; no controlled ablation with a batched version of PrefillOnly is provided.
  • domain assumption Intermediate tensors of non-attention layers dominate GPU memory during prefill.
    Profiling in §4.1 shows this for Llama-3.1-8B; the paper argues it generalizes because models inflate MLP widths while compressing KV cache size.
  • ad hoc to paper JCT can be accurately predicted from input length and prefix cache hit length using a fitted linear model.
    §6.3 trains a linear regression at 1000-token granularity; no error bars are given for the deployed predictor.
  • standard math Prefix caching allows KV reuse across requests with shared prefixes.
    Standard LLM serving assumption (§2.1); used by all baselines and PrefillOnly.
  • domain assumption The evaluation workloads (Poisson arrivals, simulated profiles) represent real prefill-only workloads.
    §7.1 describes synthetic datasets; no real production trace is used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications." pith.science (2026). https://pith.science/paper/ZZUFA66E

@misc{pith2026250507203,
  author       = {Pith},
  title        = {Pith review of: PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZZUFA66E}},
  note         = {Machine review of arXiv:2505.07203}
}
read the original abstract

Besides typical generative applications, like ChatGPT, GitHub Copilot, and Cursor, we observe an emerging trend that LLMs are increasingly used in traditional discriminative tasks, such as recommendation, credit verification, and data labeling. The key characteristic of these emerging use cases is that the LLM generates only a single output token, rather than an arbitrarily long sequence of tokens. We call this prefill-only workload. However, since existing LLM engines assume arbitrary output lengths, they fail to leverage the unique properties of prefill-only workloads. In this paper, we present PrefillOnly, the first LLM inference engine that improves the inference throughput and latency by fully embracing the properties of prefill-only workloads. First, since it generates only one token, PrefillOnly only needs to store the KV cache of only the last computed layer, rather than of all layers. This drastically reduces the GPU memory footprint of LLM inference and allows handling long inputs without using solutions that reduces throughput, such as cross-GPU KV cache parallelization. Second, because the output length is fixed, rather than arbitrary, PrefillOnly can precisely determine the job completion time (JCT) of each prefill-only request before it starts. This enables efficient JCT-aware scheduling policies such as shortest remaining job first. PrefillOnly can process upto 4x larger queries per second without inflating average and P99 latency.

Figures

Figures reproduced from arXiv: 2505.07203 by the authors.

Figure 1
Figure 1. Contrasting traditional LLM inference and prefill￾only LLM inference. A prefill-only request does not reuse its KV cache for long decoding as it only generates one output token. process raw data and are general enough to obviate fine￾tuning [12, 13, 40], allowing developers to interactively debug and improve quality by just changing the prompts. • High decision quality: By carefully choosing the LLMs and engineering… view at source ↗
Figure 2
Figure 2. Overview of PrefillOnly and its core techniques. (e.g., shortest remaining job first), and then discards the KV caches during inference. This solution is widely adopted in traditional deep learning systems, but it has two severe limitations: Simply dropping KV caches increases MIL marginally: Ideally, dropping the KV caches could allow the LLM en￾gine to handle up to the number-of-layers-time longer input length. Ho… view at source ↗
Figure 4
Figure 4. Illustrating the tensor sizes of the MLP module in Llama-3.1-8B with bfloat16 precision. The size of intermediate tensors is much larger than the size of one-layer KV cache. order to increase the decoding throughput by enlarging the batch size, and also to handle longer LLM requests) without sacrificing the number of parameters in the LLM model. As a result, the LLM models have to inflate the tensor in MLP module, s… view at source ↗
Figures from the paper (7 more)
Figure 3
Figure 3. Figure 3: GPU memory traces of prefilling 32,768 tokens through Llama-3.1-8B model. intermediate tensors allocated by linear layers create large GPU memory spikes and significantly increase peak GPU memory usage of prefilling, while hybrid prefilling forwards the non-attention l…
Figure 5
Figure 5. Figure 5: Contrasting first-in-first-out (FIFO) scheduling, shortest-remaining-job-first (SRJF) scheduling and Pre￾fillOnly’s SRJF scheduling with continuous JCT calibration. The scheduling of PrefillOnly yields one more cache hit, achiev￾ing lower average latency. In this case,…
Figure 6
Figure 6. Figure 6: QPS — mean latency trade-off of PrefillOnly and baselines on four different hardware setups and two applications. PrefillOnly significantly reduces the latency when the QPS is high and only has higher QPS than tensor parallel baseline when QPS is low. Though tensor par…
Figure 7
Figure 7. Figure 7: QPS — P99 latency trade-off of PrefillOnly and baselines on three different hardware setups and two applications. PrefillOnlyPipeline Parallel Tensor Parallel 0.00 0.05 0.10 0.15 Req Tput (req/s) Better (a) Throughput w/o NVLink PrefillOnlyPipeline Parallel Tensor Para…
Figure 8
Figure 8. Figure 8: Contrasting the throughput of PrefillOnly and base￾lines on credit verification workload under 2× H100. Though NVLink significantly accelerates the communication and thus enhances the throughput of communication-intensive paral￾lelization like tensor parallel, PrefillO…
Figure 10
Figure 10. Figure 10: Hybrid prefilling improves the MIL by 7.9× without hurting the throughput, measured on a Qwen-2.5-32B model with fp8 quantization on an A100 GPU. 0 20 40 60 Latency (s) 0.0 0.5 1.0 CDF λ = 0 λ = 200 λ = 2000 [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: The CDF of request latency of PrefillOnly, under different value of fairness parameter 𝜆. Higher 𝜆 result in better P99 latency, at the cost of inflating the average latency. using NVLink and not using NVLink in [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems

    cs.DC 2026-07 conditional novelty 6.0 of 10

    State management across streaming, serving, retrieval, and learning systems is best modeled as one coupled runtime control loop spanning access, execution, and evolution.

  2. TetriServe: Efficiently Serving Mixed DiT Workloads

    cs.LG 2025-10 conditional novelty 6.0 of 10

    TetriServe's step-level, deadline-aware sequence parallelism improves SLO attainment for mixed-resolution diffusion transformer serving by up to 32% over fixed-SP systems.

Reference graph

Works this paper leans on

58 extracted references · 36 canonical work pages · cited by 2 Pith papers

  1. [1]

    https: //character.ai/

    character.ai | personalized ai for every moment of your day. https: //character.ai/. (Accessed on 09/07/2024)

    work page 2024

  2. [2]

    [Online; accessed 2025-04-17]

    Extensions in arc: How to import, add, & open – arc help center. [Online; accessed 2025-04-17]

    work page 2025

  3. [3]

    Taming{Throughput-Latency} tradeoff in{LLM} inference with {Sarathi-Serve}

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ram- jee. Taming{Throughput-Latency} tradeoff in{LLM} inference with {Sarathi-Serve}. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages 117–134, 2024

    work page 2024

  4. [4]

    Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation

    Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, et al. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Langu...

    work page 2024

  5. [5]

    The ai code editor

    Anysphere. The ai code editor. https://cursor.com/, 2025

    work page 2025

  6. [6]

    Glimpse: Continuous, real-time object recog- nition on mobile devices

    Tiffany Yu-Han Chen, Lenin Ravindranath, Shuo Deng, Paramvir Bahl, and Hari Balakrishnan. Glimpse: Continuous, real-time object recog- nition on mobile devices. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems , pages 155–168, 2015

    work page 2015

  7. [7]

    Feature engineering for machine learning and data analytics

    Guozhu Dong and Huan Liu. Feature engineering for machine learning and data analytics. CRC press, 2018

    work page 2018

  8. [8]

    Xgrammar: Flexible and efficient struc- tured generation engine for large language models

    Yixin Dong, Charlie F Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yi- long Zhao, and Tianqi Chen. Xgrammar: Flexible and efficient struc- tured generation engine for large language models. arXiv preprint arXiv:2411.15100, 2024

    arXiv 2024

Show all 58 references
  1. [9]

    Oneadapt: Fast adaptation for deep learning applications via back- propagation

    Kuntai Du, Yuhan Liu, Yitian Hao, Qizheng Zhang, Haodong Wang, Yuyang Huang, Ganesh Ananthanarayanan, and Junchen Jiang. Oneadapt: Fast adaptation for deep learning applications via back- propagation. In Proceedings of the 2023 ACM Symposium on Cloud Computing, pages 158–176, 2023

    2023

  2. [10]

    Accmpeg: Optimizing video encoding for accurate video analytics

    Kuntai Du, Qizheng Zhang, Anton Arapin, Haodong Wang, Zhengxu Xia, and Junchen Jiang. Accmpeg: Optimizing video encoding for accurate video analytics. Proceedings of Machine Learning and Systems , 4:450–466, 2022

    2022

  3. [11]

    Empowering many, biasing a few: Generalist credit scoring through large language models

    Duanyu Feng, Yongfu Dai, Jimin Huang, Yifang Zhang, Qianqian Xie, Weiguang Han, Zhengyu Chen, Alejandro Lopez-Lira, and Hao Wang. Empowering many, biasing a few: Generalist credit scoring through large language models. arXiv preprint arXiv:2310.00566, 2023

    arXiv

  4. [12]

    360brew: A decoder-only foundation model for personalized ranking and recommendation

    Hamed Firooz, Maziar Sanjabi, Adrian Englhardt, Aman Gupta, Ben Levine, Dre Olgiati, Gungor Polatkan, Iuliia Melnychuk, Karthik Ram- gopal, Kirill Talanine, et al. 360brew: A decoder-only foundation model for personalized ranking and recommendation. arXiv preprint arXiv:2501.1...

    2025

  5. [13]

    Gpt-3: Its nature, scope, limits, and consequences

    Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681–694, 2020

    2020

  6. [14]

    Github copilot - write code faster

    GitHub. Github copilot - write code faster. https://copilot.github.com/, 2025

    2025

  7. [15]

    Tiresias: A {GPU} cluster manager for distributed deep learning

    Juncheng Gu, Mosharaf Chowdhury, Kang G Shin, Yibo Zhu, Myeong- jae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo. Tiresias: A {GPU} cluster manager for distributed deep learning. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19), pages 4...

    2019

  8. [16]

    Annollm: Making large language models to be better crowdsourced annotators

    Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.16854, 2023

    2023 arXiv

  9. [17]

    Kvquant: Towards 10 million context length llm inference with kv cache quanti- zation

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Ma- honey, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quanti- zation. arXiv preprint arXiv:2401.18079, 2024

    2024 arXiv

  10. [18]

    Epic: Efficient position-independent context caching for serving large language models, 2024

    Junhao Hu, Wenrui Huang, Haoyi Wang, Weidong Wang, Tiancheng Hu, Qin Zhang, Hao Feng, Xusheng Chen, Yizhou Shan, and Tao Xie. Epic: Efficient position-independent context caching for serving large language models, 2024

    2024

  11. [19]

    Ragcache: Efficient knowledge caching for retrieval- augmented generation

    Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. Ragcache: Efficient knowledge caching for retrieval- augmented generation. arXiv preprint arXiv:2404.12457, 2024

    2024 arXiv

  12. [20]

    Gear: An efficient kv cache com- pression recipe for near-lossless generative inference of llm, 2024

    Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache com- pression recipe for near-lossless generative inference of llm, 2024

    2024

  13. [21]

    Over-fitting and model tuning

    Max Kuhn, Kjell Johnson, Max Kuhn, and Kjell Johnson. Over-fitting and model tuning. Applied predictive modeling, pages 61–92, 2013

    2013

  14. [22]

    Efficient memory management for large language model serving with 13 pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with 13 pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles...

    2023

  15. [23]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  16. [24]

    Spam-t5: Benchmarking large language models for few-shot email spam detection

    Maxime Labonne and Sean Moran. Spam-t5: Benchmarking large language models for few-shot email spam detection. arXiv preprint arXiv:2304.01238, 2023

    2023 arXiv

  17. [25]

    Depression detection on social media with large language models

    Xiaochong Lan, Yiming Cheng, Li Sheng, Chen Gao, and Yong Li. Depression detection on social media with large language models. arXiv preprint arXiv:2403.10750, 2024

    2024

  18. [26]

    Reducto: On-camera filtering for resource-efficient real-time video analytics

    Yuanqi Li, Arthi Padmanabhan, Pengzhan Zhao, Yufei Wang, Guo- qing Harry Xu, and Ravi Netravali. Reducto: On-camera filtering for resource-efficient real-time video analytics. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Commu- nication on ...

    2020

  19. [27]

    Terapipe: Token-level pipeline parallelism for training large-scale language models

    Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. Terapipe: Token-level pipeline parallelism for training large-scale language models. In International Conference on Machine Learning, pages 6543–6552. PMLR, 2021

    2021

  20. [28]

    Edge assisted real-time object detection for mobile augmented reality

    Luyang Liu, Hongyu Li, and Marco Gruteser. Edge assisted real-time object detection for mobile augmented reality. In The 25th annual international conference on mobile computing and networking , pages 1–16, 2019

    2019

  21. [29]

    Gonzalez, Ion Stoica, and Matei Zaharia

    Shu Liu, Asim Biswal, Audrey Cheng, Xiangxi Mo, Shiyi Cao, Joseph E. Gonzalez, Ion Stoica, and Matei Zaharia. Optimizing llm queries in relational workloads, 2024

    2024

  22. [30]

    Fingpt: Democratizing internet-scale data for financial large language models

    Xiao-Yang Liu, Guoxuan Wang, Hongyang Yang, and Daochen Zha. Fingpt: Democratizing internet-scale data for financial large language models. arXiv preprint arXiv:2307.10485, 2023

    2023 arXiv

  23. [31]

    Cachegen: Kv cache compression and streaming for fast large language model serving

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anantha- narayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. Cachegen: Kv cache compression and streaming for fast large language model s...

    2024

  24. [32]

    Kivi: A tuning-free asym- metric 2bit quantization for kv cache

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asym- metric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024

    2024 arXiv

  25. [33]

    Github - lmcache/lmcache: Redis for llms

    LMCache. Github - lmcache/lmcache: Redis for llms. [Online; accessed 2025-04-17]

    2025

  26. [34]

    Chatgpt: Conversational language model

    OpenAI. Chatgpt: Conversational language model. https://chat.openai. com, 2025

    2025

  27. [35]

    Generative agents: Interac- tive simulacra of human behavior

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Mor- ris, Percy Liang, and Michael S Bernstein. Generative agents: Interac- tive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology , pages 1–22, 2023

    2023

  28. [36]

    Optimus: an efficient dynamic resource scheduler for deep learn- ing clusters

    Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, and Chuanxiong Guo. Optimus: an efficient dynamic resource scheduler for deep learn- ing clusters. In Proceedings of the Thirteenth EuroSys Conference , pages 1–14, 2018

    2018

  29. [37]

    Perplexity is a free ai search engine

    Perlexity AI. Perplexity is a free ai search engine. https://www. perplexity.ai/, 2025

    2025

  30. [38]

    Megatron-lm: Training multi- billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi- billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  31. [39]

    Rah! recsys–assistant–human: A human-centered recommendation framework with llm agents

    Yubo Shu, Haonan Zhang, Hansu Gu, Peng Zhang, Tun Lu, Dongsheng Li, and Ning Gu. Rah! recsys–assistant–human: A human-centered recommendation framework with llm agents. IEEE Transactions on Computational Social Systems, 2024

    2024

  32. [40]

    Playing games with ais: the limits of gpt-3 and similar large language models

    Adam Sobieszek and Tadeusz Price. Playing games with ais: the limits of gpt-3 and similar large language models. Minds and Machines , 32(2):341–364, 2022

    2022

  33. [41]

    Beyond classification: Financial reasoning in state-of-the-art language models

    Guijin Son, Hanearl Jung, Moonjeong Hahm, Keonju Na, and Sol Jin. Beyond classification: Financial reasoning in state-of-the-art language models. arXiv preprint arXiv:2305.01505, 2023

    2023 arXiv

  34. [42]

    En- hancing recommender systems with large language model reasoning graphs

    Yan Wang, Zhixuan Chu, Xin Ouyang, Simeng Wang, Hongyan Hao, Yue Shen, Jinjie Gu, Siqiao Xue, James Y Zhang, Qing Cui, et al. En- hancing recommender systems with large language model reasoning graphs. arXiv preprint arXiv:2308.10835, 2023

    2023 arXiv

  35. [43]

    Chain-of-thought prompt- ing elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompt- ing elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  36. [44]

    A survey on large language models for recommendation

    Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al. A survey on large language models for recommendation. World Wide Web, 27(5):60, 2024

    2024

  37. [45]

    Predict- ing loan default in peer-to-peer lending using narrative data

    Yufei Xia, Lingyun He, Yinguo Li, Nana Liu, and Yanlin Ding. Predict- ing loan default in peer-to-peer lending using narrative data. Journal of Forecasting, 39(2):260–280, 2020

    2020

  38. [46]

    Auto-gpt for online deci- sion making: Benchmarks and additional opinions

    Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online deci- sion making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224, 2023

    2023 arXiv

  39. [47]

    Mini- mizing hallucinations and communication costs: Adversarial debate and voting mechanisms in llm-based multi-agents

    Yi Yang, Yitong Ma, Hao Feng, Yiming Cheng, and Zhu Han. Mini- mizing hallucinations and communication costs: Adversarial debate and voting mechanisms in llm-based multi-agents. Applied Sciences, 15(7):3676, 2025

    2025

  40. [48]

    Cacheblend: Fast large language model serving for rag with cached knowledge fusion

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems , Euro...

    2025

  41. [49]

    Flashinfer: Efficient and customizable attention engine for llm inference serving

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. Flashinfer: Efficient and customizable attention engine for llm inference serving. arXiv preprint arXiv:2501.01005, 2025

    2025 arXiv

  42. [50]

    Orca: A distributed serving system for {Transformer-Based} generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Sympo- sium on Operating Systems Design and Implementation (OSDI 22) , pages 521–538, 2022

    2022

  43. [51]

    Towards explaining the effects of data preprocessing on machine learning

    Carlos Vladimiro González Zelaya. Towards explaining the effects of data preprocessing on machine learning. In2019 IEEE 35th international conference on data engineering (ICDE) , pages 2086–2090. IEEE, 2019

    2019

  44. [52]

    Caravan: practical online learning of in-network ml models with labeling agents

    Qizheng Zhang, Ali Imran, Enkeleda Bardhi, Tushar Swamy, Nathan Zhang, Muhammad Shahbaz, and Kunle Olukotun. Caravan: practical online learning of in-network ml models with labeling agents. In Proceedings of the 3rd Workshop on Practical Adoption Challenges of ML for Systems, ...

    2024

  45. [53]

    Ll- maaa: Making large language models as active annotators

    Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. Ll- maaa: Making large language models as active annotators. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 13088–13103, 2023

    2023

  46. [54]

    H2o: Heavy-hitter oracle for efficient generative in- ference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative in- ference of large language models. Advances in Neural Information Processing Sys...

    2024

  47. [55]

    Mpic: Position-independent multimodal context caching system for efficient mllm serving, 2025

    Shiju Zhao, Junhao Hu, Rongxiao Huang, Jiaqi Zheng, and Guihai Chen. Mpic: Position-independent multimodal context caching system for efficient mllm serving, 2025

    2025

  48. [56]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024

    2024

  49. [57]

    {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serv- ing

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serv- ing. In 18th USENIX Symposium on Operating Systems Design and Implementation (OS...

    2024

  50. [2024]

    Association for Computing Machinery

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.