HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference

Chen Zhong; Hui Lu; Jia Rao; Lingfeng Xiang; Peiran Du; Song Jiang; Weishu Deng; Yujie Yang; Zhen Lin

REVIEW 4 major objections 5 minor 53 references

HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GPU-CPU attention split serves long contexts at near-full accuracy

desk verdict Real hybrid attention system with exact LSE fusion, but accuracy claims rest on shaky perplexity numbers and a frozen selection statistic. read the letter →

arxiv 2507.03153 v1 pith:FOITHG2C submitted 2025-07-03 cs.LG

Weishu Deng , Yujie Yang , Peiran Du , Lingfeng Xiang , Zhen Lin , Chen Zhong , Song Jiang , Hui Lu

show 1 more author

Jia Rao

This is my paper · ORCID

classification cs.LG

keywords long-contextLLMinferenceKVcacheoffloadinghybridCPU-GPUattentionsparselog-sum-expfusionper-headsparsificationlocalityserving

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that long-context LLM inference can stay accurate and fast on a single commodity GPU by splitting attention computation between the GPU and the CPU, rather than keeping all key-value (KV) entries on the GPU or shipping the whole cache over PCIe. The GPU runs dense attention over the most recent KV entries, which dominate attention scores, while the CPU runs sparse, per-head attention over older entries selected by a moving average of past attention weights. The two partial outputs are merged with an exact log-sum-exp fusion, so the result is numerically the same softmax over the union of tokens. If true, this gives near-full-attention quality with much larger KV caches, longer sequences, and bigger batches than full-attention serving on the same hardware.

What carries the argument

The load-bearing object is the log-sum-exp (LSE) fusion identity for softmax: for any partition of keys into sets $I$ and $J$, the combined softmax output is $(e^{\mathrm{lse}_I}O_I + e^{\mathrm{lse}_J}O_J)/(e^{\mathrm{lse}_I}+e^{\mathrm{lse}_J})$, with $\mathrm{lse}_S = \log\sum_{j\in S} e^{s_j}$; this lets HGCA merge a GPU dense partial output and a CPU sparse partial output into the exact full softmax over the union, transferring only output vectors and scalars. The companion mechanism is the moving average of attention weights (MAW): each KV entry carries a running average of how strongly it has been attended, and on the CPU a per-head threshold $\beta$ selects entries whose MAW exceeds $\beta$ divided by the total number of GPU-side KV entries, giving finer-grained sparsification than layer-wise top-$k$. A circular buffer on the GPU keeps the newest KV blocks resident and asynchronously evicts old blocks to CPU memory.

What would settle it

Generate a long sequence or a multi-turn conversation in which the relevant context shifts to an older part of the prompt after many tokens, such as a question whose answer is in the middle of a long document followed by a question that re-opens an earlier section, and measure perplexity or task accuracy against full attention; if the CPU sparse subset under the moving-average threshold cannot recover the newly relevant old KV entries, HGCA's accuracy will fall measurably below full attention.

Watch

Extended reading notes

Core claim

HGCA's central claim is that attention can be partitioned losslessly across devices: full attention over a bounded recent window on the GPU plus sparse attention over CPU-resident older entries, combined by log-sum-exp normalization, reproduces the full-context softmax output, while the sparse subset is chosen per head by comparing each entry's moving average of past attention scores against a threshold. The paper argues that because attention is spatially and contextually local, the moving average identifies the few older KV entries that continue to matter, and the recent window covers the rest; therefore accuracy stays close to, and sometimes better than, full attention. Experiments with OPT, GPT-NeoX, and LLaMA models report near-baseline perplexity across thresholds and higher throughput and longer generation than offloading-only or fixed-budget sparse baselines.

Load-bearing premise

The claim stands or falls on whether a KV entry's moving average of past attention weights predicts which older entries the model will attend to next; if attention shifts to previously pruned entries after a topic change or an append, the CPU sparse set misses them and accuracy drops.

Editorial extensions

If this is right

Serving systems can extend generation length and batch size on a single GPU by replacing full-attention-over-offloaded-KV with hybrid attention, because PCIe traffic shrinks from whole KV tensors to partial outputs and scalars.
Per-head sparsification on the CPU can keep accuracy close to full attention with far fewer selected KV entries than a uniform top-$k$ budget, since heads with peaked attention need very few entries while flat heads keep more.
The fusion step is exact, so HGCA's output is the same as full attention over the union of GPU-resident and selected CPU KV entries; the only approximation is the sparse selection on the CPU side.
Because the mechanism is a drop-in attention-layer replacement that needs no retraining, existing pretrained models can be served longer without modifying their weights.
The performance advantage over offloading-only approaches grows as the CPU-resident KV fraction and batch size grow, since more attention work is shifted to CPU bandwidth while the merge cost stays small.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The moving-average salience criterion is a temporal-locality bet; extending it to a predictive scorer that anticipates shifts in attention, such as after topic changes or appends, could make the CPU sparse set robust to attention reallocation.
Because the GPU window is treated as lossless, the scheme's accuracy should be sensitive to the window size when recent tokens do not dominate attention, such as retrieval-heavy prompts where the answer lies in the middle; an adaptive per-head GPU window is a natural testable extension.
The reported outliers and load imbalance in CPU thread scheduling suggest that a work-stealing or core-aware scheduler for per-head sparse tasks could close the remaining throughput gap.
The same partition-and-fuse pattern applies beyond GPU/CPU pairs, such as between HBM and CXL-attached memory or between GPUs with different bandwidths, wherever a fast small memory and a slow large memory coexist.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: HGCA is a serious systems paper with a clean exact-fusion core and a plausible CPU-GPU execution idea, but the accuracy claim is supported by a perplexity table that looks wrong in places and by a selection heuristic whose predictive validity is not demonstrated. I'd send it to review, but it needs major revisions.

What's actually new: the combination of dense GPU attention on a recent window, per-head CPU sparse attention on offloaded entries selected by moving-average attention weights, and exact log-sum-exp fusion. The LSE merge is mathematically correct and is the right way to combine partial softmax outputs without transferring full KV tensors. The motivation is well argued: attention is memory-bound, CPU bandwidth is not far behind GPU bandwidth, and per-head sparsity varies a lot. The implementation is substantial: FlexGen and HF integrations, multiple models, and a 16K-token single-request decode.

Soft spots, in proportion:

- Table 1 is the main weakness. The baseline perplexities are implausible: LLaMA-2-7B at 124.4 and LLaMA-2-13B at 227.2 on WikiText are far from typical values. Many configurations degrade badly: OPT-30B at GPU ratio 0.75, beta 0.5 gives 27.83 vs baseline 16.78. The 'near-perfect' claim is really about a handful of grid-selected cells, with no error bars. Some cells beat full attention by 20% (GPT-NeoX-12B), which needs an explanation.

- The selection rule is load-bearing. MAW is updated only from the GPU-side partial attention over the recent window, then frozen at eviction. In single-request decode, an early token that becomes important later cannot be recovered until an append—which never happens in the 16K experiment. Figure 5 shows two decoding positions, one layer, one head; it does not test this failure mode. The paper asserts contextual locality is 'already established' in the Re-evaluation paragraph, but that is exactly the claim that needs evidence.

- Missing comparisons with the closest hybrid CPU-GPU systems (TwinPilots, HeadInfer) make the performance claims hard to position.

- No code release despite the drop-in claim, and a few typos (Lotus vs GPT-NeoX, Llama vs Vicuna in captions).

Who it's for: systems people working on LLM inference on commodity hardware. It deserves a serious referee, but the accuracy section needs a rework before the claims can stand.

Referee Report

4 major / 5 minor

Summary. The paper presents HGCA, a hybrid GPU-CPU attention mechanism for long-context LLM inference. The GPU maintains a bounded circular buffer of recent KV entries and performs dense attention on them, while offloaded older KV entries are stored in CPU memory and processed with a per-head sparse attention scheme that retains entries whose moving average attention weight (MAW) exceeds a threshold. The two partial attention outputs are merged on the GPU using log-sum-exp (LSE) fusion, transmitting only small intermediate tensors over PCIe. The authors implement HGCA on top of FlexGen and Hugging Face Transformers, evaluate it on OPT, GPT-NeoX, and LLaMA-family models, and report improved scalability and throughput compared with full-attention offloading and sparse-attention baselines, with perplexity claimed to be close to or better than full attention.

Significance. If the accuracy claim survives scrutiny, HGCA would be a practical contribution: it turns CPU memory and compute into a cooperative partner rather than a mere spill target, and the exact LSE fusion of the two computed subsets is mathematically correct and independent of learned parameters. The per-head, CPU-side sparsification is a sensible use of CPU control-flow flexibility, and the system requires no model retraining. The reported scalability results, especially in Figures 12–14, suggest real throughput benefits on commodity GPUs. However, the central 'near-full attention quality' claim is substantially weaker than the paper presents: Table 1 contains many configurations with large perplexity degradation, and the MAW-based selection rule has no mechanism to recover from attention-pattern shifts during a single long decode. The paper's strengths are the system design and the exact merge; its main weakness is the evidence and argument for accuracy preservation.

major comments (4)

[§5.3, Table 1] The claim of 'near-identical model accuracy compared to full attention in all models' is not supported by the full table. For LLaMA-2-13B, GPU ratios 0.25 and 0.5 yield perplexities of roughly 288–341 against a baseline of 227.2; Vicuna-33B at GPU ratio 0.25 reaches about 993–1013 versus a baseline of 889.1; OPT-30B at GPU ratio 0.75 reaches 18.33–27.83 versus 16.78. These are not near-full-attention results. Because beta and GPU KV ratio are grid-searched per model and the favorable cells are highlighted, the 'better than full attention' phrasing in the abstract and §5.3 overstates what the data show. The paper should report the exact settings used for each claimed configuration, include variance or multiple runs, and clearly separate 'best selected configuration' from a general accuracy guarantee.
[§3.2.2, Algorithm 1; §5.4] The MAW-based selection rule cannot react to late-appearing attention during a single-request decode. In Algorithm 1, line 8, A_maw is updated only from A_gpu, which comes from the GPU-resident recent window (Algorithm 2, line 10); once a KV entry is evicted its MAW is frozen. The Re-evaluation paragraph in §3.2.2 reconsiders pruned entries only after an append operation, via Algorithm 1 lines 19–22. The long-context experiment in §5.4 is a single 16,384-token request with no append, so an early token that becomes salient only after many decoding steps has already been permanently pruned from CPU sparse attention. Figure 5 and Observation O-2 demonstrate locality at two decoding positions in one layer/head but do not show whether tokens with low MAW at eviction later receive high attention. This is load-bearing for the 'near-full attention quality' claim in long decode; the paper needs a direct test of whether the retained CPU subset contains the entries that actually receive high attention at future steps, or an update mechanism that incorporates A_cpu into MAW.
[§5.3] The statement that HGCA 'even outperform[s] full attention in certain cases' is not credible as presented. A sparse approximation to full attention should not systematically improve perplexity over the exact full-attention baseline; cells such as LLaMA-2-7B at 105.4 versus 124.4 and GPT-NeoX-12B at 51.09 versus 66.88 are more plausibly explained by evaluation variance, differing text lengths, or implementation differences between the HGCA path and the Hugging Face reference path. Without multiple seeds, confidence intervals, or a mechanistic explanation, these 'better than full attention' claims should be removed or substantially qualified.
[Algorithm 1, §5.3] The free parameters of the method are not accounted for in the accuracy evaluation. Alpha, the moving-average factor in Algorithm 1 line 8, is never given a value in the paper; beta and the GPU KV ratio appear to be tuned per model in Table 1 without a stated selection protocol. The claim that 'best model performance typically resulted in a large beta' is contradicted by several rows (e.g., OPT-30B at GPU ratio 0.75 has its best result at beta=0.5 but severe degradation at beta=0.75). The paper should report the alpha value, the search ranges, and the sensitivity of perplexity to these parameters, since the practical claim depends on configurations that are not systematically justified.

minor comments (5)

[§3.3] The merge formula z = e^{lsec} + e^{lseg} is mathematically equivalent to the LSE trick but is not numerically stable as written; the standard formulation subtracts the maximum of the two lse values before exponentiating. The text mentions 'the maximum score and the sum of exponentials' but the displayed equation does not reflect that stabilization.
[§4] The implementation section says parallelism is enabled via 'torch.fork'; the intended PyTorch API is torch.jit.fork. Please correct the name.
[Algorithm 2] In the append branch, line 4 loads KV_cpu but line 7 uses KV_cpu||ctx[i], while in the decode branch line 2 loads KV_ctx. The relationship between KV_cpu and KV_ctx, and which one is the contextual cache, should be clarified in the pseudocode.
[§5.2, Figure 12] Infinigen is reported to OOM 'across all models,' which is surprising given the original paper reports results for the same class of models. The comparison would be stronger if the memory budget or configuration difference were explained, rather than presenting the OOM as a property of the baseline.
[Table 1] The baseline perplexities for LLaMA-2-7B (124.4) and LLaMA-2-13B (227.2) are unusually high for WikiText; the exact text length, dataset split, and whether the reference is computed with the same batching/implementation should be stated so that the comparison is reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HGCA's exact LSE merge is mathematically independent, and the CPU salience heuristic is empirically motivated rather than derived from the claim it supports.

full rationale

HGCA does not present a first-principles derivation whose conclusion is pre-supposed by its inputs. The only exact step in the pipeline is the log-sum-exp fusion, which the paper explicitly states yields an output 'equivalent to that produced by a single softmax operation over the union of GPU and CPU tokens'; this is a standard, parameter-free mathematical identity and is not circular. The CPU-side salience criterion, based on a moving average of attention weights and a threshold beta, is an empirically motivated heuristic, and the paper reports perplexity for a grid of beta and GPU KV ratio settings rather than fitting a parameter and then 'predicting' the same quantity. The claim of near-full attention quality is an experimental measurement, not a quantity forced by construction. No load-bearing step reduces to a self-citation: the cited works by the authors ([7], [40]) are related-work references, and the central attention mechanism does not depend on any author-derived uniqueness theorem. Concerns about the MAW being frozen after eviction, or about beta being tuned per model, are correctness and robustness caveats, not instances of circular reasoning. Therefore, no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central accuracy claim rests on two empirical assumptions about attention locality, recent-token dominance and historical-attention-as-future-salience, plus hardware assumptions about CPU-GPU balance. These are not derived and are tuned through beta and GPU KV ratio. The LSE merge is the only mathematically exact component.

free parameters (4)

beta (sparsification threshold) = Grid-searched over 0.25, 0.5, 0.75, 1.0 per model in Table 1; set to 1 in Section 5.4
Controls which KV entries survive per-head pruning on the CPU. Accuracy depends strongly on its value, and no single default is shown to work across all models and GPU ratios.
GPU KV ratio = 0.25, 0.5, 0.75 in Table 1; 0.5 and 0.25 in scaling experiments
Determines how many recent KV entries remain in the dense GPU window. Accuracy varies significantly across ratios in Table 1, including large degradations for OPT-30B at ratio 0.25.
alpha (moving average factor for MAW) = Not reported
Algorithm 1 line 8 updates attention weights with an exponential moving average; the value of alpha is not given, although it controls which KV entries are later considered salient.
KV block size and circular buffer size = Not reported
The eviction and offload granularity depends on block size and buffer length, but these values are not specified in the experiments.

assumptions (4)

domain assumption Historical cumulative attention weight (MAW) predicts future salience for each head
Used in Section 3.2.2 thresholding. It is supported only by empirical observations O-2 and Figure 5, not by a formal bound. If attention patterns shift during decoding or after appends, the sparse subset on the CPU misses critical KV entries.
domain assumption Recent tokens dominate attention for near-future decoding
Justifies the GPU-resident dense window and the offloading of older KV entries in Section 3.2.1. This is presented as an empirical pattern, not a general guarantee.
domain assumption CPU computation can overlap GPU attention without becoming the bottleneck
The roofline argument in Figure 1 and the measurements in Figures 10 and 11 show this on one dual-socket Xeon server with A6000 GPUs; the result is hardware-dependent and may not transfer to other CPU-GPU combinations.
standard math LSE fusion exactly reproduces a single softmax over the union of tokens
FlashAttention-style log-sum-exp merging is mathematically exact, and the paper uses it correctly in Section 3.3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference." pith.science (2026). https://pith.science/paper/FOITHG2C

@misc{pith2026250703153,
  author       = {Pith},
  title        = {Pith review of: HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FOITHG2C}},
  note         = {Machine review of arXiv:2507.03153}
}

read the original abstract

Scaling inference for large language models (LLMs) is increasingly constrained by limited GPU memory, especially due to growing key-value (KV) caches required for long-context generation. While existing approaches offload KV caches to CPU memory or apply sparse attention to reduce GPU load, they often underutilize CPU compute resources and compromise accuracy. We present HGCA, a hybrid CPU-GPU attention mechanism that enables scalable, high-throughput LLM inference with near-full attention quality. HGCA performs dense attention on recently generated KV entries retained in GPU memory and parallel sparse attention on selected, salient KV entries in CPU memory. The attention outputs are efficiently merged using log-sum-exp fusion, minimizing PCIe transfer overhead. HGCA also introduces a finegrained, per-head sparsification strategy optimized for CPU execution, preserving contextual relevance while reducing computation. Our implementation seamlessly integrates into existing LLM frameworks without requiring model retraining. Experiments across diverse models and workloads show that HGCA achieves superior scalability, supports longer sequences and larger batch sizes, and outperforms existing sparse attention baselines in both performance and accuracy -- all on commodity GPU hardware.

Figures

Figures reproduced from arXiv: 2507.03153 by the authors.

**Figure 1.** Roofline model of attention stages in LLM serving. accuracy or requiring algorithmic changes to the attention mechanism? Our work is motivated by two important observations. First, unlike LLM training, LLM inference often exhibits low operational density [38] – defined as the number of operations per byte of memory traffic – offering opportunities for offloading a portion of attention computation to the CPU withou… view at source ↗

**Figure 2.** The comparison of different attention mechanisms. 2.1 Standard Full Attention Standard full attention ensures that all KV entries participate in the attention process, preventing any important information from being overlooked. The following equations outline the decode and append processes for LLM inference, as described in [38]. During inference, the incoming hidden states are projected into query, key, and valu… view at source ↗

**Figure 4.** The percentage of KV entries required to achieve 99% of the cumulative attention score per head in layer 16 of OPT-6.7B, for two contexts at the same decoding step. Contextual locality Spatial locality Prefill Decode [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Figures from the paper (8 more)

**Figure 5.** Figure 5: shows the distribution of attention scores for a single inference request at two decoding positions: the 256th and 512th tokens. The orange dots represent KV entries present after generating 256 tokens, while the black 1 4 8 12 16 20 24 28 32 Heads 0.0 0.2 0.4 0.6 0.8…

**Figure 7.** Figure 7: System overview of HGCA. hardware-efficient and scalable. In response, recent work has introduced block-wise sparse attention kernels optimized for modern accelerators [18, 35, 41]. KV cache management: Efficient KV cache management is key to scalable LLM inference. Ea…

**Figure 8.** Figure 8: HGCA: KV cache manager. the offloaded block of older entries (with sparse approximations). Each side produces a partial result along with the necessary normalization statistics, with which HGCA applies normalization to merge these partial attention outputs on the GPU…

**Figure 9.** Figure 9: HGCA: hybrid attention computation. The GPU performs heavy-density computation on a small subset, and the CPU handles only a lightweight sparse subset, preventing one from becoming a bottleneck. (2) Enabling efficient collaboration: GPU and CPU workloads execute conc…

**Figure 10.** Figure 10: Speedup due to HGCA’s hybrid attention. Performance Metrics. We report the following performance metrics: 1) end-to-end task completion time as a function of output sequence length and batch size, 2) peak memory consumption attributable to KV attention in addition t…

**Figure 11.** Figure 11 [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]

**Figure 13.** Figure 13: The generation of 4096 tokens in GPT-Neox-12B. pre-allocates memory for on-GPU KV entries. We compared two variants of HGCA against HF. First, we scaled HGCA to use multiple GPUs following HF’s partitioning of model weights but only enabled GPU (full) attention (denot…

**Figure 14.** Figure 14: The generation of 4096 tokens in Llama-33B [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 22 canonical work pages

[1]

Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems 6 (2024), 114– 127

work page 2024
[2]

Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. 2023. Sarathi: Effi- cient llm inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308.16369 (2023)

arXiv 2023
[3]

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020)

arXiv 2020
[4]

Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al . 2022. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745 (2022)

arXiv 2022
[5]

Sumit Kumar Dam, Choong Seon Hong, Yu Qiao, and Chaoning Zhang

work page
[6]

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré
[7]

Weishu Deng and Jia Rao. 2024. Mega: More Efficient Graph Attention for GNNs. In 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS). IEEE, 71–81

work page 2024
[8]

Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. 2024. Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems 36 (2024)

work page 2024

Show all 53 references

[9]

2024.{Cost- Efficient} large language model serving for multi-turn conversations with{CachedAttention}

Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024.{Cost- Efficient} large language model serving for multi-turn conversations with{CachedAttention}. In 2024 USENIX Annual Technical Conference (USENIX ATC ...

2024
[10]

Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801 (2023)

2023 arXiv
[11]

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 (2020)

2020 arXiv
[12]

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica
[13]

Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. {InfiniGen}: Efficient generative inference of large language models with dynamic{KV} cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) . 155–172

2024
[14]

Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast in- ference from transformers via speculative decoding. In International Conference on Machine Learning . PMLR, 19274–19286

2023
[15]

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen
[16]

Chaofan Lin, Jiaming Tang, Shuo Yang, Hanshuo Wang, Tian Tang, Boyu Tian, Ion Stoica, Song Han, and Mingyu Gao. 2025. Twilight: Adaptive Attention Sparsity with Hierarchical Top-𝑝 Pruning. arXiv preprint arXiv:2502.02770 (2025)

2025
[17]

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchi- cal vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision . 10012–10022

2021
[18]

Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. 2025. MoBA: Mixture of Block Attention for Long-Context LLMs. arXiv preprint arXiv:2502.13189 (2025)

2025 arXiv
[19]

Advances in Neural Information Processing Systems 37 (2024), 22947– 22970

Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems 37 (2024), 22947– 22970

2024
[20]

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher
[21]

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei- Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393 (2025)

2025 arXiv
[22]

Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and Jie Zhang. 2024. Instinfer: In-storage attention offloading for cost-effective long-context llm inference.arXiv preprint arXiv:2409.04992 (2024)

2024 arXiv
[23]

Cheng Luo, Zefan Cai, Hanshi Sun, Jinqi Xiao, Bo Yuan, Wen Xiao, Junjie Hu, Jiawei Zhao, Beidi Chen, and Anima Anandkumar. 2025. HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading. arXiv preprint arXiv:2502.12574 (2025)

2025 arXiv
[24]

Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ram- jee, and Ashish Panwar. 2024. vattention: Dynamic memory man- agement for serving llms without pagedattention. arXiv preprint arXiv:2405.04437 (2024)

2024 arXiv
[25]

Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation—A{KVCache-centric} Architecture for Serving{LLM} Chatbot. In 23rd USENIX Conference on File and Storage Te...

2025
[26]

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

2023 arXiv
[27]

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning...

2023
[28]

Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Christopher Pal. 2020. On extractive and abstractive neural document summariza- tion with transformer language models. In Proceedings of the 2020 con- ference on empirical methods in natural language processing (EMNLP) . 9308–9319

2020
[29]

Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. Quest: Query-Aware Sparsity for Efficient Long- Context LLM Inference. arXiv preprint arXiv:2406.10774 (2024)

2024 arXiv
[30]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Alma- hairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

2023 arXiv
[31]

Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. 2024. Model tells you where to merge: Adaptive kv cache merging for llms on long- context tasks. arXiv preprint arXiv:2407.08454 (2024)

2024 arXiv
[32]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

2022
[33]

Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. Powerinfer: Fast large language model serving with a consumer-grade gpu. In 13 Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. 590–606

2024
[34]

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 (2023)

2023 arXiv
[35]

Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. 2025. XAttention: Block Sparse Attention with Antidiagonal Scoring. arXiv preprint arXiv:2503.16428 (2025)

2025 arXiv
[36]

Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowl- edge Fusion. In Proceedings of the Twentieth European Conference on Computer System...

2025
[37]

Wenhua Ye, Xu Zhou, Joey Zhou, Cen Chen, and Kenli Li. 2023. Accel- erating attention mechanism on fpgas based on efficient reconfigurable systolic array. ACM Transactions on Embedded Computing Systems 22, 6 (2023), 1–22

2023
[38]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019)

2019 arXiv
[39]

Zihao Ye, Ruihang Lai, Bo-Ru Lu, Chien-Yu Lin, Size Zheng, Lequn Chen, Tianqi Chen, and Luis Ceze. 2024. Cascade inference: Memory bandwidth efficient shared prefix batch decoding

2024
[40]

Chengye Yu, Tianyu Wang, Zili Shao, Linjie Zhu, Xu Zhou, and Song Jiang. 2024. Twinpilots: A new computing paradigm for gpu-cpu parallel llm inference. In Proceedings of the 17th ACM International Systems and Storage Conference . 91–103

2024
[41]

Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. 2025. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089 (2025)

2025 arXiv
[42]

Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyun- woo J Kim. 2019. Graph transformer networks. Advances in neural information processing systems 32 (2019)

2019
[43]

Zihao Ye, Lequn Chen, Ruihang Lai, Yilong Zhao, Size Zheng, Junru Shao, Bohan Hou, Hongyi Jin, Yifei Zuo, Liangsheng Yin, et al. 2024. Accelerating self-attentions for llm serving with flashinfer

2024
[44]

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Vic- toria Lin, et al . 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022)

2022 arXiv
[45]

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2024. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing...

2024
[46]

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody_Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2023. Efficiently Programming Large Language Models using SGLang. (2023)

2023
[47]

Shuzhang Zhong, Yanfan Sun, Ling Liang, Runsheng Wang, Ru Huang, and Meng Li. 2025. HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference. arXiv preprint arXiv:2504.05897 (2025)

2025 arXiv
[48]

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems 33 (2020), 17283–17297

2020
[53]

2024.{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving

Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024.{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation ...

2024
[2016]

arXiv preprint arXiv:1609.07843 (2016)

Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 (2016)

2016 arXiv
[2022]

Advances in Neural Information Processing Systems 35 (2022), 16344–16359

Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359

2022
[2023]

In Proceedings of the 29th Symposium on Operating Systems Principles

Efficient memory management for large language model serv- ing with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626
[2024]

arXiv preprint arXiv:2406.16937 (2024)

A complete survey on llm-based ai chatbots. arXiv preprint arXiv:2406.16937 (2024)

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems 6 (2024), 114– 127

work page 2024

[2] [2]

Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. 2023. Sarathi: Effi- cient llm inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308.16369 (2023)

arXiv 2023

[3] [3]

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020)

arXiv 2020

[4] [4]

Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al . 2022. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745 (2022)

arXiv 2022

[5] [5]

Sumit Kumar Dam, Choong Seon Hong, Yu Qiao, and Chaoning Zhang

work page

[6] [6]

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

[7] [7]

Weishu Deng and Jia Rao. 2024. Mega: More Efficient Graph Attention for GNNs. In 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS). IEEE, 71–81

work page 2024

[8] [8]

Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. 2024. Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems 36 (2024)

work page 2024