Pith. sign in

REVIEW 3 major objections 5 minor 38 references

SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SpindleKV claims that compressing shallow-layer KV caches with a similarity-based codebook while evicting tokens in deep layers matches or beats pyramid-style eviction baselines at the same or smaller cache.

desk verdict SpindleKV is a genuinely mixed-method KV compression idea with consistent empirical wins, but its load-bearing memory-ratio formula and GQA accounting need a measured-memory check before the same-budget claims can be trusted. read the letter →

arxiv 2507.06517 v1 pith:7Z7OLKLJ submitted 2025-07-09 cs.CL

Zicong Tang , Shi Luohe , Zuchao Li , Baoyuan Qi , Guoming Liu , Lefei Zhang , Ping Wang This is my paper · ORCID
classification cs.CL
keywords KVcachecompressiontokenevictioncodebookreplacementgrouped-queryattentionlong-contextinferencesparsitycosinesimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that KV cache compression fails in shallow layers because existing methods only exploit inter-token importance, not a second kind of redundancy: in shallow layers, many key and value vectors point in nearly the same direction. SpindleKV therefore keeps attention-based token eviction for deep layers, but replaces shallow-layer KV entries with references into a codebook of representative vectors plus stored magnitudes. The authors argue this dual strategy preserves accuracy at equal or smaller reserve ratios than pyramid-shaped eviction baselines, and even reaches a 50 percent cache reduction without losing LongBench accuracy in some settings. This matters because KV cache memory is a bottleneck for long-context inference, so compressing it without accuracy loss directly extends feasible context length and lowers deployment cost.

What carries the argument

Three parts carry the argument. First, eviction: accumulated attention scores $ac_{i,a}$ over an observation window pick the top-$k$ tokens, with $k$ set by a linear layer-wise interpolation of reserve ratios. Second, replacement: normalized K/V vectors become nodes of a cosine-similarity graph $G_\Gamma$; greedy max-degree selection builds the codebook $C_\Gamma$, per-token references $r_\Gamma$, and stored magnitudes $m_\Gamma$, and reconstruction is $\Gamma_r = C_\Gamma[r_\Gamma] \otimes m_\Gamma$. Third, the claimed memory accounting: the effective layer reserve ratio is $r_\lambda = r_\lambda^1 \times r_\lambda^2 \times r_\lambda^3$, multiplying the eviction ratio, the codebook reference ratio, and a dtype-conversion factor that accounts for storing integer indices and float magnitudes.

What would settle it

Profile the actual KV-cache GPU memory of SpindleKV and of a pyramid-eviction baseline on the same LongBench prompt and generation length, using the exact reserve ratios reported in Tables 1 and 10; if SpindleKV's measured peak KV memory exceeds the baseline's at a ratio it claims is equal or smaller, the paper's central efficiency claim is falsified. A simpler check is to inspect the stored per-head index arrays after GQA unfolding: if the codebook references are duplicated across heads instead of shared, the second factor of Equation (15) underestimates the true footprint.

Watch

Extended reading notes

Core claim

The paper's central claim is that shallow-layer KV caches have high cosine similarity among their constituent vectors, so they can be replaced by a small codebook of basis vectors with per-token references and magnitudes, while deep-layer caches have sparse attention patterns that favor token eviction. SpindleKV combines the two: deep layers use accumulated-attention-score eviction with a pyramid-shaped layer-wise allocation, and shallow layers use the codebook replacement, with the codebook updated online during decoding. The paper further claims that this design fixes the Grouped-Query Attention dilemma by repeating KV heads before per-head eviction and letting the codebook absorb the duplication. Empirically, the authors report that SpindleKV outperforms PyramidKV and PyramidInfer on LongBench across three models at equal or slightly lower reserve ratios, and that at 15 percent cache it retains retrieval quality on Needle-in-a-Haystack that the baselines lose.

Load-bearing premise

Everything hinges on Equation (15)'s claim that the end-to-end reserve ratio is just the product of the eviction ratio, the codebook reference ratio, and the dtype factor; if the per-head index state after GQA unfolding or the re-expanded key/value copies at inference cost memory that this formula does not count, SpindleKV's headline 'same or better accuracy at the same cache size' comparisons against the baselines do not hold.

Editorial extensions

If this is right

  • Long-context inference can retain full-accuracy quality at roughly 40 percent of the original KV cache, and at 50 percent in some settings according to the paper's LongBench results.
  • Shallow layers cease to be the hard bottleneck for cache compression once their near-duplicate vectors are handled by a codebook rather than by eviction.
  • Grouped-Query Attention models can be compressed by per-head eviction after repeating KV heads, if the resulting duplication is absorbed by the codebook.
  • The paper's observation of layer-dependent cosine similarity suggests that direction and magnitude carry separable information, opening a compression axis beyond token-level importance.
  • At aggressive ratios around 15 percent, the method preserves retrieval ability better than pyramid-shaped eviction baselines, per the Needle-in-a-Haystack experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same codebook idea could be applied to intermediate and even deep layers with a lower similarity threshold, since the redundancy is reported to decrease gradually rather than stop at a layer boundary.
  • A testable extension is to make the similarity threshold $\theta$ layer- or head-adaptive, which could squeeze additional compression from the codebook without changing the eviction policy.
  • The reserve-ratio identity in Equation (15) is an analytic claim; verifying it with direct GPU memory measurements, rather than formula-based ratios, would settle whether the 'same or better at the same cache' comparisons hold in practice.
  • If the codebook stores magnitudes per token and references per head after GQA unfolding, its true overhead may exceed the reported $r_\lambda^2 \times r_\lambda^3$ factor, a discrepancy that a memory profiler could expose.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SpindleKV, a KV-cache reduction method that applies attention-weight-based token eviction in deeper layers and a codebook-based token replacement in shallower layers, with a repeat-and-codebook step intended to handle Grouped-Query Attention. The method is evaluated on LongBench and Needle-in-a-Haystack using LLaMA2-7B-chat, LLaMA3-8B-instruct, and Mistral-7B-instruct-v0.2, reporting accuracy at reserve ratios around 14--41%. The central claim is that SpindleKV achieves the same or better accuracy than PyramidInfer and PyramidKV at equal or slightly lower reserve ratios, and in some settings reaches roughly 50% cache reduction without accuracy loss. The code is publicly available.

Significance. If the memory accounting is valid, SpindleKV is a useful contribution: it combines two complementary compression mechanisms (eviction and replacement), explicitly targets shallow-layer redundancy that eviction methods miss, and addresses the GQA compatibility issue by unfolding and then codebook-compressing. The evaluation covers three models and two benchmarks, and the paper includes ablations for the repeat operation, the codebook alone, and magnitude reconstruction. The method is evaluated on external benchmarks rather than fitted to them, and the reported gains over PyramidInfer and PyramidKV are consistent across many settings. However, the headline comparisons are same-budget comparisons, and the budget formula in Eq. (15) is not verified against measured memory; this is a load-bearing issue that must be resolved before the comparisons can be accepted.

major comments (3)
  1. [§3.4, Eq. (15)] The reserve-ratio formula in Eq. (15) is algebraically inconsistent with the memory it claims to model. If R is the number of stored references, C the number of codebook vectors, O the bits per reference (index plus magnitude), and M0 = N * dh * key_bit the full-cache memory, the exact ratio is (R*O + C*dh*key_bit) / M0 = r1 * O / (dh * key_bit) + r1 * r2, where r1 = R/N and r2 = C/R. Equation (15) instead reports r1 * r2 * (dh + int_bit/key_bit + 1)/dh, which multiplies the per-reference overhead by r2 and does not add the standalone R*O term. This understates the true memory cost, so the same-budget comparisons in Tables 1, 10, and 11 (e.g., 40.1% vs. 40.5%) are not yet valid as stated. The authors should correct the formula and, ideally, validate it against profiler-measured KV-cache memory.
  2. [§3.3--§3.4, GQA storage] The treatment of GQA is ambiguous in a way that directly affects the memory claim. Section 3.3 says the repeat operation can fully unfold GQA and that the resulting overhead is absorbed by the codebook, while the text after Eq. (15) says 'hg = h if a repeat operation is conducted'. If references and magnitudes are stored per unfolded Q-head, the denominator in Eq. (15) must be the unfolded h-head cache and the per-head reference arrays cost h * R entries per layer; if references are instead shared per KV-head, the paper must explain how per-Q-head eviction decisions and masks are stored. As written, the effective budget could be understated by a factor related to hn. The paper reports no measured GPU memory and Table 9 reports only decoding speed, so this ambiguity cannot be resolved from the text.
  3. [§4.2, Tables 1 and 10] The empirical comparisons lack error bars or significance tests, and many of the reported advantages are small (1--2 average points on LongBench). Because the reserve ratios are computed from the unverified Eq. (15), the comparison is not yet on a firm basis. At minimum, the authors should report measured KV-cache memory (or a validated memory model) for each configuration, and ideally run multiple seeds or report per-dataset variance, so the reader can distinguish a genuine improvement from threshold choice or measurement noise.
minor comments (5)
  1. [§3.4 / Table 3] The hyper-parameters θK, θV, β, and α are fixed without sensitivity analysis; a short sensitivity study or a statement that the results are stable over a range of thresholds would strengthen the claim that the improvement comes from the method rather than from threshold selection.
  2. [Abstract and §1] The abstract contains an incomplete sentence ('Based on our observation that, the KV cache exhibits a high degree of similarity.') and several typos elsewhere ('possessed', 'shown', 'avaiable', 'reocnstruct'); the paper would benefit from a careful proofread.
  3. [Appendix B vs. Table 3] The thresholds used in the preliminary observation (θ = 0.9 for Key and θ = 0.6 for Value) differ from the hyper-parameters used in the experiments (θK = 0.98, θV = 0.95); the relationship between these values should be clarified.
  4. [§3.4 / Eq. (15)] The notation 'int_bit/key_bit' and '+1' in rλ3 is not fully defined; in particular, the bit-width of the magnitude storage (mag_bit) is never specified, and the observation-window length lw is missing from Table 3. Please define all bit-widths and list lw with the other hyper-parameters.
  5. [Limitations] The Limitations section correctly states that cache-size control is imprecise and that evaluation is limited to three models; these caveats should be connected to the main tables, where the reported SpindleKV ratios are intentionally kept slightly below the baselines.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SpindleKV's accuracy claims rest on external LongBench and Needle-in-a-Haystack comparisons, and its compression accounting concern is a correctness issue rather than a derivation that reduces to its own inputs.

full rationale

No significant circularity found. The central accuracy claims are evaluated on external benchmarks against external baselines: LongBench averages in Tables 1/10/11 and Needle-in-a-Haystack scores in Table 4 are measured after compression and compared with PyramidKV/PyramidInfer at the reported reserve ratios, so the reported performance is not derived from the paper's own equations. The codebook construction (Eqs. 8-14 and Algorithm 1) is a transparent greedy vector-quantization procedure: token vectors are normalized, grouped by cosine similarity above a threshold, assigned to codebook entries, and reconstructed as C[r] * m. The reconstruction is defined by the same merge rule that produced the codebook, but that is a compression identity, not a prediction fitted to the evaluation data. The thresholds (theta_K = 0.98, theta_V = 0.95, alpha = 0.525, beta = 0.05) are fixed hyperparameters; the paper does not report sensitivity analysis, but there is no evidence they were fitted to the test benchmarks in a way that makes the comparisons tautological. The preliminary observation of high cosine similarity in shallow layers is motivational rather than load-bearing: the method's success is still measured on held-out tasks. Self-citations (Shi et al. 2024/2025, Ma et al. 2025, Yao et al. 2024a-c) appear only in the literature review and do not carry the argument. The reserve-ratio formula in Eq. 15 is definitional and may under-count GQA-related storage or codebook overhead, but an inaccurate compression metric is a memory-accounting/correctness concern, not a circular derivation; the LongBench and retrieval scores remain external measurements. No step in the derivation reduces, by construction, to its own input.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The contribution rests on two compression hypotheses (shallow-layer similarity and deep-layer attention sparsity), a heuristic codebook construction, and a memory-accounting formula. The main hand-set constants are the two similarity thresholds and the minimum layer ratio. No new physical entities are introduced.

free parameters (5)
  • Key similarity threshold θK = 0.98
    Table 3; controls which key vectors are merged into one codebook entry. No sensitivity analysis is reported.
  • Value similarity threshold θV = 0.95
    Table 3; same role for value vectors. No sensitivity analysis is reported.
  • Minimum layer preserve ratio β = 0.05
    Used in Eq. 5 to set the pyramid allocation between shallow and deep layers; no sensitivity study is provided.
  • Observation window length lw = not stated
    Used in Eq. 2 for accumulated attention scores; the value is not reported, so the eviction criterion is not fully specified.
  • Codebook index bit-width (int_bit) = not stated
    Appears in Eq. 15 memory accounting; without it the effective reserve ratio cannot be reproduced.
assumptions (5)
  • domain assumption Shallow-layer KV vectors can be near-losslessly replaced by representative codebook vectors selected by cosine similarity.
    Central hypothesis from Section 3.2 and Figure 3b; no bound on reconstruction error is given.
  • domain assumption Deep-layer attention concentration justifies top-k eviction based on accumulated attention scores.
    Adopted from PyramidKV and SnapKV; assumed to carry over to SpindleKV's layer allocation.
  • domain assumption Pre-RoPE keys can be normalized, clustered, and reconstructed; re-applying RoPE after reconstruction introduces negligible overhead and error.
    Section 3.4 states the method operates on pre-RoPE K and re-applies RoPE; no error analysis is provided.
  • ad hoc to paper The greedy graph-based codebook construction yields a near-minimal codebook satisfying the similarity constraint.
    Algorithm 1 repeatedly picks the highest-degree node; no optimality or approximation guarantee is given.
  • domain assumption The reserve ratio formula in Eq. 15 is an accurate measure of actual memory usage.
    Underlies all equal-ratio comparisons; not validated against measured GPU memory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers." pith.science (2026). https://pith.science/paper/7Z7OLKLJ

@misc{pith2026250706517,
  author       = {Pith},
  title        = {Pith review of: SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7Z7OLKLJ}},
  note         = {Machine review of arXiv:2507.06517}
}
read the original abstract

Large Language Models (LLMs) have achieved impressive accomplishments in recent years. However, the increasing memory consumption of KV cache has possessed a significant challenge to the inference system. Eviction methods have revealed the inherent redundancy within the KV cache, demonstrating its potential for reduction, particularly in deeper layers. However, KV cache reduction for shallower layers has been found to be insufficient. Based on our observation that, the KV cache exhibits a high degree of similarity. Based on this observation, we proposed a novel KV cache reduction method, SpindleKV, which balances both shallow and deep layers. For deep layers, we employ an attention weight based eviction method, while for shallow layers, we apply a codebook based replacement approach which is learnt by similarity and merging policy. Moreover, SpindleKV addressed the Grouped-Query Attention (GQA) dilemma faced by other attention based eviction methods. Experiments on two common benchmarks with three different LLMs shown that SpindleKV obtained better KV cache reduction effect compared to baseline methods, while preserving similar or even better model performance.

Figures

Figures reproduced from arXiv: 2507.06517 by the authors.

Figure 1
Figure 1. A quick illustration for how SpindleKV works [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of major eviction methods. searches, PyramidInfer (Yang et al., 2024b) and PyramidKV (Cai et al., 2024), show that the cost of evicting tokens from deeper layers is often lower, and the evicted KV cache exhibits a triangular pat￾tern. However, these methods are difficult to inte￾grate with GQA, as they require evaluating token acceptance or eviction for an entire group of Q heads, rather than for each ind… view at source ↗
Figure 3
Figure 3. The distribution of attention weight and cosine similarity in token level cross different layers of LLaMA2- [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: An overview of SpindleKV 3.4 Similarity Based Token Replacement The redundancy that the Eviction method cannot address arises from the high similarity between the constituent of the KV cache in shallower layers. Meanwhile, the operation of unfolding GQA in the previous…
Figure 5
Figure 5. Figure 5: LongBench result on three models cross different reserve ratios. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Visualization of Needle-in-a-Haystack. The vertical axis of the table represents the depth percentage, and [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Cosine similarity in Key.(θ = 0.9) 0 1 2 3 4 5 6 7 8 Similarity Count (log scale) 0 250 500 750 1000 1250 1500 1750 Count of Tokens layer0 layer5 layer18 layer30 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Cosine similarity in Value.(θ = 0.6) Abbreviation Full Name Na.QA narrativeqa Qasp qasper Mu.QA multifieldqa_en Ho.QA hotpotqa Wi.QA 2wikimqa Musq musique Gv.Rp gov_report QMSm qmsum M.New multi_news TREC trec Tr.QA triviaqa SASm samsum PCnt passage_count Pa.Rt passage…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 6 canonical work pages

  1. [1]

    Joshua Ainslie, James Lee - Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \' o n, and Sumit Sanghai. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.298 GQA: training generalized multi-query transformer models from multi-head checkpoints . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Si...

    doi:10.18653/v1/2023.emnlp-main.298 2023

  2. [2]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.172 Longbench: A bilingual, multitask benchmark for long context understanding . In Proceedings of the 62nd Annual Meeting of the Association for Com...

    doi:10.18653/v1/2024.acl-long.172 2024

  3. [3]

    Eleftheria Briakou, Colin Cherry, and George F. Foster. 2023. https://doi.org/10.18653/V1/2023.ACL-LONG.524 Searching for needles in a haystack: On the role of incidental bilingualism in palm's translation capability . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Can...

    doi:10.18653/v1/2023.acl-long.524 2023

  4. [4]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. 2024. https://doi.org/10.48550/ARXIV.2406.02069 Pyramidkv: Dynamic KV cache compression based on pyramidal information funneling . CoRR, abs/2406.02069

    arXiv doi:10.48550/arxiv.2406.02069 2024

  5. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur \' e lien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozi \` e...

    arXiv doi:10.48550/arxiv.2407.21783 2024

  6. [6]

    Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. 2024. https://doi.org/10.48550/ARXIV.2405.14256 Zipcache: Accurate and efficient KV cache quantization with salient token identification . CoRR, abs/2405.14256

    arXiv doi:10.48550/arxiv.2405.14256 2024

  7. [7]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \' e lio Renard Lavaud, Marie - Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth \' e e Lacroix, and William El Sayed. 2023. https://doi.org/...

    arXiv doi:10.48550/arxiv.2310.06825 2023

  8. [8]

    Roman Koshkin, Katsuhito Sudoh, and Satoshi Nakamura. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.27 T rans LL a M a: LLM -based simultaneous translation system . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 461--476, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.findings-emnlp.27 2024

Show all 38 references
  1. [9]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. https://doi.org/10.48550/ARXIV.2404.14469 Snapkv: LLM knows what you are looking for before generation . CoRR, abs/2404.14469

    2024 doi arXiv

  2. [10]

    Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. 2024 a . https://doi.org/10.48550/ARXIV.2405.14366 Minicache: KV cache compression in depth dimension for large language models . CoRR, abs/2405.14366

    2024 doi arXiv

  3. [11]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024 b . https://proceedings.mlr.press/v235/liu24bz.html KIVI : A tuning-free asymmetric 2bit quantization for KV cache . In Proceedings of the 41st International Confer...

    2024

  4. [12]

    Ziyang Ma, Zuchao Li, Lefei Zhang, Gui - Song Xia, Bo Du, Liangpei Zhang, and Dacheng Tao. 2025. https://doi.org/10.48550/ARXIV.2503.23924 Model hemorrhage and the robustness limits of large language models . CoRR, abs/2503.23924

    2025 doi arXiv

  5. [13]

    OpenAI. 2023. https://doi.org/10.48550/ARXIV.2303.08774 GPT-4 technical report . CoRR, abs/2303.08774

    2023 doi arXiv

  6. [14]

    Baptiste Rozi \` e re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \' e r \' e my Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton - Ferrer, Aaron Grattafiori, Wenhan Xiong,...

    2023 doi arXiv

  7. [15]

    Noam Shazeer. 2019. https://arxiv.org/abs/1911.02150 Fast transformer decoding: One write-head is all you need . CoRR, abs/1911.02150

    2019 arXiv

  8. [16]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher R \' e , Ion Stoica, and Ce Zhang. 2023. https://proceedings.mlr.press/v202/sheng23a.html Flexgen: High-throughput generative inference of large language models with a singl...

    2023

  9. [17]

    Luohe Shi, Zuchao Li, Lefei Zhang, Baoyuan Qi, Guoming Liu, and Hai Zhao. 2025. https://openreview.net/forum?id=vQvZQ1wDVN KV -latent: Dimensional-level KV cache reduction with frequency-aware rotary positional embedding . In The 63rd Annual Meeting of the Association for Comp...

    2025

  10. [18]

    Luohe Shi, Hongyi Zhang, Yao Yao, Zuchao Li, and Hai Zhao. 2024. https://doi.org/10.48550/ARXIV.2407.18003 Keep the cost down: A review on methods to optimize llm' s kv-cache consumption . CoRR, abs/2407.18003

    2024 doi arXiv

  11. [19]

    Qian Tao, Wenyuan Yu, and Jingren Zhou. 2024. https://doi.org/10.48550/ARXIV.2410.13212 Asymkv: Enabling 1-bit quantization of KV cache with layer-wise asymmetric quantization configurations . CoRR, abs/2410.13212

    2024 doi arXiv

  12. [20]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 doi arXiv

  13. [21]

    Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, and Mi Zhang. 2024. https://doi.org/10.48550/ARXIV.2406.13035 D2O: dynamic discriminative operations for efficient generative inference of large language models . CoRR, abs/2406.13035

    2024 doi arXiv

  14. [22]

    Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. 2024. https://doi.org/10.48550/ARXIV.2407.08454 Model tells you where to merge: Adaptive KV cache merging for llms on long-context tasks . CoRR, abs/2407.08454

    2024 doi arXiv

  15. [23]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html Chain-of-thought prompting elicits reasoning...

    2022

  16. [24]

    Samuel Williams, Andrew Waterman, and David Patterson. 2009. https://doi.org/10.1145/1498765.1498785 Roofline: an insightful visual performance model for multicore architectures . Commun. ACM, 52(4):65–76

    2009

  17. [25]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. https://openreview.net/forum?id=NG7sS51zVF Efficient streaming language models with attention sinks . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May...

    2024

  18. [26]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024 a . https://doi.org/10.48550/ARXIV.2409.12122 Qwen2.5-math techni...

    2024 doi arXiv

  19. [27]

    Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024 b . https://doi.org/10.18653/V1/2024.FINDINGS-ACL.195 Pyramidinfer: Pyramid KV cache compression for high-throughput LLM inference . In Findings of the Association for Computational Linguistics, ACL ...

    2024 doi

  20. [28]

    June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. 2024 c . https://doi.org/10.48550/ARXIV.2402.18096 No token left behind: Reliable KV cache compression via importance-aware mixed precision quantization . CoRR, a...

    2024 doi arXiv

  21. [29]

    Yifei Yang, Runhan Shi, Zuchao Li, Shu Jiang, Bao-Liang Lu, Yang Yang, and Hai Zhao. 2024 d . https://arxiv.org/abs/2408.10285 Batgpt-chem: A foundation large model for retrosynthesis prediction . Preprint, arXiv:2408.10285

    2024 arXiv

  22. [30]

    Yao Yao, Zuchao Li, and Hai Zhao. 2024 a . https://doi.org/10.18653/V1/2024.FINDINGS-ACL.204 GKT: A novel guidance-based knowledge transfer framework for efficient cloud-edge collaboration LLM deployment . In Findings of the Association for Computational Linguistics, ACL 2024,...

    2024 doi

  23. [31]

    Yao Yao, Zuchao Li, and Hai Zhao. 2024 b . https://doi.org/10.18653/v1/2024.findings-naacl.183 G o T : Effective graph-of-thought reasoning in language models . In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2901--2921, Mexico City, Mexico. Ass...

    2024 doi

  24. [32]

    Yao Yao, Zuchao Li, and Hai Zhao. 2024 c . https://doi.org/10.18653/V1/2024.ACL-LONG.143 Sirllm: Streaming infinite retentive LLM . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, ...

    2024 doi

  25. [33]

    Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. https://doi.org/10.1145/3490099.3511105 Wordcraft: Story writing with large language models . In Proceedings of the 27th International Conference on Intelligent User Interfaces, IUI '22, page 841–852, New York, NY, ...

    2022

  26. [34]

    Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer...

    2022 doi arXiv

  27. [35]

    Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. 2024. https://openreview.net/forum?id=LCTmppB165 Cam: Cache merging for memory-efficient llms inference . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, ...

    2024

  28. [36]

    Barrett, Zhangyang Wang, and Beidi Chen

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \' e , Clark W. Barrett, Zhangyang Wang, and Beidi Chen. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/6ceefa7b15572587b78ecfcebb2827f8-Abstract-C...

    2023

  29. [37]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.