Pith. sign in

REVIEW 5 major objections 5 minor 37 references

Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FourierAttention claims that most KV cache dimensions can be stored as fixed-length Fourier coefficients without hurting long-context accuracy.

desk verdict The paper has an appealing idea but a load-bearing normalization error in the decompression equation, so the reported numbers don't follow from the math as written. read the letter →

arxiv 2506.11886 v1 pith:PHIZODRQ submitted 2025-06-13 cs.CL

Xiaoran Liu , Siyang He , Qiqi Wang , Ruixiao Li , Yuerong Song , Zhigeng Liu , Linlin Li , Qun Liu
show 4 more authors
Zengfeng Huang Qipeng Guo Ziwei He Xipeng Qiu
This is my paper · ORCID
classification cs.CL
keywords KVcachecompressionFouriertransformlong-contextlanguagemodelsattentiondimensionspecializationHiPPOtraining-freeLongBenchneedle-in-a-haystack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that most transformer head dimensions are insensitive to long-range context and can be stored as fixed-length Fourier coefficients rather than per-token values, cutting KV cache memory without retraining. It introduces FourierAttention, which projects the temporal evolution of these 'compressible' dimensions onto a translated Fourier basis during prefilling, keeps only k = 512 coefficients per dimension, and reconstructs the cache on the fly during decoding. On LongBench and Needle-In-A-Haystack with LLaMA3.1-8B and LLaMA3.2-3B, the method stays close to the uncompressed model while other training-free methods degrade. If the claim holds, most of the KV cache can be represented by a constant-size state per compressed dimension, making memory independent of context length for those dimensions.

What carries the argument

The translated Fourier transform (HiPPO-FourierT) with real-valued compression matrix $F \in \mathbb{R}^{2k \times L}$ whose rows are constant, cosine, and sine basis vectors; the forward transform yields fixed-length coefficients and the inverse transform $(1/k)F^{\mathsf{T}}$ reconstructs the cache segment during attention, fused into the FlashAttention kernel. This machinery is what lets the method replace a growing per-token cache with a fixed-size spectral state for each compressed dimension.

What would settle it

Compute the reconstruction MSE of the inverse Fourier transform for every compressed dimension over the full 30k context, aggregated across all layers and heads; if any sizable fraction of dimensions show large MSE, the band-limited assumption is violated. Alternatively, rerun NIAH with k reduced to 64: if retrieval accuracy collapses, the fixed-length representation is genuinely responsible; if it barely changes, the claim that k = 512 is needed loses force.

Watch

Extended reading notes

Core claim

FourierAttention establishes the central claim that KV cache dimensions have heterogeneous roles—lower dimensions track local context, upper dimensions carry long-range dependencies—and that the locally biased dimensions can be replaced by a truncated Fourier representation. Concretely, for each dimension selected for compression, the sequence of K or V values over the middle of the context is replaced by 2k real numbers (k complex Fourier coefficients), and during decoding the inverse Fourier transform rebuilds an approximate cache that is fed to flash attention. The paper reports that with overall 76% of dimensions compressed (90/95% in early layers, 50/70% in the last eight), NIAH scores reach 93.17 and 94.04 on the two LLaMA models, close to the 100.0 of the uncompressed models and above StreamingLLM, SnapKV, Palu, and KIVI.

Load-bearing premise

The assumption that each compressed KV dimension is band-limited—that the first 512 Fourier modes of a roughly 30,000-token span faithfully reproduce the original values—is the load-bearing premise; if a compressed dimension carries high-frequency signal, reconstruction is unfaithful and attention degrades.

Editorial extensions

If this is right

  • Compressed dimensions consume constant memory regardless of context length: 2k = 1024 real numbers each, so for the 76% of compressed dimensions the cache stops growing with L.
  • Long-context retrieval (NIAH) is preserved near the full-attention baseline, suggesting the upper, uncompressed dimensions carry the information needed for needle retrieval.
  • FourierT's shift-invariance and matrix form allow compression in a single pass during prefilling, unlike recurrent HiPPO bases such as LegT.
  • The inverted-pyramid schema implies the optimal compression profile differs from token-eviction heuristics: lower layers and V cache can tolerate more compression.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The actual memory footprint is never measured in the paper: the reported 76% compressed-dimension fraction does not translate into a stated end-to-end memory saving, since initial/local tokens and uncompressed dimensions still scale with L.
  • On the LongBench averages in the reported tables, SnapKV is numerically ahead on both LLaMA models, so the 'best' claim is really carried by NIAH; whether that matters depends on whether retrieval or general long-context understanding is the target.
  • The band-limited assumption predicts a testable scaling: if a compressed dimension has significant high-frequency energy, reconstruction error grows; one could measure the Fourier-energy decay of all compressed dimensions to predict which will fail.
  • The method is orthogonal to quantization: combining fixed-length Fourier states with 4-bit storage of the coefficients could compound savings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes FourierAttention, a training-free KV cache compression method for LLMs. It observes that lower head dimensions attend mostly to local context while upper dimensions carry long-range information, and it compresses the supposedly long-context-insensitive dimensions by projecting their temporal evolution onto a fixed set of Fourier basis functions. The compressed dimensions are stored as a fixed number k=512 of spectral coefficients, while initial and local tokens are preserved in full. At decoding time, the cache is reconstructed with an inverse Fourier transform during attention. The paper reports results on LLaMA3.1-8B and LLaMA3.2-3B on LongBench and NIAH, compares against StreamingLLM, SnapKV, Palu, and KIVI, and claims the best long-context accuracy on average with lower memory consumption. A custom Triton kernel, FlashFourierAttention, is described as in progress for efficiency.

Significance. The paper's core observation—that head dimensions differ in local versus long-range sensitivity—is interesting and potentially useful for dimension-aware KV cache compression. The idea of storing fixed-length Fourier coefficients for compressible dimensions is novel, and the comparison of FourierT versus LegT in Table 3 is a useful ablation. However, as written, the method is not reproducible from the equations, the efficiency claims are not measured, and the NIAH evaluation appears to be coupled to the dimension-selection procedure. If corrected and verified with code, the approach could be a meaningful contribution to training-free KV cache compression.

major comments (5)
  1. [Section 3.3, Eqs. (4) and (6)] The reconstruction equation is not mathematically consistent with the compression matrix. Equation (4) defines an unnormalized real Fourier matrix F, and Eq. (3) sets Kmc = F Kmn. Equation (6) reconstructs with (1/k) F^T Kmc. For a constant signal of length L, the DC coefficient is L·c, so the reconstruction from the DC row is (L/k)·c, not c; with k=512 and L≈29,676 this is a factor of ≈58. The correct least-squares inverse requires diagonal scaling by 1/L (or by 2/L for the nonzero harmonics), not a uniform 1/k. This scale error applies to all compressed dimensions while uncompressed dimensions retain their original scale, so the attention logits in Eq. (6) are distorted. Since no code is provided, the reported NIAH and LongBench scores cannot be reproduced from the equations as written.
  2. [Abstract and Section 4.2] The abstract's claim that FourierAttention 'achieves the best long-context accuracy on LongBench and NIAH' is not supported by the authors' own summary in Section 4.2, which states that on LLaMA3.1-8B the method is 'slightly inferior to SnapKV' on LongBench. To make the claim precise, the authors should state the aggregate statistics across both models and both benchmarks, and temper the abstract accordingly.
  3. [Section 4.2 and Limitations] The central efficiency premise is unmeasured. Section 4.2 says efficiency experiments 'will' be conducted and that the custom Triton kernel 'is still in progress,' and the Limitations section repeats this. The abstract nevertheless claims 'maintaining lower memory consumption' and 'efficient deployment without performance compromise.' No memory footprint, latency, or throughput numbers are reported; the only evidence is the theoretical 76% dimension-compression ratio, which does not by itself establish a memory reduction because the uncompressed dimensions still grow with sequence length and decompression overhead is not quantified.
  4. [Sections 3.4 and 4.1] The compressed-dimension selection is coupled to the evaluation. Section 3.4 says dimensions are chosen by reconstruction MSE, and Section 4.1 says this reconstruction loss is evaluated on the prompt portion of the 32k Needle-In-A-Haystack benchmark. The NIAH results in Figures 4 and 5 are then reported on the same benchmark. This is a form of test-set tuning; at minimum, the authors should use a separate validation set or report selection on held-out prompts, and state clearly which NIAH split was used for selection.
  5. [Section 5.1 and Figure 6] The band-limited assumption underlying the method is validated only indirectly. Figure 6 shows reconstruction for four hand-picked dimensions of layer 0, and Table 3 reports aggregate scores on Multi-Key/Multi-Value NIAH for one model at 4k context. There is no reconstruction-error measurement across all compressed dimensions and layers, and no analysis of how reconstruction error translates into attention error. The claim that the first 512 Fourier modes suffice for 76% of dimensions needs a systematic error analysis before the method can be relied upon.
minor comments (5)
  1. [Eq. (6)] In the reconstruction of the value cache, the formula reads '1/k F^T Kmc' but should read '1/k F^T Vmc'.
  2. [Section 3.1] The references to 'Figure 1(a)' and 'Figure 1(b)' should be to Figure 2(a) and Figure 2(b); Figure 1 is the method overview.
  3. [Section 3.3] The period T used in Eq. (4) is never defined explicitly; the text says the translated window length is set to the maximum context length, but the relationship between T, the compressed segment length, and the total sequence length should be stated.
  4. [Figure 1 caption] The caption contains placeholder '???' text, which should be replaced with actual labels.
  5. [Section 5.2] The ablation in Table 3 is run only on LLaMA3.2-3B at 4k context; the conclusion that the asymmetric compression schema is superior would be stronger with results on the second model and at longer context lengths.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the method is an explicit compression/decompression pipeline, LongBench provides an independent benchmark, and the reported claims do not reduce to their inputs by construction.

full rationale

The paper's core operation is stated explicitly rather than derived: Eq. (3) projects the middle KV segment onto an unnormalized real Fourier matrix, Eq. (6) reconstructs the cache with (1/k)F^T, and attention is then computed over the reconstructed cache. No reported accuracy number is obtained by algebraic manipulation of the compression equations; LongBench and NIAH scores are empirical measurements against external benchmarks. The self-citations present (e.g., He et al. 2023 for Fourier bases and Liu et al. 2025 for long-context LLM behavior) are contextual rather than load-bearing: the Fourier-vs-Legendre choice is independently ablated in Section 5.1, and no uniqueness theorem from the authors' prior work is invoked to rule out alternatives. The dimension-selection criterion in Section 3.4 is reconstruction MSE, so compressibility is defined by the selection objective, but the paper's accuracy claims are not defined through that objective; they are tested downstream. The most serious issue in the paper is a correctness/reproducibility problem, not circularity: with k=512 and sequence length L much larger than k, the 1/k normalization in Eq. (6) is inconsistent with the unnormalized rows of Eq. (4), which would scale reconstructed compressed dimensions by roughly L/k and make the reported scores unreproducible as written. Independent of that, there is minor benchmark coupling in that compression ratios are evaluated using NIAH prompt portions in Section 4.1 and NIAH is subsequently presented as a headline result, but this is hyperparameter selection on a benchmark distribution rather than a constructional equivalence, and LongBench supplies an independent evaluation. Because no load-bearing step reduces by definition or by self-citation to its own inputs, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or architectural entities are postulated; the Fourier coefficients are a lossy representation of existing KV states. The method rests on four hand-set geometry hyperparameters (k, Linit, Llocal, layer-group ratios) plus a data-dependent dimension mask, and on two ad-hoc modeling assumptions: band-limitedness of KV trajectories and transferability of NIAH-derived masks. The local and global dimension bifurcation is presented as an empirical discovery but is only loosely tied to the actual algorithm.

free parameters (4)
  • Number of Fourier states k = 512
    Hand-set fixed length for all compressed dimensions (Section 4.1); this sets both the memory-accuracy tradeoff and the reconstruction ceiling; not ablated or swept.
  • Uncompressed prefix and local window (Linit, Llocal) = 4, 1024
    The initial 4 and last 1024 tokens are always stored in full (Section 4.1); values mirror StreamingLLM's configuration and are not justified or swept.
  • Layer-group compression ratios (K/V) = 90%/95% first 4 layers, 80%/80% middle, 50%/70% last 8 layers
    Chosen from standard-deviation observations and validated with Multi-Key and Multi-Value NIAH ablations (Table 3); effectively tuned on the same task family used for the headline NIAH result.
  • Dimension subset selection = per-model binary masks, chosen by reconstruction MSE on 32k NIAH prompts
    Section 4.1 selects compressed dimensions by reconstruction loss computed on the prompt portion of the 32k NIAH benchmark; this couples the method's choices to the eval distribution; masks are not released.
assumptions (4)
  • ad hoc to paper KV cache temporal trajectories are approximately band-limited: the first k=512 Fourier modes (over window T) suffice to reconstruct the parts of the cache that matter
    The entire compression is a truncation to k modes (Equation 4, Section 3.3). The paper shows low MSE only for four hand-picked dimensions of layer 0 (Figure 6) and never quantifies reconstruction error across all compressed dims or its effect on attention.
  • domain assumption A fixed hard split exists between local dims (0-69) and long-range dims (70-127) across heads and layers
    Section 3.1 and Figure 2 support this with attention-score visualization and a Gaussian-noise experiment with unspecified noise scale; it is a motivation, while the actual algorithm uses MSE-based selection (Section 3.4), so the two stories are only loosely connected.
  • ad hoc to paper Reconstruction loss measured on NIAH prompts transfers to other long-context tasks
    Dimension masks are computed on the 32k NIAH prompt distribution (Section 4.1) and then applied to LongBench; no cross-distribution validation of the masks is given.
  • domain assumption Reconstruct-then-attend (Equation 6) with standard FlashAttention2 is a faithful proxy for the intended fused kernel
    All accuracy numbers come from materializing the decompressed cache and running stock attention, since FlashFourierAttention 'is still in progress' (Section 3.3); the proxy's numerical behavior equals the target only if the fused kernel is bitwise equivalent, which is unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache." pith.science (2026). https://pith.science/paper/PHIZODRQ

@misc{pith2026250611886,
  author       = {Pith},
  title        = {Pith review of: Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PHIZODRQ}},
  note         = {Machine review of arXiv:2506.11886}
}
read the original abstract

Large Language Models struggle with memory demands from the growing Key-Value (KV) cache as context lengths increase. Existing compression methods homogenize head dimensions or rely on attention-guided token pruning, often sacrificing accuracy or introducing computational overhead. We propose FourierAttention, a training-free framework that exploits the heterogeneous roles of transformer head dimensions: lower dimensions prioritize local context, while upper ones capture long-range dependencies. By projecting the long-context-insensitive dimensions onto orthogonal Fourier bases, FourierAttention approximates their temporal evolution with fixed-length spectral coefficients. Evaluations on LLaMA models show that FourierAttention achieves the best long-context accuracy on LongBench and Needle-In-A-Haystack (NIAH). Besides, a custom Triton kernel, FlashFourierAttention, is designed to optimize memory via streamlined read-write operations, enabling efficient deployment without performance compromise.

Figures

Figures reproduced from arXiv: 2506.11886 by the authors.

Figure 1
Figure 1. Overview of FourierAttention. process all head dimensions, FourierAttention iden￾tifies localized, context-insensitive dimensions in KV states and approximates their temporal evolu￾tion via a fixed set of orthogonal Fourier basis func￾tions. By retaining only the dominant Fourier coef￾ficients (k ≪ L, where L is the sequence length), our method projects sequences into a compact spec￾tral representation. Unlike polyn… view at source ↗
Figure 2
Figure 2. Visualization of the average attention score and its components in LLaMA3.1-8B ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualization of standard deviation of KV [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Results of LLaMA3.1-8B (Dubey et al., 2024) on Needle-In-A-Haystack (Kamradt, 2023). Fourier￾Attention achieves a highest average score over StreamingLLM (Xiao et al., 2024), SnapKV (Li et al., 2024b), Palu (Chang et al., 2024), and KIVI (Liu et al., 2024c) and shows c…
Figure 5
Figure 5. Figure 5: Results of LLaMA3.2-3B (Meta, 2024a) on Needle-In-A-Haystack (Kamradt, 2023). FourierAttention achieves a highest average score over StreamingLLM (Xiao et al., 2024), SnapKV (Li et al., 2024b), Palu (Chang et al., 2024), and KIVI (Liu et al., 2024c) and shows closest p…
Figure 6
Figure 6. Figure 6: Visualization of KV cache reconstruction in LLaMA3.2-3B ( [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The statistics of each dimension selected for compression, averaged across attention heads in different [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 8 canonical work pages

  1. [1]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \'o n, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245

    arXiv 2023

  2. [2]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, and 1 others. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508

    arXiv 2023

  3. [3]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and 1 others. 2024. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069

    arXiv 2024

  4. [4]

    Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. 2024. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118

    arXiv 2024

  5. [5]

    OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass

    2023

  6. [6]

    Tri Dao. 2024. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations

    work page 2024

  7. [7]

    Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. 2023. https://crfm.stanford.edu/2023/10/12/flashdecoding.html Flash-decoding for long-context inference

    work page 2023

  8. [8]

    Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. 2024. Skvq: Sliding-window key and value cache quantization for large language models. arXiv preprint arXiv:2405.06219

    arXiv 2024

Show all 37 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    Yao Fu. 2024. Challenges in deploying long-context transformers: A theoretical peak performance analysis. arXiv preprint arXiv:2405.08944

    2024 arXiv

  3. [11]

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher R \'e . 2020. Hippo: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems, 33:1474--1487

    2020

  4. [12]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  5. [13]

    Ziwei He, Meng Yang, Minwei Feng, Jingcheng Yin, Xinbing Wang, Jingwen Leng, and Zhouhan Lin. 2023. Fourier transformer: Fast long range modeling by removing sequence redundancy with fft operator. arXiv preprint arXiv:2305.15099

    2023 arXiv

  6. [14]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079

    2024 arXiv

  7. [15]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654

    2024 arXiv

  8. [16]

    Greg Kamradt. 2023. Needle in a haystack - pressure testing llms. https://github.com/gkamradt/LLMTest_NeedleInAHaystack

    2023

  9. [17]

    Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. 2024 a . Needlebench: Can llms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.11963

    2024

  10. [18]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024 b . Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469

    2024 arXiv

  11. [19]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, and 1 others. 2024 a . Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434

    2024 arXiv

  12. [20]

    Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, Siyang He, Qiqi Wang, Linlin Li, Qun Liu, and 1 others. 2025. Thus spake long-context large language model. arXiv preprint arXiv:2502.17129

    2025

  13. [21]

    Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. 2024 b . Scaling laws of rope-based extrapolation. In The Twelfth International Conference on Learning Representations

    2024

  14. [22]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024 c . Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750

    2024 arXiv

  15. [23]

    AI Meta. 2024 a . Introducing meta llama 3: The most capable openly available llm to date. Meta AI

    2024

  16. [24]

    AI Meta. 2024 b . Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. Meta AI

    2024

  17. [25]

    OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    2023 arXiv

  18. [26]

    OpenAI. 2024. https://openai.com/o1/ O1: Openai's first model . Accessed: 2024-12-25

    2024

  19. [27]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations

    2024

  20. [28]

    Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. 2024. Eigen attention: Attention in low-rank space for kv cache compression. arXiv preprint arXiv:2408.05646

    2024 arXiv

  21. [29]

    Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, and 5 others. 2024. https://doi.o...

    2024 doi

  22. [30]

    Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10--19

    2019

  23. [31]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  24. [32]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations

    2024

  25. [33]

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and 1 others. 2024. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247

    2024 arXiv

  26. [34]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. 2025. Native sparse attention: Hardware-aligned and natively trainable sparse att...

    2025 arXiv

  27. [35]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, and 1 others. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information...

    2023

  28. [36]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  29. [37]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.