Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

CommVQ: Commutative Vector Quantization for KV Cache Compression

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CommVQ compresses the KV cache of long-context LLMs to roughly 1–2 bits per scalar with near-lossless benchmark accuracy by vector-quantizing keys and values with a RoPE-commutative codebook.

desk verdict CommVQ has a genuinely clever commutative-codebook idea and the math holds up, but the paper overclaims: its own Table 2 contradicts the '1-bit with minimal accuracy loss' headline. read the letter →

arxiv 2506.18879 v1 pith:TULFDY5N submitted 2025-06-23 cs.CL cs.AI

Junyan Li , Yang Zhang , Muhammad Yusuf Hassan , Talha Chafekar , Tianle Cai , Zhile Ren , Pengsheng Guo , Foroozan Karimzadeh
show 3 more authors
Colorado Reed Chong Wang Chuang Gan
This is my paper · ORCID
classification cs.CLcs.AI
keywords KVcachecompressionvectorquantizationadditiveRoPElong-contextLLMinferencecodebooklearning1-bit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the KV cache—the per-token key and value tensors that dominate GPU memory during long-context LLM inference—can be quantized at the vector level to roughly one to two bits per scalar without the accuracy collapse that scalar 1-bit and 2-bit quantization suffers. The proposed method, CommVQ, encodes each key and value vector into binary codes with a lightweight encoder and decodes them through a learned codebook, and it constrains the codebook's 2x2 blocks to commute with RoPE rotation matrices so the costly query-key products can be precomputed and reused. The authors report that 2-bit CommVQ stays within fractions of a point of FP16 on LongBench and GSM8K while cutting KV cache memory by 87.5%, and that 1-bit CommVQ remains usable where scalar quantizers collapse. If these results hold, long-context inference at 128K tokens could run on a single consumer GPU, making vector quantization a practical KV-cache compression layer.

What carries the argument

The central object is the RoPE-commutative codebook. RoPE applies a block-diagonal rotation $R_m^i$ to each 2-dimensional sub-vector of a key; the codebook stores, per subspace, matrices $C=\begin{pmatrix}x&y\\-y&x\end{pmatrix}$ that satisfy $R_m^i C = C R_m^i$. Because of this commutativity, the attention score between the current query and the $i$-th quantized key can be reordered so that the query-dependent and codebook-dependent factors are computed once and reused across all cached positions. The codebook is fit by an EM-style clustering algorithm on a calibration set; consecutive sub-vectors share one quantized index (group size $g$), and $R$ residual rounds refine the reconstruction, giving an average bit rate of $R\log_2(N_c')/g$.

What would settle it

Run CommVQ-2 on the InfiniteBench KVRetrieval task, whose random UUID key-value pairs are absent from the FineWeb-Edu calibration set, and track retrieval accuracy at 32K, 64K, and 128K contexts against FP16. The paper's own table already shows CommVQ-2 at 12.20 versus FP16 at 55.20 on this task; a systematic length sweep would show whether the gap widens with context length, and whether the 1-bit variant retains any retrieval signal at all.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that KV-cache quantization does not have to be per-scalar to be fast and accurate; per-vector additive quantization with a codebook constrained to commute with RoPE can be integrated into self-attention at modest overhead. The paper reports that CommVQ-2 keeps LongBench average at 47.98 versus the FP16 baseline's 48.05 and GSM8K at 76.04 versus 76.27, while storing the cache at 2 bits per scalar (an 87.5% cut from FP16), and that CommVQ-1 retains LongBench 44.94, InfiniteBench 36.76, and GSM8K 66.57, where scalar-quantization baselines collapse. The method also reports real memory savings: a 120K-context LLaMA-3.1-8B-Instruct cache fits in about 20GB instead of 60GB, and decoding latency drops 6x to 9.6x versus the naive implementation of the same quantization.

Load-bearing premise

The load-bearing premise is that a codebook and encoder fit on a sample of FineWeb-Edu text remain accurate for the keys and values that matter on unseen long-context, retrieval, and math data; the paper's evidence is perplexity on a few shifted domains and single-run task scores, not a distributional guarantee.

Editorial extensions

If this is right

  • At 2 bits per scalar, CommVQ matches the FP16 baseline on LongBench (47.98 vs 48.05) and GSM8K (76.04 vs 76.27) while reducing the FP16 KV cache by 87.5%.
  • At 1 bit per scalar, CommVQ keeps average LongBench at 44.94 and InfiniteBench at 36.76, well above the best prior 1-bit baseline (27.42 and 18.48), and keeps GSM8K at 66.57.
  • A LLaMA-3.1-8B-Instruct model with a 120K-token context needs about 20GB of KV cache under CommVQ-1 instead of 60GB in FP16, and at 32K context the batch size scales from 8 to 128 without OOM.
  • Reordering attention so that softmax scores multiply the quantized value codes before the codebook multiply cuts value-decoding cost by a factor close to the hidden dimension, and the commutative key codebook makes overall decoding overhead roughly (R+1)/2 times vanilla attention instead of Nc times.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The rotation-commutativity trick generalizes: any score computation that applies RoPE-style rotations to a sum of codebook vectors can precompute the rotated inner products once, so long-context cross-attention and retrieval heads built on the same structure should inherit the speedup.
  • Because the codebook is fixed-size while the cache grows with context, there is a context-length threshold below which codebook memory (2–5 MB for LLaMA-3.1-8B) outweighs the savings; computing that threshold per model would tell practitioners when CommVQ is worthwhile.
  • The 1-bit results on retrieval tasks suggest the residual codebook preserves coarse attention patterns but blurs fine-grained token identity; a testable next step is to spend more residual rounds on the key sub-vectors that retrieval-heavy attention heads actually use.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CommVQ, a KV-cache compression method based on additive vector quantization. Value and key vectors are encoded into compact indices and decoded with learned codebooks; the key codebook is restricted to 2x2 blocks of the form [x y; -y x] that commute with RoPE rotation matrices, allowing part of the attention computation to be precomputed and reused. The codebooks and encoder are trained on a FineWeb-Edu subset with an EM-style clustering objective, and the method is evaluated on LongBench, InfiniteBench, GSM8K, and NIAH with LLaMA-3.1-8B-Instruct, plus additional experiments with LLaMA-2 and Mistral. The authors report near-lossless 2-bit compression on LongBench and state that 1-bit compression achieves minimal accuracy loss while outperforming competing KV quantization baselines.

Significance. If the results hold after revision, the RoPE-commutative codebook idea is a useful contribution: the commutativity derivation is mathematically sound, the EM update has a closed form, and the paper provides a Triton implementation and code release. The comparison with KIVI, KVQuant, and VQLLM using their official implementations is a strength. However, the paper's headline claims go beyond what its own tables support, especially for 1-bit quantization and retrieval tasks, and the motivating memory calculations are inconsistent with the stated KV dimension. The core idea is promising, but the presentation and claims need substantial revision.

major comments (3)
  1. [Abstract and Section 5.2, Tables 2-3] The unqualified claim of '1-bit KV cache quantization with minimal accuracy loss' is contradicted by the paper's own results. On InfiniteBench (Table 2), CommVQ-1 achieves 0.00 on R.KV versus 55.20 for FP16, 62.37 on R.Num versus 99.49, and an average of 36.76 versus 48.67; even CommVQ-2 drops R.KV to 12.20. On GSM8K (Table 3), CommVQ-1 drops to 66.57 from 76.27. Section 5.2's statement that for retrieval 'ours retains some capacity even at low quantization levels' is not true for R.KV at 1-bit. The claims need to be qualified: 2-bit is near-lossless on LongBench (average 47.98 vs 48.05) but not on InfiniteBench, and 1-bit is a high-compression regime with substantial degradation on retrieval and math tasks.
  2. [Section 1, Section 5.6, Figure 3] The motivating memory numbers are inconsistent with the paper's own KV dimension d=1024. For LLaMA-3.1-8B with 32 layers, batch 2, and 128K context, the FP16 KV cache is approximately 34.4 GB (2 x 131072 x 32 x 2 x 1024 x 2 bytes), not the claimed 88 GB. Figure 3(a) reports 60 GB for a 120K context at batch size 1, while the same calculation gives approximately 15.7 GB. This inflates the FP16 baseline by roughly 2.5-3.8x and should be corrected, as it directly supports the paper's motivation for extreme compression.
  3. [Section 5.5, Table 6] The domain-shift robustness analysis uses perplexity only. Perplexity is insensitive to the loss of exact key-value identity, as demonstrated by Table 2, where R.KV collapses to 0.00 at 1-bit while Table 6 shows a PPL increase of only 0.79 on KVRetrieval. To support the statement that the method is 'robust and generalizable under domain shifts,' the authors should report downstream retrieval accuracy on the shifted domains, or temper the claim to PPL-based robustness.
minor comments (5)
  1. [Section 5.1 and Table 4] The text names 'LLaMA-2-8B' and 'Mistral-8B', but Table 4 lists 'Llama-2-7B' and 'Mistral-7B'; please align the model names.
  2. [Appendix A.4, Table 9] In Table 9, the last row (g=64, Nc'=64, R=1) has average bit 6/64 = 0.09375, not the 0.125 of the other rows, so the phrase 'maintaining a consistent Avg. bit' is inaccurate; please correct or clarify.
  3. [Section 4.2, Eq. (18)] The notation [s^j_i = l] is described as a 2-dimensional boolean indicator vector, which is nonstandard and easy to confuse with a scalar indicator; please define explicitly how the two dimensions are used.
  4. [Section 4.1 and 4.2] Section 4.1 uses s_i in {0,1}^{Nc} for value vectors, while Section 4.2 uses a different index-sharing representation with 2-dimensional entries for keys; please add a sentence clarifying that the value and key encodings are distinct.
  5. [Section 5.6, Table 5] The latency comparison in Table 5 lacks measurement details such as GPU model, kernel configuration, and averaging methodology; please report them.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CommVQ's codebook is learned on a calibration set and evaluated on held-out benchmarks; no prediction reduces to its inputs.

full rationale

CommVQ's derivation is self-contained with respect to its evaluation: the encoder and RoPE-commutative codebooks are learned on a FineWeb-Edu subset (Sec. 5.1, Appendix A.2), and the reported LongBench, InfiniteBench, and GSM8K numbers come from held-out benchmarks, not from the calibration data or from an objective that includes those task scores. The codebook's commutative form (Eq. 7) is a mathematical construction from the rotation structure of RoPE, and the bit-rate formula (Eq. 21) is a definition of storage cost, not a fitted prediction. Hyperparameters g, Nc', and R are chosen by MSE ablations on the calibration set (Appendix A.4), so there is no fitted-input-called-prediction pattern. The paper contains no load-bearing self-citation: the cited VQ, AQ, and RVQ results are external background, and the EM update is derived in Appendix A.2 rather than imported from the authors' prior work. The abstract's 'minimal accuracy loss' claim is strained by the paper's own Table 2 (e.g., CommVQ-1 scores 0.00 on InfiniteBench R.KV vs. 55.20 for FP16), but that is an evidence/claim mismatch, not circularity, because the low retrieval score is an empirical outcome rather than an input to the method's construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the four free hyperparameters listed above plus the standard algebraic and domain assumptions. No new physical or conceptual entities are introduced. The codebook entries themselves are learned parameters, but they are the method rather than ad hoc fitted constants.

free parameters (4)
  • Nc (value codebook rows) = 1024 for 1-bit, 2048 for 2-bit
    Sets the value-side average bit width to Nc/d; chosen to reach the target compression rate (Appendix A.4, Table 8).
  • Nc' (key codebook quantization levels) = 64
    Number of 2x2 commutative codebook entries per subspace; chosen by ablation on key MSE (Appendix A.4, Tables 9-10).
  • R (residual quantization rounds for keys) = 11 for 1-bit, 21 for 2-bit
    Number of iterative clustering passes on residuals; balances accuracy and compression (Appendix A.4, Figure 4).
  • g (group size of shared indices) = 64
    Number of consecutive 2D sub-vectors sharing the same quantized indices; chosen by ablation (Appendix A.4, Tables 9-10).
assumptions (4)
  • domain assumption RoPE is applied as block-diagonal 2x2 rotation matrices (Eqns 5-6).
    True for LLaMA, Mistral, Qwen, which the paper targets. If a model used a different positional embedding, the commutative codebook would not apply.
  • standard math Matrices of the form [[x,y],[-y,x]] commute with all 2D rotation matrices (Eqn 8).
    Standard algebraic property of the complex-number representation; the paper states it as Property 1.
  • standard math The EM-like alternating assignment/update (Algorithm 1) converges to a usable local optimum of the clustering objective (Eqn 20).
    Standard k-means-style alternating optimization; no convergence guarantee beyond local. The paper adds soft assignment and annealing to stabilize.
  • domain assumption The calibration set (FineWeb-Edu subset) is representative of test-time KV cache distributions.
    Needed for the codebook and encoder to generalize; Table 6 checks perplexity on a few shifted domains but not task-score generalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CommVQ: Commutative Vector Quantization for KV Cache Compression." pith.science (2026). https://pith.science/paper/TULFDY5N

@misc{pith2026250618879,
  author       = {Pith},
  title        = {Pith review of: CommVQ: Commutative Vector Quantization for KV Cache Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TULFDY5N}},
  note         = {Machine review of arXiv:2506.18879}
}
read the original abstract

Large Language Models (LLMs) are increasingly used in applications requiring long context lengths, but the key-value (KV) cache often becomes a memory bottleneck on GPUs as context grows. To address this, we propose Commutative Vector Quantization (CommVQ) to significantly reduce memory usage for long-context LLM inference. We first introduce additive quantization with a lightweight encoder and codebook to compress the KV cache, which can be decoded via simple matrix multiplication. To further reduce computational costs during decoding, we design the codebook to be commutative with Rotary Position Embedding (RoPE) and train it using an Expectation-Maximization (EM) algorithm. This enables efficient integration of decoding into the self-attention mechanism. Our approach achieves high accuracy with additive quantization and low overhead via the RoPE-commutative codebook. Experiments on long-context benchmarks and GSM8K show that our method reduces FP16 KV cache size by 87.5% with 2-bit quantization, while outperforming state-of-the-art KV cache quantization methods. Notably, it enables 1-bit KV cache quantization with minimal accuracy loss, allowing a LLaMA-3.1 8B model to run with a 128K context length on a single RTX 4090 GPU. The source code is available at: https://github.com/UMass-Embodied-AGI/CommVQ.

Figures

Figures reproduced from arXiv: 2506.18879 by the authors.

Figure 1
Figure 1. An illustration of vector quantization for KV cache compression. Each vector in the KV cache, corresponding to a token, is first encoded into a low-bitwidth representation, signifi￾cantly reducing storage size. This process is applied to all vectors in the KV cache individually. When needed, the compressed repre￾sentations are loaded and decoded back to the original KV cache using a codebook. summation of several ve… view at source ↗
Figure 2
Figure 2. Needle-in-a-Haystack test using the LLaMA-3.1 8B model. We present the test result for FP16 baseline (top-left), KIVI-1 (bottom-left), CommVQ-2 (top-right) and CommVQ-1 (bottom-right). Our method’s 2-bit version matches FP16 performance, while the 1-bit CommVQ variant outperforms KIVI, demonstrating strong retrieval fidelity under extreme compression. Method Avg. bit (↓) GSM8K (↑) FP16 Baseline 16 76.27 KIVI-2 3.00 … view at source ↗
Figure 3
Figure 3. Per-token decoding memory usage of CommVQ (1-bit) compared to FP16 model. Experiments are conduct on LLaMA￾3.1-8B-Instruct model. validating the commutative codebook’s effectiveness in re￾ducing computational overhead. We also implement Triton kernels to achieve real memory savings [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ablation study on R while keeping g = 64 and Nc ′ = 64. Next, we keep g = 64 and Nc ′ = 64, and vary R to get the quantization error for different compression rate. The result is shown in [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

    cs.LG 2026-08 conditional novelty 7.0 of 10

    A non-orthogonal attention-aware transform plus fixed-width vector quantization keeps 2-bit KV caches accurate on long-context retrieval, where scalar 2-bit methods degrade or collapse.

  2. WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

    cs.AR 2026-07 conditional novelty 7.0 of 10

    A per-request sound upper bound on KV-cache quantization error, implemented as a runtime meter that gates and repairs compression to restore quality.

  3. When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Under dynamic W4A4KV4 quantisation, a head-shared RoPE-aligned pairwise rotation that exactly minimizes a pooled variance surrogate still yields higher perplexity than full-head Hadamard mixing in all evaluated comparisons.

  4. C$^2$KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

    cs.CL 2026-07 conditional novelty 6.0 of 10

    C2KV trains a sidecar extractor to produce compressed, position-agnostic KV cache segments that can be concatenated at inference time, reducing prefill, storage, and memory-bandwidth costs without modifying the frozen...

Reference graph

Works this paper leans on

20 extracted references · 3 canonical work pages · cited by 4 Pith papers

  1. [6]

    Categorical repa- rameterization with gumbel-softmax

    Jang, E., Gu, S., and Poole, B. Categorical repa- rameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144,

    arXiv

  2. [7]

    We calculate the codebook size based on LLaMA-3.1-8B-Instruct model

    Analysis on Codebook Size. We calculate the codebook size based on LLaMA-3.1-8B-Instruct model. 1 bit 2 bit Nc 1024 2048 R 11 21 Nc′ 64 64 Table

    work page

  3. [9]

    Residual vector quantization for kv cache compression in large language model

    Kumar, A. Residual vector quantization for kv cache compression in large language model. arXiv preprint arXiv:2410.15704,

    arXiv

  4. [10]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

    2000

  5. [11]

    The MSE is evaluated on a small subset of the FineWeb-Edu dataset (Lozhkov et al., 2024)

    Comparison of MSE between CommVQ and asymmetric quantization used in KIVI (Liu et al., 2024b), calculated using the cached value matrix from the first layer of LLaMA-3.1-8B-Instruct. The MSE is evaluated on a small subset of the FineWeb-Edu dataset (Lozhkov et al., 2024). 15

    work page 2024

  6. [12]

    Shi, L., Zhang, H., Yao, Y ., Li, Z., and Zhao, H

    URL https://huggingface.co/ datasets/HuggingFaceFW/fineweb-edu. Shi, L., Zhang, H., Yao, Y ., Li, Z., and Zhao, H. Keep the cost down: A review on methods to optimize llm’s kv-cache consumption. arXiv preprint arXiv:2407.18003,

    arXiv

  7. [13]

    Llama 2: Open foundation and fine- tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 ,

    arXiv

  8. [15]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115,

    arXiv

Show all 20 references
  1. [16]

    Kv cache compression, but what must we give in return? a compre- hensive benchmark of long context capable approaches

    Yuan, J., Liu, H., Zhong, S., Chuang, Y .-N., Li, S., Wang, G., Le, D., Jin, H., Chaudhary, V ., Xu, Z., et al. Kv cache compression, but what must we give in return? a compre- hensive benchmark of long context capable approaches. arXiv preprint arXiv:2407.01527,

    arXiv

  2. [17]

    Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization

    Zhang, T., Yi, J., Xu, Z., and Shrivastava, A. Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization. arXiv preprint arXiv:2405.03917, 2024a. Zhang, X., Chen, Y ., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, ...

    arXiv

  3. [19]

    g Nc′ R MSE 8 2 1 0.2699 16 4 1 0.2011 32 16 1 0.1265 64 64 1 0.0906 Table

    2011

  4. [1984]

    W., Shao, Y

    Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y . S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length llm in- ference with kv cache quantization. arXiv preprint arXiv:2401.18079,

    arXiv

  5. [1996]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  6. [2000]

    Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V ., Xu, Z., Kyril- lidis, A., and Shrivastava, A

    Morgan Kaufmann. Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V ., Xu, Z., Kyril- lidis, A., and Shrivastava, A. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, ...

    arXiv

  7. [2010]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  8. [2014]

    Longbench: A bilingual, multitask benchmark for long context under- standing

    Bai, Y ., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context under- standing. arXiv preprint arXiv:2308.14508,

    arXiv

  9. [2021]

    L., Zhang, C., Xu, Y ., Shang, N., Xu, J., Yang, F., and Yang, M

    Ding, Y ., Zhang, L. L., Zhang, C., Xu, Y ., Shang, N., Xu, J., Yang, F., and Yang, M. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753,

    arXiv

  10. [2022]

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads

    10 CommVQ: Commutative Vector Quantization for KV Cache Compression Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y ., and Han, S. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.10819,

    arXiv

  11. [2023]

    Llm maybe longlm: Self- extend llm context window without tuning.arXiv preprint arXiv:2401.01325,

    Jin, H., Han, X., Yang, J., Jiang, Z., Liu, Z., Chang, C.- Y ., Chen, H., and Hu, X. Llm maybe longlm: Self- extend llm context window without tuning.arXiv preprint arXiv:2401.01325,

    arXiv

  12. [2024]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.