{"id":"a2d65705-7751-4a63-a1d6-ddb0cf4860bb","arxiv_id":"2506.18879","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CommVQ compresses LLM key-value caches to roughly one bit per value using additive quantization with a codebook designed to commute with rotary position embeddings, preserving most benchmark accuracy.","lead":"A new compression method, CommVQ, shrinks the key-value cache of large language models down to about one bit per number, cutting memory for long-context inference by over 90 percent. It keeps accuracy close to the full-precision model on several long-context and math benchmarks, and could let 128K-token models run on a single consumer GPU.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's '1-bit quantization with minimal accuracy loss' is contradicted by Table 2: CommVQ-1 scores 0.00 on InfiniteBench R.KV (FP16: 55.20) and 36.76 average vs 48.67, so the paper's own data refute the unqualified claim.","rationale":"The strongest claim is the abstract's 'minimal accuracy loss' at 1-bit and 'lossless accuracy on most tasks' at 2-bit. The most direct threat is not an external distributional worry but the paper's own Table 2: exact retrieval drops catastrophically. This is load-bearing because retrieval fidelity is precisely what KV cache compression must preserve for long-context applications, and the paper highlights retrieval as a strength. The domain-shift analysis (Table 6) measures perplexity, which can stay close while exact-match keys are destroyed, so it does not defuse the issue. A focused R.KV re-run with attention-mass analysis would settle whether this is a benchmark artifact or a structural limitation of the shared-index commutative codebook. Because the reader already flagged the overstatement and issued a conditional verdict, my read does not change the verdict; it sharpens the condition: the claims about lossless/minimal-loss accuracy must be qualified to exclude exact retrieval tasks.","tokens_in":16201,"tokens_out":20188,"duration_ms":211348,"concrete_test":"Run CommVQ-1 and FP16 on InfiniteBench R.KV with 1,000 random UUID key-value pairs and several context lengths (e.g., 32K/64K/128K), reporting exact-match accuracy and the attention mass allocated to the correct key. If exact-match remains near zero while FP16 stays above 50% and the correct key's attention mass is not preserved under quantization, the 'minimal accuracy loss' claim should be removed from the abstract and conclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CommVQ enables '1-bit KV cache quantization with minimal accuracy loss' is not supported by the paper's own evaluation. In Table 2, CommVQ-1 achieves 0.00 on R.KV (exact UUID retrieval) versus 55.20 for the FP16 baseline, 62.37 on R.Num versus 99.49, and an average of 36.76 versus 48.67; the 2-bit variant also drops R.KV to 12.20. Section 5.2 states that for retrieval tasks 'ours retains some capacity even at low quantization levels,' but the R.KV column is identically zero for CommVQ-1. The robustness argument in Section 5.5/Table 6 uses perplexity only, and perplexity is insensitive to the loss of a few critical exact-match key-value identities; it does not establish that the RoPE-commutative codebook, which shares one quantized index across 64 consecutive sub-vectors (Sec. 4.2), preserves the precise key information attention needs for retrieval. Because the headline claim is unqualified ('minimal accuracy loss'), the paper's own numbers are the load-bearing counterexample.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CommVQ, a KV-cache compression method based on additive vector quantization. Value and key vectors are encoded into compact indices and decoded with learned codebooks; the key codebook is restricted to 2x2 blocks of the form [x y; -y x] that commute with RoPE rotation matrices, allowing part of the attention computation to be precomputed and reused. The codebooks and encoder are trained on a FineWeb-Edu subset with an EM-style clustering objective, and the method is evaluated on LongBench, InfiniteBench, GSM8K, and NIAH with LLaMA-3.1-8B-Instruct, plus additional experiments with LLaMA-2 and Mistral. The authors report near-lossless 2-bit compression on LongBench and state that 1-bit compression achieves minimal accuracy loss while outperforming competing KV quantization baselines.","tokens_in":16427,"tokens_out":7989,"duration_ms":76084,"significance":"If the results hold after revision, the RoPE-commutative codebook idea is a useful contribution: the commutativity derivation is mathematically sound, the EM update has a closed form, and the paper provides a Triton implementation and code release. The comparison with KIVI, KVQuant, and VQLLM using their official implementations is a strength. However, the paper's headline claims go beyond what its own tables support, especially for 1-bit quantization and retrieval tasks, and the motivating memory calculations are inconsistent with the stated KV dimension. The core idea is promising, but the presentation and claims need substantial revision.","major_comments":[{"comment":"The unqualified claim of '1-bit KV cache quantization with minimal accuracy loss' is contradicted by the paper's own results. On InfiniteBench (Table 2), CommVQ-1 achieves 0.00 on R.KV versus 55.20 for FP16, 62.37 on R.Num versus 99.49, and an average of 36.76 versus 48.67; even CommVQ-2 drops R.KV to 12.20. On GSM8K (Table 3), CommVQ-1 drops to 66.57 from 76.27. Section 5.2's statement that for retrieval 'ours retains some capacity even at low quantization levels' is not true for R.KV at 1-bit. The claims need to be qualified: 2-bit is near-lossless on LongBench (average 47.98 vs 48.05) but not on InfiniteBench, and 1-bit is a high-compression regime with substantial degradation on retrieval and math tasks.","section":"Abstract and Section 5.2, Tables 2-3"},{"comment":"The motivating memory numbers are inconsistent with the paper's own KV dimension d=1024. For LLaMA-3.1-8B with 32 layers, batch 2, and 128K context, the FP16 KV cache is approximately 34.4 GB (2 x 131072 x 32 x 2 x 1024 x 2 bytes), not the claimed 88 GB. Figure 3(a) reports 60 GB for a 120K context at batch size 1, while the same calculation gives approximately 15.7 GB. This inflates the FP16 baseline by roughly 2.5-3.8x and should be corrected, as it directly supports the paper's motivation for extreme compression.","section":"Section 1, Section 5.6, Figure 3"},{"comment":"The domain-shift robustness analysis uses perplexity only. Perplexity is insensitive to the loss of exact key-value identity, as demonstrated by Table 2, where R.KV collapses to 0.00 at 1-bit while Table 6 shows a PPL increase of only 0.79 on KVRetrieval. To support the statement that the method is 'robust and generalizable under domain shifts,' the authors should report downstream retrieval accuracy on the shifted domains, or temper the claim to PPL-based robustness.","section":"Section 5.5, Table 6"}],"minor_comments":[{"comment":"The text names 'LLaMA-2-8B' and 'Mistral-8B', but Table 4 lists 'Llama-2-7B' and 'Mistral-7B'; please align the model names.","section":"Section 5.1 and Table 4"},{"comment":"In Table 9, the last row (g=64, Nc'=64, R=1) has average bit 6/64 = 0.09375, not the 0.125 of the other rows, so the phrase 'maintaining a consistent Avg. bit' is inaccurate; please correct or clarify.","section":"Appendix A.4, Table 9"},{"comment":"The notation [s^j_i = l] is described as a 2-dimensional boolean indicator vector, which is nonstandard and easy to confuse with a scalar indicator; please define explicitly how the two dimensions are used.","section":"Section 4.2, Eq. (18)"},{"comment":"Section 4.1 uses s_i in {0,1}^{Nc} for value vectors, while Section 4.2 uses a different index-sharing representation with 2-dimensional entries for keys; please add a sentence clarifying that the value and key encodings are distinct.","section":"Section 4.1 and 4.2"},{"comment":"The latency comparison in Table 5 lacks measurement details such as GPU model, kernel configuration, and averaging methodology; please report them.","section":"Section 5.6, Table 5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should look at this one. The core idea is real: using 2x2 codebook blocks of the form [[x,y],[-y,x]] to commute with RoPE rotations, then folding the key-codebook decode into QK^T, is a neat trick that makes additive quantization practical for KV caches. The EM training with group-shared indices is a solid contribution, and the 2-bit results on LongBench are genuinely close to FP16. I have no doubt the authors know what they are doing mathematically.\n\nThat said, the paper oversells its own data. The abstract says 1-bit quantization with \"minimal accuracy loss,\" but Table 2 shows CommVQ-1 at 0.00 on the R.KV retrieval task (FP16: 55.20) and an 11.9-point average drop on InfiniteBench. GSM8K drops from 76.27 to 66.57. That is not minimal by any reasonable reading. Even the 2-bit version loses most retrieval capacity on R.KV (12.20 vs. 55.20). The authors claim \"ours retains some capacity\" on retrieval tasks, which is simply false for R.KV. This needs a serious rewrite of the claims, not just a footnote.\n\nOther soft spots are less damaging but worth flagging. The intro's memory accounting is off by roughly a factor of four relative to the paper's own d=1024 for LLaMA-3.1-8B, and Figure 3 repeats the inflation. All evaluation numbers are single-run without error bars or variance. The EM appendix has undefined variables (the T matrix definition is garbled) and would be hard to reproduce from the text alone. The perplexity-based robustness check is not a good probe for exact-match retrieval failures, which are exactly where the method loses ground.\n\nNone of this kills the method. The constructive contribution is novel, the comparison against KIVI/KVQuant/VQLLM is informative, and the 1-bit results are still much better than baselines on most tasks. The paper deserves a serious referee, but it needs major revision: fix the memory accounting, add error bars, reword the claims about minimal loss, and clarify the EM derivation. I would bring it to a reading group with the caveat that the headline should be read as \"1-bit is better than previous quantization\" rather than \"1-bit is nearly lossless.\"\n\nRecommendation: accept for peer review, but expect the authors to fix the overclaims and the numerics before publication.","headline":"CommVQ has a genuinely clever commutative-codebook idea and the math holds up, but the paper overclaims: its own Table 2 contradicts the '1-bit with minimal accuracy loss' headline.","tokens_in":17068,"tokens_out":2244,"would_cite":true,"duration_ms":22122,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CommVQ compresses the KV cache of long-context LLMs to roughly 1–2 bits per scalar with near-lossless benchmark accuracy by vector-quantizing keys and values with a RoPE-commutative codebook.","keywords":["KV cache compression","vector quantization","additive quantization","RoPE","long-context LLM inference","codebook learning","1-bit quantization"],"falsifier":"Run CommVQ-2 on the InfiniteBench KVRetrieval task, whose random UUID key-value pairs are absent from the FineWeb-Edu calibration set, and track retrieval accuracy at 32K, 64K, and 128K contexts against FP16. The paper's own table already shows CommVQ-2 at 12.20 versus FP16 at 55.20 on this task; a systematic length sweep would show whether the gap widens with context length, and whether the 1-bit variant retains any retrieval signal at all.","tokens_in":15948,"feed_emoji":"⚡","tokens_out":9662,"duration_ms":82728,"temperature":0.7,"pith_summary":"This paper tries to establish that the KV cache—the per-token key and value tensors that dominate GPU memory during long-context LLM inference—can be quantized at the vector level to roughly one to two bits per scalar without the accuracy collapse that scalar 1-bit and 2-bit quantization suffers. The proposed method, CommVQ, encodes each key and value vector into binary codes with a lightweight encoder and decodes them through a learned codebook, and it constrains the codebook's 2x2 blocks to commute with RoPE rotation matrices so the costly query-key products can be precomputed and reused. The authors report that 2-bit CommVQ stays within fractions of a point of FP16 on LongBench and GSM8K while cutting KV cache memory by 87.5%, and that 1-bit CommVQ remains usable where scalar quantizers collapse. If these results hold, long-context inference at 128K tokens could run on a single consumer GPU, making vector quantization a practical KV-cache compression layer.","feed_headline":"KV cache drops to 1 bit per scalar with near-lossless scores","feed_subtitle":"A rotation-compatible codebook runs a 128K-context LLaMA-3.1 8B on one RTX 4090.","key_machinery":"The central object is the RoPE-commutative codebook. RoPE applies a block-diagonal rotation $R_m^i$ to each 2-dimensional sub-vector of a key; the codebook stores, per subspace, matrices $C=\\begin{pmatrix}x&y\\\\-y&x\\end{pmatrix}$ that satisfy $R_m^i C = C R_m^i$. Because of this commutativity, the attention score between the current query and the $i$-th quantized key can be reordered so that the query-dependent and codebook-dependent factors are computed once and reused across all cached positions. The codebook is fit by an EM-style clustering algorithm on a calibration set; consecutive sub-vectors share one quantized index (group size $g$), and $R$ residual rounds refine the reconstruction, giving an average bit rate of $R\\log_2(N_c')/g$.","core_discovery":"On the paper's own terms, the discovery is that KV-cache quantization does not have to be per-scalar to be fast and accurate; per-vector additive quantization with a codebook constrained to commute with RoPE can be integrated into self-attention at modest overhead. The paper reports that CommVQ-2 keeps LongBench average at 47.98 versus the FP16 baseline's 48.05 and GSM8K at 76.04 versus 76.27, while storing the cache at 2 bits per scalar (an 87.5% cut from FP16), and that CommVQ-1 retains LongBench 44.94, InfiniteBench 36.76, and GSM8K 66.57, where scalar-quantization baselines collapse. The method also reports real memory savings: a 120K-context LLaMA-3.1-8B-Instruct cache fits in about 20GB instead of 60GB, and decoding latency drops 6x to 9.6x versus the naive implementation of the same quantization.","pith_inferences":["The rotation-commutativity trick generalizes: any score computation that applies RoPE-style rotations to a sum of codebook vectors can precompute the rotated inner products once, so long-context cross-attention and retrieval heads built on the same structure should inherit the speedup.","Because the codebook is fixed-size while the cache grows with context, there is a context-length threshold below which codebook memory (2–5 MB for LLaMA-3.1-8B) outweighs the savings; computing that threshold per model would tell practitioners when CommVQ is worthwhile.","The 1-bit results on retrieval tasks suggest the residual codebook preserves coarse attention patterns but blurs fine-grained token identity; a testable next step is to spend more residual rounds on the key sub-vectors that retrieval-heavy attention heads actually use."],"forward_implications":["At 2 bits per scalar, CommVQ matches the FP16 baseline on LongBench (47.98 vs 48.05) and GSM8K (76.04 vs 76.27) while reducing the FP16 KV cache by 87.5%.","At 1 bit per scalar, CommVQ keeps average LongBench at 44.94 and InfiniteBench at 36.76, well above the best prior 1-bit baseline (27.42 and 18.48), and keeps GSM8K at 66.57.","A LLaMA-3.1-8B-Instruct model with a 120K-token context needs about 20GB of KV cache under CommVQ-1 instead of 60GB in FP16, and at 32K context the batch size scales from 8 to 128 without OOM.","Reordering attention so that softmax scores multiply the quantized value codes before the codebook multiply cuts value-decoding cost by a factor close to the hidden dimension, and the commutative key codebook makes overall decoding overhead roughly (R+1)/2 times vanilla attention instead of Nc times."],"supporting_citations":[{"why":"It supplies the additive-quantization formulation that CommVQ adapts: representing a vector as a sum of rows from a learned codebook.","marker":"Babenko & Lempitsky, 2014"},{"why":"It defines RoPE as block-diagonal rotation matrices and establishes the commutativity property with matrices of the form [[x,y],[-y,x]], the basis of the commutative codebook.","marker":"Su et al., 2024"},{"why":"It provides the LLaMA-3.1-8B-Instruct model on which the main long-context and GSM8K evaluations are run.","marker":"Dubey et al., 2024"},{"why":"KIVI is the primary scalar-quantization baseline, and its LongBench task setup is reused for the comparison.","marker":"Liu et al., 2024b"},{"why":"KVQuant is the non-uniform quantization baseline that CommVQ must beat at comparable bit widths.","marker":"Hooper et al., 2024"},{"why":"VQLLM is the residual vector-quantization baseline that CommVQ extends and outperforms, especially in the 1-bit regime.","marker":"Kumar, 2024"},{"why":"FineWeb-Edu is the calibration distribution on which the encoder and codebooks are trained.","marker":"Lozhkov et al., 2024"},{"why":"LongBench is one of the two long-context benchmarks used to measure near-lossless accuracy.","marker":"Bai et al., 2023"},{"why":"InfiniteBench supplies the ultra-long-context retrieval tasks, including KVRetrieval, that stress 1-bit compression.","marker":"Zhang et al., 2024b"},{"why":"GSM8K provides the math-reasoning evaluation showing CommVQ-2 stays within 0.23 points of FP16.","marker":"Cobbe et al., 2021"}],"fun_headline_variants":["KV cache down 8x with commutative additive quantization","1-bit KV cache fits 128K context on one RTX 4090","Commutative codebook enables near-lossless 2-bit KV cache","KV cache quantization hits 8x compression with minimal loss","RoPE-compatible codebook cuts KV cache to 2 bits, 1-bit works"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a codebook and encoder fit on a sample of FineWeb-Edu text remain accurate for the keys and values that matter on unseen long-context, retrieval, and math data; the paper's evidence is perplexity on a few shifted domains and single-run task scores, not a distributional guarantee.","fun_headline_variants_meta":{"raw":{"variants":["KV cache down 8x with commutative additive quantization","1-bit KV cache fits 128K context on one RTX 4090","Commutative codebook enables near-lossless 2-bit KV cache","KV cache quantization hits 8x compression with minimal loss","RoPE-compatible codebook cuts KV cache to 2 bits, 1-bit works"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001333,"raw_usage":{"total_tokens":5464,"prompt_tokens":1031,"completion_tokens":4433,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":4338}},"tokens_in":647,"tokens_out":4433,"duration_ms":28328,"temperature":1.0,"reasoning_tokens":4338,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:41:55.114305+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CommVQ-2 on the InfiniteBench KVRetrieval task, whose random UUID key-value pairs are absent from the FineWeb-Edu calibration set, and track retrieval accuracy at 32K, 64K, and 128K contexts against FP16. The paper's own table already shows CommVQ-2 at 12.20 versus FP16 at 55.20 on this task; a systematic length sweep would show whether the gap widens with context length, and whether the 1-bit variant retains any retrieval signal at all.","supporting_citations":[],"review_version":1}