Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

ParisKV claims that KV-cache retrieval can match or exceed full attention quality across million-token decoding by replacing learned centroids with data-independent hypersphere centroids, and it backs that claim with a GPU-native collision-

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:29 UTC pith:5VU7SF25

load-bearing objection A well-engineered KV-retrieval system with a genuinely new combination of ideas; the empirical claims are credible, but the theory is oversold and needs one key validation. the 3 major comments →

arxiv 2602.07721 v3 pith:5VU7SF25 submitted 2026-02-07 cs.LG cs.CLcs.DB

ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs

Yanlin Qi , Xinhang Chen , Huiqiang Jiang , Qitong Wang , Botao Peng , Themis Palpanas This is my paper
classification cs.LG cs.CLcs.DB
keywords KV-cache retrievallong-context LLM inferencesparse attentiondrift robustnesscollision-based candidate selectionquantized inner-product rerankingCPU offloadingunified virtual addressing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ParisKV argues that the real failure mode of sparse KV-cache retrieval during long generation is not sparsity itself but stale learned structure: centroids fitted to prefill keys drift as new tokens arrive. Instead of adapting centroids, it replaces them with a fixed set of sign-pattern directions on a rotated unit hypersphere, chosen so that every key, old or newly generated, stays close to some centroid. On top of that it builds a fully GPU-side two-stage retriever—fast subspace collision voting for candidate pruning, then calibrated 4-bit inner-product reranking—and fetches only the final top-k KV pairs from CPU memory through unified virtual addressing. If the paper is right, long-context decoding can keep full-attention quality while running much faster and scaling past the point where full attention runs out of memory.

Core claim

The paper's central discovery is that retrieval drift during decoding can be removed by construction instead of repaired by adaptation. After l2-normalizing keys and queries and applying a shared random orthogonal rotation, subspace directions become approximately isotropic; ParisKV therefore defines a data-independent codebook of 2^m sign-pattern centroids per subspace, uniformly covering the unit sphere. Any newly generated key, whatever its origin, lies close to at least one centroid, so the coarse voting stage never faces stale centroids. The reranking stage estimates raw query–key inner products from 4-bit direction codes with a precomputed per-key, per-subspace scaling factor that corr

What carries the argument

The load-bearing object is the analytic centroid codebook: after l2-normalization and a shared random orthogonal rotation (implemented with SRHT), each m-dimensional subspace is assigned to one of 2^m sign-pattern centroids—vectors with all coordinates equal to ±1/√m—so centroids are uniformly spread on the unit sphere and depend on no training or prefill data. This codebook is what makes the coarse collision-voting stage drift-robust, because newly generated keys always have a nearby bucket. Two secondary mechanisms carry efficiency: subspace-split collision counting for candidate pruning, and a quantized inner-product estimator with a precomputed scaling factor w_{i,b}, which lets rerankin

Load-bearing premise

Everything rests on the claim that after l2-normalization and the fixed random rotation, key directions are spread evenly enough across each subspace that fixed sign-pattern centroids are as good as learned ones—if real decoding keys cluster in a few orthants under that rotation, the collision-voting stage loses recall and the drift-robustness collapses.

What would settle it

A concrete disproof would be to run ParisKV's coarse stage on real decoding keys with several different SRHT rotations and compare collision Recall@100; if recall swings widely across rotations, or if the empirical distribution of squared subspace coordinates (u_b)_j^2 deviates sharply from the Beta(1/2,(m-1)/2) prediction, the isotropy premise is false. An even simpler check: train k-means centroids on the first 1% of decoding keys and measure whether they substantially outperform the analytic centroids after a few thousand steps—if so, the claim of drift-robustness by construction fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, long-context decoding can attend to a fixed top-100 KV set and still match full attention, so memory bandwidth no longer needs to scale with full context length.
  • GPU-native retrieval makes CPU offload of the full KV cache viable without CPU-side search bottlenecks, extending single-GPU serving to contexts where full attention runs out of memory.
  • Because the centroids are data-independent, the drift-robustness should transfer to new models and tasks without retraining or re-clustering.
  • The reported result that sparse retrieval can match full-attention speed at batch size 1 implies that quality does not have to be traded away to gain latency.
  • At million-token scale, the claimed 17x–44x decode-latency reductions over prior retrieval methods follow directly if the pipeline measurements hold.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the isotropy assumption is the fragile hinge: SRHT is not Haar-random, and if real decoding keys concentrate in a limited set of orthants under the fixed rotation, the collision-voting stage would lose recall. A cheap test is to rerun the pipeline with several SRHT seeds and measure recall variance.
  • Because the centroids are data-independent, ParisKV is robust to drift but also blind to any exploitable structure in the key distribution; on tasks where attended keys are highly clustered in a narrow region, learned centroids could require a smaller candidate budget than fixed sign patterns.
  • The Beta priors from Proposition 4.1 could be used beyond quantization, for example to allocate the candidate budget non-uniformly across subspaces based on expected subspace energy fractions, instead of the uniform schedule the paper uses.
  • The same normalize-rotate-plus-analytic-centroid recipe is not limited to LLM KV caches; it could make other inner-product retrieval indexes drift-robust, though the paper does not explore that extension.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. ParisKV proposes a GPU-native, two-stage KV-cache retrieval method for long-context LLM decoding. In prefill, keys are L2-normalized, rotated via SRHT, split into subspaces, assigned to data-independent sign-pattern centroids, and summarized by 4-bit direction codes plus per-key scaling factors. At decode time, a coarse collision-voting stage prunes candidates using centroid IDs, a fused reranking kernel estimates inner products from quantized codes, and only the final top-k full-precision KV pairs are fetched from CPU via UVA. The paper claims that this design is drift-robust, matches or exceeds full-attention quality on long-input and long-generation benchmarks, and reduces decode latency by 17x-44x versus MagicPIG and PQCache at million-token scale.

Significance. If the claims hold, ParisKV is a significant systems contribution: it replaces learned, drift-prone centroids with analytic centroids, keeps the entire retrieval path on GPU, and uses UVA to avoid CPU-side search and explicit copies. The evaluation is broad (three model families, long-generation and long-input benchmarks, ablation studies) and the code is released, which strengthens reproducibility. The main strengths are the data-independent centroid construction, the alpha-correction ablation, and the UVA-based offloading design. However, the paper's theoretical grounding has a load-bearing gap: the key isotropy assumption is verified neither theoretically for SRHT nor empirically, and the reranking estimator in Eq. (8) is asserted without error analysis. These issues do not necessarily invalidate the empirical results, but they need to be resolved before the drift-robustness and 'accurate estimator' claims can be accepted.

major comments (3)
  1. [§4.1.2 and Proposition 4.1] The claim that 'after L2 normalization and SRHT rotation, subspace directions become approximately isotropic' is the foundation for the data-independent centroids and for the Beta priors in Proposition 4.1. However, Proposition 4.1 is proved for Haar-random orthogonal R, while the implementation uses SRHT. SRHT is a structured transform and does not, by itself, make an arbitrary key distribution isotropic; no theorem or measurement is provided showing that the actual subspace-energy fractions and coordinate-wise squared directions match the Beta(m/2,(D-m)/2) and Beta(1/2,(m-1)/2) laws. If the isotropy assumption fails, the sign-pattern centroids are miscalibrated and the 4-bit quantization levels are suboptimal, directly affecting both the coarse collision-voting recall and the reranking accuracy. Figure 10 shows coarse Recall@100 of only 16.1% and end-to-end Recall@100 of 64.3%, so the
  2. [§4.1.3, Eq. (8)] The estimator <u_{i,b}, q_tilde_b> ≈ <v_{i,b}, q_tilde_b> / alpha_{i,b} is exact only when v_{i,b} is collinear with u_{i,b}. For general 4-bit coordinate-wise quantization, v is not a scalar multiple of u, and the residual u - v/alpha has squared norm 1/alpha^2 - 1, which can be large (e.g., ~0.56 for alpha=0.8). The error term <u - v/alpha, q> is query-dependent and not controlled by alpha alone. The ablation shows an empirical benefit of the alpha correction, but the text calls the estimator 'accurate' without a bound or error analysis. Please provide a formal error characterization or replace the claim with an empirical distribution of the residual error over real queries and keys.
  3. [Abstract, §5.1, Tables 2-3] The abstract states that ParisKV 'matches or outperforms full attention quality on long-input and long-generation benchmarks,' but the reported numbers do not fully support this. In Table 3, ParisKV is below full attention on LongBench-V2 overall for Qwen3-4B (24.60 vs 25.84) and Qwen3-8B (33.07 vs 33.59), and in Table 2 it is below full attention on AIME25 for Qwen3-4B (80.00 vs 86.67) and Qwen3-8B (73.33 vs 83.33). The accurate summary is that ParisKV exceeds full attention on most long-generation settings and is comparable within about one point on long-input tasks, with some degradations. Please qualify the abstract and Section 6 accordingly.
minor comments (4)
  1. [Abstract, §5.2, §6] Speedup numbers are inconsistent: the abstract and contributions mention 17x and 44x/45x speedups, while §5.2 reports 44.4x and 16.9x at 1024K. Please unify these figures.
  2. [§4.1.3, Eq. (9)] The definition of w_{i,b} is ambiguous in the typeset equation: it is not clear whether alpha_{i,b} multiplies or divides the numerator. Please clarify the formula and the derivation leading from Eq. (8) to Eq. (10).
  3. [§5.4] The design-space exploration reports only the default configuration (B=16, m=8, K_omega=256, K_r=1) and one recall number. A short sensitivity table over B, m, K_omega, and K_r would help readers understand the robustness of the default choice.
  4. [§5.1, Fig. 10] The term 'near-lossless' is used in the contributions, but the reported end-to-end Recall@100 is 64.3% with substantial variance (std=4.7%). Please clarify that 'near-lossless' refers to end-task accuracy, not retrieval recall, or provide a mapping between retrieval recall and task accuracy.

Circularity Check

0 steps flagged

No circular reduction in the claimed derivation; score reflects only minor non-load-bearing self-citations. The SRHT-vs-Haar gap is a correctness risk, not circularity.

full rationale

I walked the derivation chain. The analytic centroids (§4.1.2, Eq. 5–6) are defined directly from sign patterns and assignment (Eq. 6); they are not fitted to the benchmarks. Proposition 4.1 derives Beta marginals for a Haar-random rotation (Eq. 11–12) and is used to set the 4-bit quantization levels, but the implementation uses SRHT ('shared orthogonal rotation R (implemented by SRHT)'). This is an untested assumption gap that could affect recall if decoded-key directions are not approximately isotropic—the paper's own Fig. 10 shows end-to-end Recall@100 of only 64.3±4.7%—but it is a validity/correctness concern, not an equivalence-by-construction. The α-correction (Eq. 7–10) is computed from each key's true direction and its quantized code (wi,b 'depends only on the key and the quantization metadata'), so it is calibration of the inner-product estimator, not fitting to downstream accuracy. Hyperparameters (ρ, β) and the (B,m) configuration are tuned, but no equation makes a prediction equal to a fitted value. Self-citations to Qwen3 (Qwen Team, 2025), MInference (Jiang et al., 2024), RetrievalAttention (Liu et al., 2024a), and RetroInfer (Chen et al., 2025) involve author Huiqiang Jiang but are related-work/model references and are not load-bearing for the central drift-robustness or efficiency claims. Hence no prediction reduces to its inputs by construction; the score of 2 reflects only those minor non-load-bearing self-citations.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim rests on the adequacy of top-k inner-product attention approximation, the empirical isotropy of SRHT-transformed key directions, and the validity of the alpha-scaled quantized reranker. No new physical entities are invented; free parameters are system and tuning choices selected in §5.4 and Table 1.

free parameters (4)
  • adaptive coarse-stage schedule (rho, beta) = beta=5%-10%, rho>=beta, adaptive w.r.t. KV length
    Candidate ratio and per-subspace top fraction are hand-tuned and varied with KV length; they directly trade recall vs speed (§4.2.2, §5.4).
  • subspace configuration (B, m) = B=16, m=8 (B*m=128); K_omega=256, K_r=1
    Chosen as 'compact default' from design space exploration to achieve 72.74% recall (§5.4); not derived from theory.
  • retrieval budget K = 100
    Fixed top-k budget used in all experiments; no ablation reported (§5).
  • per-task window sizes (Local, Update, Full-thres) = e.g., 256/512/2K for AIME, 128/512/2K for GPQA
    Table 1 lists per-dataset configuration; these affect which tokens are dense vs retrieved.
axioms (5)
  • domain assumption Approximating softmax attention by top-k inner-product keys is sufficient for quality
    Eq. (1)-(3) assume restricting softmax to C(q) containing TopK(q) closely matches exact output; no error bound is given.
  • ad hoc to paper After L2 normalization and shared SRHT rotation, subspace directions are approximately isotropic
    §4.1.2: this justifies data-independent sign-pattern centroids; SRHT is not Haar-random, only empirically approximate.
  • domain assumption Proposition 4.1's Haar-random distribution applies to the finite fixed SRHT transform used
    §4.2.2: Beta priors for z_b and (u_b)_j^2 are derived for Haar-random R, but the implementation uses SRHT.
  • domain assumption Recall@100 on benchmark queries is a valid proxy for end-to-end accuracy
    Ablations evaluate Recall@100 and claim quality translation; no direct proof.
  • domain assumption Pass@8 accuracy at fixed sampling is representative
    Evaluation uses pass@8 on AIME25 but no error bars or variance are reported.

pith-pipeline@v1.3.0-alltime-deepseek · 14849 in / 15020 out tokens · 155633 ms · 2026-08-03T03:29:52.505250+00:00 · methodology

0 comments
read the original abstract

KV-cache retrieval is essential for long-context LLM inference, yet existing methods struggle with distribution drift and high latency at scale. We introduce ParisKV, a drift-robust, GPU-native KV-cache retrieval framework based on collision-based candidate selection, followed by a quantized inner-product reranking estimator. For million-token contexts, ParisKV supports CPU-offloaded KV caches via Unified Virtual Addressing (UVA), enabling on-demand top-$k$ fetching with minimal overhead. ParisKV matches or outperforms full attention quality on long-input and long-generation benchmarks. It achieves state-of-the-art long-context decoding efficiency: it matches or exceeds full attention speed even at batch size 1 for long contexts, delivers up to 2.8$\times$ higher throughput within full attention's runnable range, and scales to million-token contexts where full attention runs out of memory. At million-token scale, ParisKV reduces decode latency by 17$\times$ and 44$\times$ compared to MagicPIG and PQCache, respectively, two state-of-the-art KV-cache Top-$k$ retrieval baselines, code is available at https://github.com/amy-77/ParisKV/tree/main.

Figures

Figures reproduced from arXiv: 2602.07721 by Botao Peng, Huiqiang Jiang, Qitong Wang, Themis Palpanas, Xinhang Chen, Yanlin Qi.

Figure 1
Figure 1. Figure 1: Retrieval drift results. (a) Recall comparison of different methods on AIME. (b) Centroid drift induced by decoding keys, measured as the mismatch between prefill-only centroids (original centroids in blue) and reference centroids (correct centroids in red) obtained by clustering all keys from both prefill and decoding. has motivated sparse/selective attention, which exploits the empirical sparsity of atte… view at source ↗
Figure 2
Figure 2. Figure 2: ParisKV pipeline. Offline, we construct an analytic centroid codebook and a quantization configuration. During prefill, we materialize the KV cache and build GPU-resident key summaries (centroid IDs for Stage-I vote-based filtering, and low-bit codes with lightweight weights for Stage-II reranking), while asynchronously offloading full-precision KV to CPU memory. During decoding, summaries for newly genera… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of rotation-based codebook assignment on the unit sphere. Candidate key indices Step1: Select top-𝝆 keys for each subspace Step2: Add collision count Step3: Total collision count Step4: Sorted keys Cnt K1 5 K2 5 K3 1 K4 3 K5 3 K6 3 K7 4 K8 4 Cnt K1 5 K2 5 K7 4 K8 4 K4 3 K5 3 K6 3 K3 1 Sub1 Sub2 Sub3 Sub4 C4 C3 C5 C3 K1 K2 C1 C2 K3 C4 C2 K4 K5 C4 C2 C3 K6 C1 K7 K8 Sub1 Sub2 Sub3 Sub4 +2 +0 K1 +… view at source ↗
Figure 5
Figure 5. Figure 5: Sliding-window KV-cache update. buffer); and (iii) we encode and index the evicted keys on GPU (centroid ids, 4-bit codes, and wi,b), while offloading the corresponding full-precision KV pairs asynchronously. This streaming update supports unbounded generation while keeping retrieval metadata fresh and preserving fast access to both recent context (Local) and long-range history (Re￾trieval). 4.2.2. RETRIEV… view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of runtime between Torch and custom kernels. Methods Overall Short Medium Long Easy Hard Easy Hard Easy Hard Qwen3-4B 25.84 27.12 16.53 36.36 25.20 26.67 28.57 PQCache 17.91 16.95 19.00 13.60 19.00 20.00 19.05 MagicPIG 16.70 18.64 10.74 14.77 20.47 28.89 12.70 ParisKV (Ours) 24.60 35.59 19.49 26.14 22.05 28.89 23.81 Qwen-3-8B 33.59 50.85 34.71 32.95 25.98 37.78 28.57 PQCache 25.50 23.70 31.60 28… view at source ↗
Figure 7
Figure 7. Figure 7: Longbench V2 Decoding throughput vs. Context Length and Batch Size 128K 256K 384K 512K 1024K Decode Latency (ms/step) 128K 256K 384K 512K 1024K Prefill Latency (s) MagicPig PQCache ParisKV Full Attention 128K 256K 384K 512K 1024K Context Length (K) 10 20 50 100 200 500 1000 2000 Decode Latency (ms/step) (a) Decode Latency 128K 256K 384K 512K 1024K Context Length (K) 0 200 400 600 800 Prefill Latency (s) (b… view at source ↗
Figure 10
Figure 10. Figure 10: Retrieval quality comparisons. (a) Coarse-stage collision quality. (b) End-to-end recall. Ours (N+R+T) denotes normalization + rotation + theoretical centroids. than full attention within its runnable range, and reduces decode latency by 17× and 44× compared to MagicPIG and PQCache at million-token scale. Our results show that carefully designed KV-cache retrieval can simultaneously improve both efficienc… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DataMagic: Transforming Tabular Data into Data Insight Video

    cs.HC 2026-06 unverdicted novelty 5.0

    DataMagic generates narrative data videos from tabular data and queries via DVSpec declarative bindings and a Generate-then-Orchestrate multi-agent pipeline.

Reference graph

Works this paper leans on

23 extracted references · 18 linked inside Pith · cited by 1 Pith paper

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [4]

    Retroinfer: A vector-storage approach for scalable long-context llm inference.arXiv preprint arXiv:2505.02922,

    Chen, Y ., Zhang, J., Lu, B., Zhang, Q., Zhang, C., Luo, J., Liu, D., Jiang, H., Chen, Q., Liu, J., et al. Retroinfer: A vector-storage approach for scalable long-context llm inference.arXiv preprint arXiv:2505.02922,

    Pith/arXiv arXiv

  3. [5]

    Mag- icpig: Lsh sampling for efficient llm generation.arXiv preprint arXiv:2410.16179,

    Chen, Z., Sadhukhan, R., Ye, Z., Zhou, Y ., Zhang, J., Nolte, N., Tian, Y ., Douze, M., Bottou, L., Jia, Z., et al. Mag- icpig: Lsh sampling for efficient llm generation.arXiv preprint arXiv:2410.16179,

    Pith/arXiv arXiv

  4. [6]

    Flashattention-2: Faster attention with bet- ter parallelism and work partitioning.arXiv preprint arXiv:2307.08691,

    Dao, T. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning.arXiv preprint arXiv:2307.08691,

    Pith/arXiv arXiv

  5. [8]

    Feng, Y ., Lv, J., Cao, Y ., Xie, X., and Zhou, S. K. Ada- kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference.arXiv preprint arXiv:2407.11550,

    Pith/arXiv arXiv

  6. [9]

    Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning.arXiv preprint arXiv:2410.19258,

    Fu, Y ., Cai, Z., Asi, A., Xiong, W., Dong, Y ., and Xiao, W. Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning.arXiv preprint arXiv:2410.19258,

    arXiv

  7. [10]

    K.-H., Cao, T., Yang, F., et al

    Gao, Y ., Zeng, Z., Du, D., Cao, S., Zhou, P., Qi, J., Lai, J., So, H. K.-H., Cao, T., Yang, F., et al. Seerattention: Learning intrinsic sparse attention in your llms.arXiv preprint arXiv:2410.13276,

    Pith/arXiv arXiv

  8. [11]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  9. [13]

    W., Yun, S., and Song, H

    9 ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs Kim, J.-H., Kim, J., Kwon, S., Lee, J. W., Yun, S., and Song, H. O. Kvzip: Query-agnostic kv cache compression with context reconstruction.arXiv preprint arXiv:2505.23416,

    arXiv

  10. [14]

    Flex- prefill: A context-aware sparse attention mechanism for efficient long-sequence inference.arXiv preprint arXiv:2502.20766,

    Lai, X., Lu, J., Luo, Y ., Ma, Y ., and Zhou, X. Flex- prefill: A context-aware sparse attention mechanism for efficient long-sequence inference.arXiv preprint arXiv:2502.20766,

    Pith/arXiv arXiv

  11. [15]

    J., and Yuan, M

    Li, X., Xing, Z., Li, Y ., Qu, L., Zhen, H.-L., Liu, W., Yao, Y ., Pan, S. J., and Yuan, M. Kvtuner: Sensitivity-aware layer-wise mixed-precision kv cache quantization for ef- ficient and nearly lossless llm inference.arXiv preprint arXiv:2502.04420,

    arXiv

  12. [16]

    Re- trievalattention: Accelerating long-context llm inference via vector retrieval.arXiv preprint arXiv:2409.10516, 2024a

    Liu, D., Chen, M., Lu, B., Jiang, H., Han, Z., Zhang, Q., Chen, Q., Zhang, C., Ding, B., Zhang, K., et al. Re- trievalattention: Accelerating long-context llm inference via vector retrieval.arXiv preprint arXiv:2409.10516, 2024a. Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V ., Chen, B., and Hu, X. Kivi: A tuning-free asym- metric 2bit quant...

    Pith/arXiv arXiv

  13. [17]

    J., Goel, R., Lee, M., and Lott, C

    Park, J., Jones, D., Morse, M. J., Goel, R., Lee, M., and Lott, C. Keydiff: Key similarity-based kv cache eviction for long-context llm inference in resource-constrained environments.arXiv preprint arXiv:2504.15364,

    arXiv

  14. [18]

    Shadowkv: Kv cache in shadows for high-throughput long-context llm inference.arXiv preprint arXiv:2410.21465,

    Sun, H., Chang, L.-W., Bao, W., Zheng, S., Zheng, N., Liu, X., Dong, H., Chi, Y ., and Chen, B. Shadowkv: Kv cache in shadows for high-throughput long-context llm inference.arXiv preprint arXiv:2410.21465,

    Pith/arXiv arXiv

  15. [19]

    Quest: Query-aware sparsity for efficient long-context llm inference.arXiv preprint arXiv:2406.10774,

    Tang, J., Zhao, Y ., Zhu, K., Xiao, G., Kasikci, B., and Han, S. Quest: Query-aware sparsity for efficient long-context llm inference.arXiv preprint arXiv:2406.10774,

    Pith/arXiv arXiv

  16. [20]

    Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,

    Pith/arXiv arXiv

  17. [22]

    Xattention: Block sparse attention with antidiagonal scoring.arXiv preprint arXiv:2503.16428,

    Xu, R., Xiao, G., Huang, H., Guo, J., and Han, S. Xattention: Block sparse attention with antidiagonal scoring.arXiv preprint arXiv:2503.16428,

    Pith/arXiv arXiv

  18. [23]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  19. [2021]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b.arXiv preprint arXiv:2310.06825,

    Pith/arXiv arXiv

  20. [2022]

    Expected attention: Kv cache compression by estimating attention from future queries distribution.arXiv preprint arXiv:2510.00636,

    Devoto, A., Jeblick, M., and J´egou, S. Expected attention: Kv cache compression by estimating attention from future queries distribution.arXiv preprint arXiv:2510.00636,

    arXiv

  21. [2023]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. InInterna- tional conference on machine learning, pp. 38087–38099. PMLR, 2023a. Xiao, G., Tian, Y ., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv pre...

    Pith/arXiv arXiv

  22. [2024]

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks.arXiv preprint arXiv:2412.15204,

    Bai, Y ., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y ., Tang, J., and Li, J. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks.arXiv preprint arXiv:2412.15204,

    Pith/arXiv arXiv

  23. [2025]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069,

    Cai, Z., Zhang, Y ., Gao, B., Liu, Y ., Li, Y ., Liu, T., Lu, K., Xiong, W., Dong, Y ., Hu, J., et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069,

    Pith/arXiv arXiv