Pith. sign in

REVIEW 3 major objections 5 minor 51 references

Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PoD compresses the KV cache of long-context LLMs by up to 35% without a measured loss in accuracy, by keeping the full cache for the first and most recent tokens and letting layers with similar attention share one set of keys for distant…

desk verdict A competent new KV-sharing mechanism with broad experiments; the derivation gap and a slightly overstated headline are addressable. read the letter →

arxiv 2412.02252 v2 pith:PUN4BJJK submitted 2024-12-03 cs.CL

Da Ma , Lu Chen , Situo Zhang , Yuxun Miao , Su Zhu , Zhi Chen , Hongshen Xu , Hanqi Li
show 3 more authors
Shuai Fan Lei Pan Kai Yu
This is my paper · ORCID
classification cs.CL
keywords KVcachecompressionlong-contextLLMinferenceinter-layerattentionsimilaritysharingproximaltokenspost-trainingadaptationlayergroupingtokenimportance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the main memory bottleneck of long-context LLM inference, the key-value cache that grows with every layer and every token, can be cut by about a third without discarding any token and without losing accuracy. The proposed method, PoD, rests on two measured regularities: for next-token prediction the first and most recent tokens dominate, and attention scores on distant tokens are nearly identical across consecutive layers. So PoD keeps full keys and values for proximal tokens while storing distant tokens' keys once per block of similar layers, letting every layer in the block reuse them after a short post-training step adapts the model to this shared structure. If correct, long-context models would need about 35% less cache memory, which the paper shows translates into roughly 30% larger batches on a single GPU, and the method would compose with token-selection compression such as SnapKV for even larger savings.

What carries the argument

The carrying mechanism is key-sharing in layer blocks, defined by measured inter-layer attention similarity. For each head, layers that are mutually similar under Jensen-Shannon divergence (threshold $\ge 0.5$) are merged greedily into blocks; distant-token logits $a^D_{\ell,i}$ at each higher layer are then computed with the block's lowest-layer query and key, $Q_{\ell_a}$ and $K_{\ell_a}$, so only that layer's distant keys need to be stored and cached. A parameter-free gate $g_{\ell,i} = \sum \exp a^P_{\ell,i} / (\sum \exp a^P_{\ell,i} + \sum \exp a^D_{\ell,i})$ combines proximal and distant attention outputs, which is exactly the decomposition of the dense attention output into two softmax terms over the two token groups. The whole scheme is trained in by the lightweight post-training adaptation, and the gate also enables approximate compute savings by skipping distant-token computation whenever $g \ge \tau$ for a tunable threshold.

What would settle it

Run the same offline similarity measurement the paper uses to form blocks, but on the post-adaptation model: for each head and block, compare each higher layer's true attention logits over distant tokens against the shared lowest-layer logits, and check whether the Jensen-Shannon divergence stays below the 0.5 threshold on inputs at the full context length. If for some head the divergence rises well above threshold, or if a needle planted in the distant region becomes retrievable only when the higher layer uses its own keys, then sharing has collapsed and the no-performance-loss claim would fail for that configuration.

Watch

Extended reading notes

Core claim

PoD's central claim is that distant tokens can be compressed, not forgotten: the entire long context stays in memory at lower cost by exploiting redundancy in how layers attend to it. For each attention head, the paper measures the average Jensen-Shannon divergence between attention-score distributions of consecutive layers, groups consecutive layers whose distant-token attention agrees above a threshold of 0.5 into blocks, and then within each block keeps only the lowest layer's keys for distant tokens. Every higher layer in the block computes its distant-token logits with the block's lowest-layer query and key rather than its own, and a parameter-free gate blends the proximal and distant attention outputs. A light post-training run on 5 billion tokens adapts the model to this structure, and the resulting models match or slightly exceed the dense baseline on LongBench, LEval, InfiniteBench, and Needle in a Haystack while using up to 35% less KV cache memory.

Load-bearing premise

The argument assumes that attention scores for distant tokens computed at the lowest layer of a block, using that layer's query and key states, stay accurate enough for every higher layer in the block, even though the similarity that groups those layers is measured on the original model before the adaptation training.

Editorial extensions

If this is right

  • KV cache memory drops by up to 35% with no measured accuracy loss, because distant-token keys are stored once per layer block instead of once per layer.
  • The saved memory shows up operationally: maximum batch size on one GPU grows by more than 30% at prompt lengths from 2K to 16K tokens.
  • PoD composes with token-selection methods; PoD+SnapKV reaches about 92% KV cache savings with only a small performance drop, so the two compression philosophies are additive.
  • Retaining every token keeps mid-context needles retrievable, unlike token-eviction methods; PoD scores 98.9 on Needle in a Haystack versus 56.8 for StreamingLLM and 55.6 for H2O.
  • A gate threshold $\tau$ lets decoding skip distant-token computation for tokens whose attention is already dominated by proximal context, trading about 25% of computation for roughly 5% performance at $\tau = 0.7$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper shares keys but not values for distant tokens; an immediate extension would apply the same block structure to value states or low-rank value projections, roughly doubling the savings, though values carry more content and the adaptation cost would likely rise.
  • Because the offline layer grouping is computed once and then fixed, a testable variant is to add a small training regularizer that actively encourages inter-layer attention similarity, which should make the block structure stable for input distributions far from the adaptation corpus.
  • The paper's finding that token-selection methods degrade at 128K context while PoD holds suggests a general principle, that keeping a degraded copy of every token beats keeping an exact copy of some tokens, which would predict PoD stays ahead of eviction methods on long inputs with many repeated distractors.
  • The gate $g_{\ell,i}$ is a per-token, per-layer measure of dependence on distant context, computed for free by the method; it could serve as a runtime signal for detecting when the model is about to need retrieval of a distant fact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PoD, a KV cache compression method for long-context LLM inference. PoD keeps the full KV cache for proximal tokens (a fixed number of initial and most recent tokens) and, for all other "distant" tokens, shares a single set of attention scores — equivalently, a single key state and query state from the lowest layer of each block — across consecutive layers whose attention patterns are similar. Blocks are discovered offline by a per-head Jensen-Shannon divergence criterion. The model is then post-trained on 5B tokens to adapt to the new attention structure. The evaluation covers Needle in a Haystack, LongBench, LEval, InfiniteBench, standard benchmarks, and memory/throughput measurements, reporting a KV cache reduction of about 35% with an average performance degradation of 2.8% relative to the dense model and better performance than token-eviction and naive layer-sharing baselines.

Significance. If the central claim holds, PoD is a practically useful and conceptually interesting alternative to token-eviction compression: it keeps all tokens in a compact shared form rather than discarding them, and it is orthogonal to token-selection methods such as SnapKV. The empirical work is unusually broad for a KV-cache paper, covering multiple long-context benchmarks, a second model family (LLaMA3.1-8B), and explicit memory-batch-size measurements that confirm the theoretical savings. The authors also disclose the main engineering limitations (latency overhead and post-training cost) in their Impacts and Limitations section. However, the paper's theoretical justification has a load-bearing gap: the gating derivation in Appendix C.1 assumes that both proximal and distant logits come from the current layer's query and key, while Eq. (3) actually computes distant logits from the block's lowest layer. The empirical claim "without compromising performance" is also stronger than the reported 2.8% average degradation. These issues are fixable with additional analysis, so I do not see a basis for rejection, but they require a revision before the claims can be accepted as stated.

major comments (3)
  1. [§2.2, Eq. (3); Appendix C.1] The derivation of the gating mechanism is inconsistent with the attention-sharing implementation. In Eq. (3), the distant-token logits aD_ℓ,i are computed with Q_ℓa and K_ℓa, the query and key of the lowest layer in the block, not with the current layer's Q_ℓ and K_ℓ. In contrast, Appendix C.1 (Eqs. (5)-(6)) derives the gate g_ℓ,i as the exact softmax weight under the assumption that both aP and aD are produced by the same layer's Q_ℓ and K_ℓ. Consequently, Eq. (4) is not an exact decomposition of the current layer's attention output even when inter-layer attention distributions are perfectly similar, since the normalizing constant uses sums of exponentials of a different query-key pair. This is the load-bearing approximation behind the "no performance loss" claim, and the paper currently treats it as an exact derivation. I ask the authors to either (a) state explicitly that this is an approximation that post-training adaptation compensates for, or (b) add an ablation that isolates the effect of the shared query: for example, compare the current Eq. (3) against a variant that uses each layer's own Q_ℓ with the shared K_ℓa for distant tokens. Reporting the post-adaptation substitution error, e.g., the distance between softmax(Q_ℓ K_ℓ^T/sqrt(d)) and softmax(Q_ℓa K_ℓa^T/sqrt(d)) for distant tokens at non-base layers, would directly quantify the risk identified here.
  2. [§3.2, Table 5; Abstract; §5] The abstract and conclusion state that PoD reduces KV cache memory "without compromising performance," but Table 5 reports a 2.8% average performance degradation for PoD relative to the dense model, and Table 3 shows practical memory savings of 30.8-33.3% rather than the nominal 35%. The 2.8% number may well be acceptable in practice, especially relative to the 7.4-8.0% degradations of StreamingLLM and H2O and the 31.4% degradation of CLA, but the wording of the claim should be calibrated to the evidence. I request that the authors either soften the claims to "with negligible / minimal degradation" or add statistical significance information (multiple seeds, confidence intervals, or per-task error bars) showing that the 2.8% average is not driven by a few outlier tasks. The distinction matters because the central contribution is exactly that compression does not hurt accuracy.
  3. [§2.1, Algorithm 1; §3.3, Figure 3] The layer grouping is computed offline on the pre-adaptation model, but the attention-sharing structure is then post-trained for 5B tokens. The paper does not verify that the inter-layer attention similarities used to form blocks survive the post-training adaptation. If post-training changes attention patterns, the chosen blocks may no longer satisfy the similarity criterion, and the shared distant logits would then be an even rougher approximation for higher layers. I recommend reporting the attention-similarity matrix of the adapted PoD model on held-out sequences, or at minimum ablating grouping choices (e.g., random grouping of equal block sizes vs. the similarity-based grouping) to show that the offline similarity signal is what makes the method work.
minor comments (5)
  1. [Table 1; Table 2] There are typos in the headers: "LM-Infite" should be "LM-Infinite" and "T oken-selection" should be "Token-selection." Please correct these and check all table headers for similar spacing issues.
  2. [§2.2, Eq. (3)] The symbol ℓa is used in Eq. (3) for the block's lowest layer but is not defined in the main text; please define it explicitly (e.g., "where ℓa = min B_ℓ is the lowest layer in the block") so that the equation is self-contained.
  3. [§3.1, Table 1] The PoD row reports its configuration as "16+4080+28K," which is not explained in the table caption or surrounding text. Please clarify that 16 and 4080 are the initial and recent proximal token counts, and that 28K denotes the remaining distant tokens whose logits are shared, so the total context is 32K.
  4. [§3.3, Figure 3] The ablation study selects the 4K proximal-token budget and the 35% saving rate based on the same LEval validation set that is later used for the headline results. Please state this explicitly and report whether the LongBench, InfiniteBench, and NIAH numbers in Tables 1, 2, and 4 were obtained with this exact configuration, so readers can assess any potential selection bias.
  5. [Appendix B.4] The τ threshold in the computation-optimization analysis is an additional free hyperparameter, and Figure 7 shows that it trades 5% performance loss for 25% computation savings at τ=0.7. This is a useful analysis, but it should be referenced in the main text and its relationship to the 2.8% average degradation in Table 5 should be clarified, since the reported throughput numbers presumably include this computation skipping.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PoD's compression mechanism is defined from measured attention similarity and evaluated on held-out benchmarks; the tuning of proximal-token count and savings rate is standard hyperparameter selection, not a fit that forces the reported performance.

full rationale

PoD's derivation chain is: measure inter-layer attention similarity on held-out inputs, group consecutive layers into blocks, then share base-layer query/key logits for distant tokens within each block, followed by lightweight post-training adaptation on unlabeled data. The performance claim is supported by external long-context benchmarks (LongBench, LEval, InfiniteBench, Needle-in-a-Haystack) and standard benchmarks, and the layer grouping was determined from attention statistics rather than from the evaluation labels. The hyperparameter choice of 4K proximal tokens and a 35% savings rate is selected from Figure 3 using LEval; this is tuning of the final configuration, not a circular prediction, because no equation reduces the reported accuracy to the fitted hyperparameters by construction. Appendix C.1 derives the gating mechanism from standard softmax normalization using the layer's own logits, while Eq. (3) computes distant-token logits with the block base layer's Q and K. This mismatch is a modeling approximation and a potential correctness risk if the inter-layer similarity assumption fails, but it is not circular: the gating formula is defined by the method and its consequences are checked against external data, not assumed into the conclusion. No load-bearing self-citation chain or imported uniqueness theorem appears; the prior observations of attention redundancy are cited from independent sources. Therefore the paper's central derivation is self-contained with respect to its empirical evaluation, and no significant circularity is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a few empirical observations about attention behavior and on hyperparameters tuned by validation. The most fragile item is the unstated substitution of the block's lowest-layer query/key in Eq. (3), which the gating derivation in Appendix C.1 does not cover. The layer grouping itself is derived from attention similarities, not from benchmark labels, so it is not a fitted parameter in the strict sense, but the proximal-token budget and 35% saving target are chosen from ablations.

free parameters (5)
  • ns (initial proximal tokens) = 16
    Number of initial tokens kept with full KV; set following StreamingLLM/LM-Infinite and used in all PoD runs.
  • nr (recent proximal tokens) = 4080
    Number of recent tokens kept with full KV; with ns=16 gives 4096 proximal tokens, chosen in ablation (Figure 3).
  • Layer grouping similarity threshold = 0.5
    Threshold on JS divergence used in Algorithm 1 to decide whether two layers can share attention; choice is not derived from theory.
  • Target KV cache saving rate = 35%
    Resulting savings from the grouping, but the paper explicitly balances compression vs. LEval performance in Figure 3 (right), making it a tuned design target.
  • Gate threshold tau for computation skipping = 0.7
    Hyperparameter in Appendix B.4 controlling when distant-token computation can be skipped; used only for the computational optimization, not the main memory claim.
assumptions (5)
  • standard math Standard attention decomposition: the output can be separated into weighted sums over proximal and distant token groups (Eq. 5).
    Used in Appendix C.1 to justify the parameter-free gating in Eq. (4); it is exact for standard attention with the layer's own Q and K.
  • domain assumption Attention scores for distant tokens are highly similar between consecutive layers, and this similarity is strong enough to support key sharing.
    Observation 2, Figure 1(d) and Appendix B.2. This is the core redundancy the method exploits; the paper measures it on 1000 sequences but does not prove it holds beyond the tested model.
  • domain assumption Proximal tokens (initial and recent) carry most of the information needed for next-token prediction, so they can keep full KV while distant tokens are compressed.
    Observation 1, Figure 1(c). The paper shows a correlation between window size and identical prediction rate, not a guarantee that distant tokens never matter.
  • domain assumption Post-training on 5B tokens is sufficient for the model to adapt to the shared-key attention structure without losing general capabilities.
    Implied by the training recipe in Sec. 3; supported only by the reported benchmark scores, not by an independent analysis.
  • ad hoc to paper The gating decomposition derived for standard attention remains a valid weighting when distant-token scores are computed from the block's lowest layer instead of the current layer.
    Eq. (3) uses Q_ℓa and K_ℓa for aD while Appendix C.1 derives g for Qℓ and Kℓ; this substitution is necessary for the method to work but is not stated as an assumption or tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity." pith.science (2026). https://pith.science/paper/PUN4BJJK

@misc{pith2026241202252,
  author       = {Pith},
  title        = {Pith review of: Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PUN4BJJK}},
  note         = {Machine review of arXiv:2412.02252}
}
read the original abstract

The rapid expansion of context window sizes in Large Language Models~(LLMs) has enabled them to tackle increasingly complex tasks involving lengthy documents. However, this progress comes at the cost of a substantial increase in memory usage during inference, primarily due to the linear growth of the key-value~(KV) cache. Existing KV cache compression methods often discard less relevant tokens, which can lead to significant performance degradation when critical information is lost. In this paper, we propose \textsc{PoD}~(Proximal tokens over Distant tokens), a novel KV cache compression framework that allocates memory according to token importance, retaining less important tokens in a more compact, shared form rather than discarding them entirely. Our approach is motivated by two key observations: (1) proximal tokens -- those at the beginning and end of the context -- are significantly more important for next-token prediction, and (2) attention scores for distant tokens are highly redundant across consecutive layers. Leveraging these insights, \textsc{PoD} preserves the full KV cache for proximal tokens, while for distant tokens, it shares key states across layers. Since attention scores are determined by both queries and keys, sharing key states enables multiple layers to reuse a single set of keys for distant tokens, substantially reducing KV cache memory without discarding essential context. We further introduce a lightweight post-training adaptation to enable the model to adjust to this new attention-sharing structure. Extensive experiments on both synthetic~(Needle in a Haystack) and real-world long-context benchmarks demonstrate that \textsc{PoD} reduces KV cache memory usage by up to 35\% without compromising performance. Our method is orthogonal to existing token-selection-based techniques and can be combined with them for further KV cache compression.

Figures

Figures reproduced from arXiv: 2412.02252 by the authors.

Figure 1
Figure 1. Experimental results from the LLaMA3-8B-32K model, including: (a) prediction failure example, (b) benchmark [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the POD framework. Left: Example of head-wise layer partitioning based on inter-layer attention sim￾ilarity; Middle: Key states for distant tokens are shared across layers within each block to reduce KV cache memory; Right: Example of KV cache update in POD. mally, for each sample, we obtain n S ℓ,h i o 1≤ℓ≤L, 1≤h≤H = M(si), (1) where L and H denote the number of layers and attention heads, respectively,… view at source ↗
Figure 4
Figure 4. Case study of different methods. s ∗ n means re￾peating n times of the string s. + represents the concatena￾tion of strings. 4 Related Work Long-context LLMs face significant memory challenges due to their large parameter sizes and lengthy input sequences. Existing optimization approaches for reducing KV cache memory can be broadly categorized into three areas. Context Compression and Computation Optimization Many m… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Offline exploration of inter-layer attention sharing for P [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Visual Illustration of the Search Results for Needle in a Haystack [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Computation saving and performance loss rates vs. the gate threshold [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 6 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    GQA : Training generalized multi-query transformer models from multi-head checkpoints

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S. GQA : Training generalized multi-query transformer models from multi-head checkpoints. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 4895--4901, Singapore, December 2023. Associ...

    doi:10.18653/v1/2023.emnlp-main.298 2023

  3. [3]

    L -eval: Instituting standardized evaluation for long context language models

    An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., and Qiu, X. L -eval: Instituting standardized evaluation for long context language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 14388--14411, Bangkok, Thail...

    doi:10.18653/v1/2024.acl-long.776 2024

  4. [4]

    L ong B ench: A bilingual, multitask benchmark for long context understanding

    Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. L ong B ench: A bilingual, multitask benchmark for long context understanding. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    doi:10.18653/v1/2024.acl-long.172 2024

  5. [5]

    Codeplan: Repository-level coding using llms and planning

    Bairi, R., Sonwane, A., Kanade, A., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B., and Shet, S. Codeplan: Repository-level coding using llms and planning. Proceedings of the ACM on Software Engineering, 1 0 (FSE): 0 675--698, 2024

    work page 2024

  6. [6]

    E., and Cohan, A

    Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer. arXiv:2004.05150, 2020

    arXiv 2004

  7. [7]

    Leveraging redundancy in attention with reuse transformers

    Bhojanapalli, S., Chakrabarti, A., Veit, A., Lukasik, M., Jain, H., Liu, F., Chang, Y.-W., and Kumar, S. Leveraging redundancy in attention with reuse transformers. arXiv preprint arXiv:2110.06821, 2021

    arXiv 2021

  8. [8]

    Brandon, W., Mishra, M., Nrusimha, A., Panda, R., and Kelly, J. R. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405.12981, 2024

    arXiv 2024

Show all 51 references
  1. [9]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litw...

    1901

  2. [10]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Lazyllm: Dynamic token pruning for efficient long context llm inference, 2024

    Fu, Q., Cho, M., Merth, T., Mehta, S., Rastegari, M., and Najibi, M. Lazyllm: Dynamic token pruning for efficient long context llm inference, 2024. URL https://arxiv.org/abs/2407.14057

    2024 arXiv

  4. [12]

    Challenges in deploying long-context transformers: A theoretical peak performance analysis, 2024

    Fu, Y. Challenges in deploying long-context transformers: A theoretical peak performance analysis, 2024. URL https://arxiv.org/abs/2405.08944

    2024 arXiv

  5. [13]

    How to train long-context language models (effectively), 2024

    Gao, T., Wettig, A., Yen, H., and Chen, D. How to train long-context language models (effectively), 2024. URL https://arxiv.org/abs/2410.02660

    2024

  6. [14]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Rojas, D., Feng, G., Zhao, H., Lai, H., et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  7. [15]

    LM -infinite: Zero-shot extreme length generalization for large language models

    Han, C., Wang, Q., Peng, H., Xiong, W., Chen, Y., Ji, H., and Wang, S. LM -infinite: Zero-shot extreme length generalization for large language models. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Associ...

    2024 doi

  8. [16]

    A., Tanaka, M., Zhang, C., Zhang, M., Song, S

    Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, S. L., Rajbhandari, S., and He, Y. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509, 2023

    2023 arXiv

  9. [17]

    H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L

    Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490, 2024 a

    2024 arXiv

  10. [18]

    L ong LLML ingua: Accelerating and enhancing LLM s in long context scenarios via prompt compression

    Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. L ong LLML ingua: Accelerating and enhancing LLM s in long context scenarios via prompt compression. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Associat...

    2024 doi

  11. [19]

    Compressing context to enhance inference efficiency of large language models

    Li, Y., Dong, B., Guerin, F., and Lin, C. Compressing context to enhance inference efficiency of large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 6342--6353, Singa...

    2023 doi

  12. [20]

    Snapkv: Llm knows what you are looking for before generation

    Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024

    2024 arXiv

  13. [21]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. In Gibbons, P., Pekhimenko, G., and Sa, C. D. (eds.), Proceedings of Machine Lear...

    2024

  14. [22]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024 a

    2024 arXiv

  15. [23]

    Minicache: Kv cache compression in depth dimension for large language models

    Liu, A., Liu, J., Pan, Z., He, Y., Haffari, G., and Zhuang, B. Minicache: Kv cache compression in depth dimension for large language models. arXiv preprint arXiv:2405.14366, 2024 b

    2024 arXiv

  16. [24]

    Retrievalattention: Accelerating long-context llm inference via vector retrieval, 2024 c

    Liu, D., Chen, M., Lu, B., Jiang, H., Han, Z., Zhang, Q., Chen, Q., Zhang, C., Ding, B., Zhang, K., Chen, C., Yang, F., Yang, Y., and Qiu, L. Retrievalattention: Accelerating long-context llm inference via vector retrieval, 2024 c . URL https://arxiv.org/abs/2409.10516

    2024 arXiv

  17. [25]

    QLLM : Accurate and efficient low-bitwidth quantization for large language models

    Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B. QLLM : Accurate and efficient low-bitwidth quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024 d . URL https://openreview.net/forum?id=FIplmUWdm3

    2024

  18. [26]

    and Liu, B

    Mazumder, S. and Liu, B. Lifelong and Continual Learning Dialogue Systems. Springer, 2024

    2024

  19. [27]

    The jensen-shannon divergence

    Menéndez, M., Pardo, J., Pardo, L., and Pardo, M. The jensen-shannon divergence. Journal of the Franklin Institute, 334 0 (2): 0 307--318, 1997. ISSN 0016-0032. doi:https://doi.org/10.1016/S0016-0032(96)00063-4. URL https://www.sciencedirect.com/science/article/pii/S0016003296000634

    1997 doi

  20. [28]

    V., Qiu, L., and Zhang, D

    Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., R \"u hle, V., Yang, Y., Lin, C.-Y., Zhao, H. V., Qiu, L., and Zhang, D. LLML ingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Ku, L.-W., Martins, A., and Srikumar, V. (...

    2024 doi

  21. [29]

    Pytorch: An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  22. [30]

    Efficiently scaling transformer inference

    Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5: 0 606--624, 2023

    2023

  23. [31]

    Zero: memory optimizations toward training trillion parameter models

    Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. IEEE Press, 2020. ISBN 9781728199986

    2020

  24. [32]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, pp.\ 3505–35...

    2020

  25. [33]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  26. [34]

    Fast transformer decoding: One write-head is all you need

    Shazeer, N. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

    1911 arXiv

  27. [35]

    Flexgen: high-throughput generative inference of large language models with a single gpu

    Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., R\' e , C., Stoica, I., and Zhang, C. Flexgen: high-throughput generative inference of large language models with a single gpu. In Proceedings of the 40th International Conference on Machine Learning, I...

    2023

  28. [36]

    Dolma: an open corpus of three trillion tokens for language model pretraining research

    Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M., Ravichander,...

    2024

  29. [37]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

    2023 arXiv

  30. [38]

    QUEST : Query-aware sparsity for efficient long-context LLM inference

    Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., and Han, S. QUEST : Query-aware sparsity for efficient long-context LLM inference. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=KzACYw0MTV

    2024

  31. [39]

    M., Hauth, A., et al

    Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  32. [40]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  33. [41]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  34. [42]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Informatio...

    2017

  35. [43]

    Transformers: State-of-the-art natural language processing

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State...

    2020

  36. [44]

    and Tu, K

    Wu, H. and Tu, K. Layer-condensed KV cache for efficient inference of large language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 11175--11188, ...

    2024 doi

  37. [45]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=NG7sS51zVF

    2024

  38. [46]

    Sharing attention weights for fast transformer

    Xiao, T., Li, Y., Zhu, J., Yu, Z., and Liu, T. Sharing attention weights for fast transformer. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 , pp.\ 5292--5298. International Joint Conferences on Artificial Intelligence ...

    2019 doi

  39. [47]

    A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H

    Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H. Effective long-context scaling o...

    2024

  40. [48]

    B ench: Extending long context evaluation beyond 100 K tokens

    Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, Z., and Sun, M. B ench: Extending long context evaluation beyond 100 K tokens. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Associatio...

    2024 doi

  41. [49]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling

    Zhang, Y., Gao, B., Liu, T., Lu, K., Xiong, W., Dong, Y., Chang, B., Hu, J., Xiao, W., et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069, 2024 b

    2024 arXiv

  42. [50]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Re, C., Barrett, C., Wang, Z., and Chen, B. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Thirty-seventh Conference on Neural Information Processing Sys...

    2023

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.