Pith. sign in

REVIEW 3 major objections 6 minor 30 references

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A ladder-shaped KV cache stores different token spans in different layers, letting a fixed-size cache cover a longer effective context than recency-only caching.

desk verdict LaCache is a credible training-free KV cache scheme that improves on StreamingLLM on most benchmarks, but the paper's 'consistent gains' claim is undercut by its own RULER table and the hyperparameters are tuned on the same benchmarks used for reporting. read the letter →

arxiv 2507.14204 v1 pith:OHA27JS5 submitted 2025-07-14 cs.LG cs.AIcs.CL

Dachuan Shi , Yonggan Fu , Xiangchi Yuan , Zhongzhi Yu , Haoran You , Sixu Li , Xin Dong , Jan Kautz
show 2 more authors
Pavlo Molchanov Yingyan (Celine) Lin
This is my paper
classification cs.LGcs.AIcs.CL
keywords KVcachelong-contextinferencelargelanguagemodelsmemory-efficientgenerationtraining-freeStreamingLLMiterativecompactionladder-shaped
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LaCache is a training-free KV cache scheme that stores different spans of past tokens in different layers of a transformer, forming a ladder-like pattern. Under a fixed cache budget this retains more distinct tokens than recency-only windows, so the model sees a longer effective context. The paper claims this improves long-range language modeling and understanding, and an iterative compaction pass repeatedly re-compresses older caches to support continuous generation without running out of memory. If the claim holds, LaCache is a drop-in replacement for StreamingLLM-style caching with better accuracy at equal memory cost and compatibility with FlashAttention.

What carries the argument

The ladder-shaped cache pattern: each layer keeps a contiguous span of token KV states, and the span shifts rightward as layers go deeper, with two design parameters, the span $S$ (the number of consecutive layers storing the same token, setting the ladder's vertical extent) and the overlap $O$ (the per-layer overlap between neighboring cached segments). Two rationales support it: spreading coverage evenly across layers improves worst-case information retention, and overlapping adjacent cached segments smooths the fade-out of older tokens. The iterative compaction loop then re-applies this pattern to an already-compressed cache every time capacity is reached, so memory stays constant while older tokens are re-compressed more aggressively than newer ones.

What would settle it

Evaluate LaCache against a control with the ladder direction reversed at the same budget; if the reversed pattern matches or outperforms LaCache, the claimed benefit comes from span distribution rather than temporal ordering. Also measure accuracy on a task where an early fact must be combined with tokens in the most recent window in every layer; if shallow layers drop those recent tokens, such a task should expose a clear degradation.

Watch

Extended reading notes

Core claim

The paper's central claim is that a ladder-shaped KV cache, with early tokens preserved in shallow layers and progressively later tokens preserved in deeper layers, lets a fixed-size cache cover a wider span of past positions than keeping the same recent window in every layer. The ladder raises the lower bound of information retention across all positions, and because it never uses attention maps to select tokens, it works with IO-aware attention kernels. Benchmarks on Wikitext-2, PG19, LongBench, Needle-In-A-Haystack and RULER show lower perplexity and higher retrieval and understanding scores than StreamingLLM at identical budgets, and continuous generation on inputs longer than ten million tokens without OOM.

Load-bearing premise

The method assumes a transformer still predicts well when each layer attends to a different set of past positions, even though information dropped in one layer cannot be recovered by deeper layers unless deeper layers keep it.

Editorial extensions

If this is right

  • LaCache can serve as a training-free alternative to StreamingLLM-style windows, improving language modeling perplexity and long-context task scores at the same cache size.
  • Because token selection is attention-free, LaCache is compatible with FlashAttention and thus reaches higher throughput than importance-based eviction methods like H2O at similar accuracy.
  • Iterative compaction keeps the cache size constant regardless of generation length, so models can generate far beyond their pretraining context without running out of memory.
  • The two hyperparameters $S$ and $O$ offer a task-dependent trade-off: larger overlap helps tasks needing global context, while smaller overlap helps local question-answering tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension worth testing is whether the ladder's direction matters: a reversed ladder that keeps recent tokens in shallow layers and early tokens in deep layers would reveal whether the benefit comes from temporal ordering or simply from distributing spans across layers.
  • The layer-wise independence assumption could be probed by measuring attention entropy per layer under LaCache versus uniform windows; if shallow layers lose access to recent tokens, tasks requiring joint local and distant information should show degradation.
  • A testable variant is applying the same span-shifting pattern to encoder-decoder models or prefix-caching scenarios, where the ordering is document structure rather than time; this would generalize the insight beyond autoregressive decoding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes LaCache, a training-free KV cache eviction method that stores different token spans in different transformer layers according to a ladder-shaped pattern (early layers retain older tokens, deeper layers retain newer tokens), and couples this with an iterative compaction mechanism that re-applies the ladder pattern once the cache is full, enabling unbounded generation under a fixed cache budget. The method is evaluated on Wikitext-2, PG19, LongBench, Needle-In-A-Haystack, and RULER across several Llama, SmolLM, and LongChat models, with StreamingLLM as the main baseline and additional throughput/accuracy comparisons against H2O, TOVA, PyramidInfer, and SnapKV. The central claim is that, at equal KV memory, the ladder-shaped layer-to-token assignment retains a larger and more useful effective context than recency-only caching, improving long-range modeling and understanding while remaining OOM-safe.

Significance. If the central claim holds, LaCache is a practically valuable contribution: it is training-free, attention-map-free, and FlashAttention-compatible, and it addresses a real bottleneck in long-context and infinite-generation inference. The paper's consistently lower perplexity relative to StreamingLLM under identical budgets, and its very large Needle-In-A-Haystack gains, are encouraging, as is the public code release and the randomized-pattern Pareto analysis. However, the current validation does not fully separate the ladder's ordered shape from the larger number of unique cached tokens, several per-task RULER/LongBench results go against the 'consistently better' narrative, and the span/overlap hyperparameters are calibrated on the same benchmarks used for the headline comparisons. The idea is worth pursuing, but the evidence as presented is not yet conclusive enough for acceptance.

major comments (3)
  1. [Sec. 3.2 and Table 5] The load-bearing assumption that different layers can safely attend to different dropped token spans is asserted but never analyzed or directly tested. Table 5 shows this assumption is not uniformly satisfied: LaCache is below StreamingLLM on single2 (43 vs 49), single3 (26 vs 45), and multiquery (31 vs 45), and the +5.06 average is driven mainly by large gains on vt and cwe. The text's statement that the RULER results 'further validate the consistently better performance' is therefore not supported by the table. Please add either an error-propagation analysis for the layer-wise dropping, or a diagnostic experiment that varies which layers drop which spans and shows when deep layers can compensate for information dropped in shallow layers, and report the win/loss pattern across all 13 RULER tasks.
  2. [Sec. 4.4, Fig. 10, and Table 6] Span S and overlap O are tuned using ablations on the same datasets and task groups (Wikitext-2 for language modeling, LongBench QA/synthetic splits for understanding) that are later used to claim superiority in the main tables. Table 6 shows O has opposite effects on QA and synthetic tasks, so the reported averages may reflect benchmark-specific calibration rather than a general property of the ladder. Please specify the exact selection protocol (e.g., a fixed rule, a held-out validation split, or per-dataset tuning), report results over multiple seeds with error bars, and avoid describing benchmark-tuned settings as evidence for a universal 'consistent' advantage.
  3. [Sec. 3.2 and Sec. 4] No reported experiment isolates the ladder's ordered shallow-to-deep assignment from the larger number of unique cached token positions. Under a fixed budget, LaCache simply caches more distinct tokens than StreamingLLM, and the gains could come from that count alone rather than from the layer-to-span ordering. Figure 3 varies patterns and cache sizes jointly but does not hold the multiset of cached positions fixed, and its sampling distribution is not described. Please add an ablation that fixes the multiset of cached positions and compares (a) the proposed ladder, (b) a reversed ladder, (c) a shuffled assignment, and (d) StreamingLLM expanded to the same number of unique positions, all at equal total KV memory, to establish that the shape itself is the cause of the reported improvements.
minor comments (6)
  1. [Sec. 4.2, Table 2] The sentence 'LaCache consistently achieves lower PPL than StreamingLLM and the full cache setting under the same decoding length' is incorrect as stated: at 1K--8K, LaCache PPL (7.13, 7.44, 7.99, 8.36) is substantially higher than full-cache PPL (4.28, 4.39, 5.82, 6.16), and at 16K it beats the full cache only because the full cache explodes to 109.94. Please rephrase to claim improvement over StreamingLLM, and note explicitly that LaCache is worse than the full cache at short decoding lengths.
  2. [Sec. 4.4, Table 6] The claim that 'a larger overlap consistently improves performance' on global tasks is based only on two aggregated groups (QA and synthetic). Please report the per-task breakdown for the four QA and three synthetic tasks named in the text, since the aggregation could hide large individual differences.
  3. [Fig. 3] The randomized-pattern experiment is underspecified: the paper does not state how the 1,500+ patterns were sampled (layer-wise Bernoulli? fixed cache sizes? which model and sequence length?), how many seeds were used, or what the axes exactly represent. Please provide this methodology so the Pareto claim can be reproduced.
  4. [Table 5] The RULER task names are inconsistently formatted ('single 1', 'single2', 'multikey1', etc.) and undefined. Please define each abbreviation and add a note on whether the per-task differences are within run-to-run noise, especially given the unusual pattern of large gains and large losses.
  5. [Sec. 4.1 and Fig. 5/6] The PG19 experimental setup is ambiguous: the text says a sliding window of 256 tokens is used and that 'the compacted KV cache output from each window is then passed to generate subsequent tokens,' but it is not clear how LaCache's iterative compaction interacts with this window, nor how the full-cache baseline is run up to 160K before OOM. Please clarify the procedure and add axis labels to Figures 5 and 6.
  6. [Throughout] Minor textual issues: 'TOV A' should be 'TOVA'; footnote 1's 'to avoid bubbles' should be replaced by a precise boundary-padding rule; and the claim of 'no additional computation or storage costs' in Sec. 4.3 should be qualified as 'at the same cache budget as StreamingLLM,' since the compaction loop itself has compute overhead.

Circularity Check

1 steps flagged · score 4.0 of 10

LaCache's benchmark validation partly reduces to hyperparameter calibration: the span S and overlap O are explicitly tuned on Wikitext-2 and LongBench task groups, and the same benchmarks are then reported as consistent validation; the central ladder mechanism itself is not definitionally forced.

  1. fitted input called prediction [Sec. 3.2 Implementation details; Sec. 4.4 Ablation Studies; validation claims in Sec. 4.2-4.3 (Tables 1, 3, 4)]
    "As demonstrated in Sec. 4, we calibrate these two design factors to minimize cache redundancy and maximize generation accuracy. ... S is set to 1/4 of the number of model layers, which was given by the empirical results from our ablation studies, as shown in Fig. 10, where Llama2-7B-Chat model under a 256 KV cache budget and the Wikitext-2 dataset are used for examining the impact of hyperparameters. ... The choice of Overlap O depends on the task type."

    The method's two design factors are explicitly calibrated on the same evaluation data that is later presented as validation. S is chosen from Wikitext-2 perplexity ablations (Fig. 10) and then Table 1 reports Wikitext-2 perplexity as evidence that LaCache 'consistently shows stronger capabilities' than StreamingLLM. O is chosen per task type from LongBench subtask ablations (Table 6) and then Tables 3-4 report LongBench averages as validation. The headline improvements (e.g., 5% vs 35% degradation on Wikitext-2) are partly selected-for rather than predicted, so the 'consistent validation' claim is partially circular. The central ladder idea still has independent content, and results on NIAH, PG19, and RULER provide some external check.

full rationale

The paper contains no formal derivation whose conclusion equals its premise, no load-bearing self-citation chain, and no uniqueness theorem imported from the authors' prior work. The method's core idea (different layers cache different token spans; iterative compaction re-applies the pattern) is a genuine algorithmic proposal evaluated against StreamingLLM, H2O, TOVA, PyramidInfer, and SnapKV. The main circular element is benchmark-specific hyperparameter calibration: Sec. 3.2 says the span S and overlap O are 'calibrate[d]' in Sec. 4, and Sec. 4.4 confirms that S was chosen from Wikitext-2 ablations and O from LongBench subtask ablations. Reporting improvements on exactly those benchmarks as 'consistent validation' makes part of the empirical claim self-confirming. However, the effect is not definitional: the ladder pattern does not reduce to the metric being predicted, the same hyperparameters transfer across several models, and external benchmarks (Needle-In-A-Haystack, PG19, RULER) are also reported; RULER in fact shows several tasks where LaCache is far worse than StreamingLLM (single2 43 vs 49, single3 26 vs 45, multiquery 31 vs 45), which the paper itself acknowledges indirectly by calling the pattern 'not optimal for every scenario.' Weighing the explicit calibration against the independent comparisons, the appropriate score is 4, not higher: the central claim still has independent content, but one load-bearing validation route is partly fitted.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on empirical tuning rather than formal derivation. The two shape parameters S and O are fitted per task family on the same benchmarks used for the main tables, and the layer-wise independence assumption is stated without analysis. There are no invented entities; the method only rearranges existing KV states.

free parameters (3)
  • Span S = 1/4 of layers for language modeling (Wikitext-2); 1/2 of layers for LongBench
    Defines how many consecutive layers retain the same token. Tuned via ablation on Wikitext-2 and LongBench, the same benchmarks used for main results (Sec. 4.4, Fig. 10, Tab. 6).
  • Overlap O = S/2 for language modeling; S/4 or S/2 for long-context understanding
    Number of tokens preserved per layer. Chosen per task type based on ablation on LongBench QA and synthetic subsets (Sec. 4.4, Tab. 6).
  • Edge padding adjustment = Unspecified ('slightly more positions')
    Footnote 1: more positions are preserved for tokens at the beginning and end of ladders to avoid bubbles; the amount is not quantified.
assumptions (3)
  • domain assumption Different layers can cache disjoint or partially overlapping token sets without breaking the Transformer's forward computation.
    Invoked in Sec. 3.2 'The key insight'; the method's feasibility rests on this, but error propagation across layers is not analyzed.
  • domain assumption Neighboring tokens have higher semantic relevance, so overlapping cache segments provide smooth information fade-out.
    Sec. 3.2 'Further analysis', second rationale; no empirical or theoretical support beyond the random-pattern scatter.
  • ad hoc to paper In the worst case, important tokens may appear in the layer with least coverage, so equalizing coverage across layers improves the lower bound of information retention.
    Sec. 3.2 'Further analysis', first rationale; stated as a 'lower bound' but no formal bound is proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models." pith.science (2026). https://pith.science/paper/OHA27JS5

@misc{pith2026250714204,
  author       = {Pith},
  title        = {Pith review of: LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OHA27JS5}},
  note         = {Machine review of arXiv:2507.14204}
}
read the original abstract

Recent advancements in Large Language Models (LLMs) have spurred interest in numerous applications requiring robust long-range capabilities, essential for processing extensive input contexts and continuously generating extended outputs. As sequence lengths increase, the number of Key-Value (KV) pairs in LLMs escalates, creating a significant efficiency bottleneck. In this paper, we propose a new KV cache optimization paradigm called LaCache, a training-free method for efficient and accurate generative inference of LLMs. LaCache enables LLMs to simultaneously address both of the critical challenges in long-range modeling: robust long-range capabilities and continuous generation without running out-of-memory (OOM). Specifically, LaCache integrates two key innovations: (1) a ladder-shaped KV cache pattern that stores KV pairs not only sequentially (left-to-right within each layer) but also across layers (from shallow to deep), providing an extended span for capturing long-range dependencies under a fixed storage budget, thereby boosting long-range capabilities; and (2) an iterative compaction mechanism that progressively compresses older caches, freeing up space for new tokens within a fixed cache size. This token distance-based dynamic compression enables more effective continuous generation under constrained cache budgets. Experiments across various tasks, benchmarks, and LLM models consistently validate LaCache's effectiveness in enhancing LLMs' long-range capabilities. Our code is available at https://github.com/GATECH-EIC/LaCache.

Figures

Figures reproduced from arXiv: 2507.14204 by the authors.

Figure 1
Figure 1. Illustrative comparisons among (a) recency-based KV cache (Xiao et al., 2023b), (b) retrieval-based KV cache (Tang et al., 2024), and (c) our proposed LaCache featuring a ladder-shaped pattern. Previous KV cache storage strategies struggle to simultaneously balance the needs for both continuous generation without OOM and robust long-range capabilities. In contrast, our proposed LaCache allows LLMs to simultaneously … view at source ↗
Figure 2
Figure 2. An illustration of LaCache’s KV cache storage pat￾tern. LaCache is leveraged to compact the original full KV cache into a compressed, ladder-shaped pattern, allowing for the storage of information from longer-range tokens compared to StreamingLLM (Xiao et al., 2023b) under the same KV cache budget, thereby providing stronger long-range sequence modeling. of most recent tokens across all layers, our key insight is th… view at source ↗
Figure 4
Figure 4. An illustration of LaCache’s iterative compaction. Iter￾ative compaction is introduced to support continuous generation without running out of memory, even for infinite-length genera￾tion. Once the KV cache reaches the predefined size, LaCache’s ladder-shaped pattern is applied to the already-compacted KV cache, freeing up space for new tokens. Implementation details. To balance storage efficiency and generation acc… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Evaluate LaCache on the first ten books of the concate￾nated PG19 dataset, corresponding to a length of 600K tokens [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Evaluate LaCache on the entire concatenated PG19 dataset, corresponding to a length of 10 million tokens. consistently achieves lower PPL than StreamingLLM and the full cache setting under the same decoding length. 4.3. Long-Context Understanding Benchmarks Benchmark o…
Figure 7
Figure 7. Figure 7: Evaluate the score-throughput trade-offs on a single H200 GPU with StreamingLLM (Xiao et al., 2023b), H2O (Zhang et al., 2024), TOVA (Oren et al., 2024), PyramidInfer (Yang et al., 2024), and SnapKV (Li et al., 2024) on LongBench (Bai et al., 2023). The top left subfig…
Figure 8
Figure 8. Figure 8: Benchmark LaCache and StreamingLLM on Needle￾In-A-Haystack (Fu et al., 2024a) using Llama3.2-3B-Instruct￾128k (Dubey et al., 2024) under a 50% cache budget setting. Greener indicates better performance [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Benchmark LaCache and StreamingLLM on Needle-In￾A-Haystack (Fu et al., 2024a) using LongChat-7b-v1.5-32k (Li et al., 2023) under a 25% cache budget setting. Greener indicates better performance. The experimental results further validate the consistently better performa…
Figure 10
Figure 10. Figure 10: Ablation studies on the hyperparameters for language modeling. Perplexity (lower is better) is reported in the figure [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 3 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [5]

    Flashattention-2: Faster attention with bet- ter parallelism and work partitioning

    Dao, T. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691,

    arXiv

  3. [8]

    Data engineering for scaling language models to 128k context.arXiv preprint arXiv:2402.10171, 2024a

    Fu, Y ., Panda, R., Niu, X., Yue, X., Hajishirzi, H., Kim, Y ., and Peng, H. Data engineering for scaling language models to 128k context.arXiv preprint arXiv:2402.10171, 2024a. Fu, Y ., Yu, Z., Li, J., Qian, J., Zhang, Y ., Yuan, X., Shi, D., Yakunin, R., and Lin, Y . C. Amoeballm: Constructing any-shape large language models for efficient and instant de...

    arXiv

  4. [9]

    google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/ #gemini-2-5-thinking

    URL https://blog. google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/ #gemini-2-5-thinking . Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv 2025

  5. [10]

    Lm-infinite: Zero-shot extreme length generalization for large language models

    Han, C., Wang, Q., Peng, H., Xiong, W., Chen, Y ., Ji, H., and Wang, S. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pp. 3991–4008,

    work page 2024

  6. [11]

    Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654,

    Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y ., and Ginsburg, B. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654,

    arXiv

  7. [12]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    10 LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276,

    arXiv

  8. [13]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

Show all 30 references
  1. [14]

    Reformer: The efficient transformer

    Kitaev, N., Kaiser, Ł., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451,

    2001 arXiv

  2. [15]

    How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following,

    Li, D., Shao, R., Xie, A., Sheng, Y ., Zheng, L., Gonzalez, J., Stoica, I., Ma, X., and Zhang, H. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following,

    2023

  3. [16]

    Oren, M., Hassid, M., Adi, Y ., and Schwartz, R

    URL https://cdn.openai.com/ gpt-4-5-system-card-2272025.pdf . Oren, M., Hassid, M., Adi, Y ., and Schwartz, R. Transform- ers are multi-state rnns. arXiv preprint arXiv:2401.06104,

    arXiv

  4. [17]

    Pytorch: An imperative style, high-performance deep learning library

    Paszke, A. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703,

    1912 arXiv

  5. [18]

    Yarn: Efficient context window extension of large language models

    Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071,

    arXiv

  6. [20]

    Quest: Query-aware sparsity for efficient long-context llm inference

    Tang, J., Zhao, Y ., Zhu, K., Xiao, G., Kasikci, B., and Han, S. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774,

    arXiv

  7. [21]

    S., Love, J., et al

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi`ere, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295,

    arXiv

  8. [22]

    Llama: Open and efficient foundation lan- guage models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971,

    arXiv

  9. [23]

    D2o: Dynamic dis- criminative operations for efficient generative inference of large language models

    Wan, Z., Wu, X., Zhang, Y ., Xin, Y ., Tao, C., Zhu, Z., Wang, X., Luo, S., Xiong, J., and Zhang, M. D2o: Dynamic dis- criminative operations for efficient generative inference of large language models. arXiv preprint arXiv:2406.13035,

    arXiv

  10. [24]

    Z., Khabsa, M., Fang, H., and Ma, H

    Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768,

    2006 arXiv

  11. [25]

    Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks

    Wang, Z., Jin, B., Yu, Z., and Zhang, M. Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks. arXiv preprint arXiv:2407.08454,

    arXiv

  12. [26]

    Huggingface’s transformers: State-of-the- art natural language processing

    11 LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models Wolf, T. Huggingface’s transformers: State-of-the- art natural language processing. arXiv preprint arXiv:1910.03771,

    1910 arXiv

  13. [27]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning , pp. 38087–38099. PMLR, 2023a. Xiao, G., Tian, Y ., Chen, B., Han, S., and...

    arXiv

  14. [28]

    Effectively com- press kv heads for llm

    Yu, H., Yang, Z., Li, S., Li, Y ., and Wu, J. Effectively com- press kv heads for llm. arXiv preprint arXiv:2406.07056,

    arXiv

  15. [29]

    Superficial self-improved reasoners benefit from model merging

    Yuan, X., Zhang, C., Liu, Z., Shi, D., V osoughi, S., and Lee, W. Superficial self-improved reasoners benefit from model merging. arXiv preprint arXiv:2503.02103,

  16. [30]

    Towards the law of capacity gap in distilling language models

    Zhang, C., Song, D., Ye, Z., and Gao, Y . Towards the law of capacity gap in distilling language models. arXiv preprint arXiv:2311.07052,

    arXiv

  17. [2019]

    Shah, J., Bikshandi, G., Zhang, Y ., Thakkar, V ., Ramani, P., and Dao, T

    URL https://arxiv.org/abs/1911.05507. Shah, J., Bikshandi, G., Zhang, Y ., Thakkar, V ., Ramani, P., and Dao, T. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37:68658–68685,

    1911 arXiv

  18. [2020]

    Extending context window of large language models via positional interpolation

    Chen, S., Wong, S., Chen, L., and Tian, Y . Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595,

    arXiv

  19. [2022]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  20. [2023]

    E., and Cohan, A

    Beltagy, I., Peters, M. E., and Cohan, A. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150,

    2004 arXiv

  21. [2024]

    Gptq: Accurate post-training quantization for generative pre- trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323,

    arXiv

  22. [2025]

    Bai, Y ., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y ., Tang, J., and Li, J

    URL https: //www.anthropic.com/news/claude-4. Bai, Y ., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y ., Tang, J., and Li, J. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.