Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single 48GB GPU can serve 3-million-token contexts without fine-tuning, via staged token pruning and KV offload.

desk verdict A solid systems paper with real gains at 128K–256K contexts, but the 3M-token headline is a memory/throughput claim, not a verified accuracy claim. read the letter →

arxiv 2502.08910 v1 pith:CMZ6LELQ submitted 2025-02-13 cs.CL cs.LG

Heejun Lee , Geon Park , Jaduk Suh , Sung Ju Hwang This is my paper · ORCID
classification cs.CLcs.LG
keywords long-contextLLMinferencesparseattentiontokenpruningKVcacheoffloadingout-of-lengthgeneralizationRoPEadjustmenttraining-freeblock
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

InfiniteHiP sets out to solve three problems at once: the quadratic cost of attention in long contexts, the GPU memory pressure of the key-value cache, and the failure of pretrained LLMs to extrapolate beyond their training context length. Its claim is that all three can be addressed without any fine-tuning by pruning each query's context down to a few thousand important tokens, offloading the rest of the KV cache to host memory, and applying layer-dependent rotary-position adjustments. If the claim holds, a single 48GB GPU can handle contexts up to 3 million tokens, three times what the authors say is otherwise possible on their hardware, while attention decoding on a 1-million-token context runs about 19x faster than dense FlashAttention-2.

What carries the argument

Modular hierarchical context pruning is the central mechanism: a cascade of pruning stages where each stage partitions the current candidate key indices into contiguous chunks, selects one representative token per chunk by hierarchical top-1 estimation, estimates each chunk's attention score as the max over heads and query positions of the representative's score, and keeps only the top-$K$ chunks for the next stage. The candidate count drops from the full context to 32K, then 8K, then a final 2K-4K window per query. The same kernel is reused across stages, and the output is a paged block-sparse attention mask. Two supporting mechanisms carry the practical claims: per-stage mask caching with refresh intervals of 16, 8, and 4 decoding steps, and an LRU-based KV cache offloaded to CPU memory with dynamically loaded GPU banks.

What would settle it

Run a single-needle retrieval task at 256K tokens with the needle placed inside a chunk whose representative token has low attention score; if InfiniteHiP fails to retrieve while dense attention succeeds, the chunk-representative assumption is falsified. Alternatively, compute the attention-score recall of the selected chunks against full softmax attention on the same prompt, and look for a systematic gap.

Watch

Extended reading notes

Core claim

The central discovery is an empirical pattern plus an algorithm that exploits it: within a typical LLM attention matrix, the tokens that matter are concentrated in a small number of chunks, so a block-sparse mask built from the right chunks is a faithful approximation of the top-k tokens. InfiniteHiP builds that mask with modular pruning stages. Each stage divides surviving key indices into fixed-size chunks, picks one representative token per chunk using a hierarchical top-1 search that runs in $O(\log^2 l_c)$ time, scores each chunk as the maximum attention score of its representative across heads and query positions, keeps only the top-$K$ chunks, and feeds the survivors to the next stage. Three such stages reduce a multi-million-token context to a 2048- or 4096-token window per query. A per-stage mask cache exploits temporal locality during decoding, and an LRU-managed KV cache offloads cold tokens to host memory. On top of this, the method assigns different RoPE adjustments to different layers, chunk-indexed positions in the first three layers and relative-style positions elsewhere, so that pretrained models can attend beyond their original length. The paper reports that this combination preserves or improves long-context benchmark accuracy relative to prior sparse-attention baselines while cutting both latency and VRAM.

Load-bearing premise

The load-bearing premise is that one representative token per chunk, found by hierarchical top-1 selection, estimates the chunk's true maximum attention score accurately enough that discarding the lower-scoring chunks loses no information the model would have used.

Editorial extensions

If this is right

  • A pretrained model can be served at 3 million tokens on a single 48GB GPU without fine-tuning, using about 3.34% of the VRAM that dense attention would need.
  • Attention decoding on a 1-million-token context runs 18.95x faster than dense FlashAttention-2, and end-to-end decoding on a 3M-token context is 7.24x faster than the reference serving runtime.
  • Long-context benchmarks LongBench and ∞Bench show equal or better scores than InfLLM while attending to a 3K-5K token window rather than a 12K one.
  • Short-context models trained on 4K or 8K tokens gain usable performance on 128K-256K contexts through the layer-dependent RoPE adjustments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The per-stage mask cache makes decoding cost nearly independent of context length, so throughput per token stops growing with history; a direct test would be measuring throughput at 6M tokens on the same GPU.
  • The KV offloading framework is orthogonal to other cache-compression techniques, so combining it with quantized or compressed KV caches could plausibly push the same hardware envelope past 3 million tokens.
  • The hierarchical top-1 estimator is a general primitive that could be reapplied to retrieval, RAG, or long-form span selection where picking informative contiguous segments matters.
  • Because the method is training-free and implemented as a serving-layer change, it can wrap already-deployed instruction-tuned models without altering their weights.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. InfiniteHiP is a training-free inference framework for long-context LLMs that combines three mechanisms: (i) multi-stage hierarchical block pruning that selects a small set of key chunks per query block using representative tokens; (ii) a RoPE adjustment scheme that applies chunk-indexed RoPE in early layers and relative-style RoPE in later layers for out-of-length generalization; and (iii) KV-cache offloading to host memory with an LRU-managed GPU key bank. The paper reports quality improvements over InfLLM on LongBench and ∞Bench, additional RULER and InfiniteBench results, passkey retrieval up to 1M tokens, and latency/throughput measurements claiming up to 3M-token contexts on an L40s 48GB GPU, with an 18.95x attention-decoding speedup at 1M tokens.

Significance. The contribution is practically significant if the supported claims hold: a drop-in, training-free replacement for dense attention that extends usable context on commodity GPUs would be valuable for serving. The paper's strengths are its detailed system implementation (Triton kernels, SGLang integration, stage caching, LRU offload), honest ablations of the RoPE choices (Table 5, Table 6), and comparative evaluation on several standard long-context benchmarks. However, the headline 3M-token claim is currently a memory/throughput extrapolation; no accuracy or recall measurement at 3M is provided, and part of the speedup rests on estimated baselines. The central algorithmic idea is plausible and well supported at 128K–256K, but the advertised capability goes beyond the evidence.

major comments (4)
  1. [§5.3, Figure 5, Tables 11–12] The 3M-token claim is not demonstrated by measured end-to-end results. All L40S entries at T ≥ 512K and all RTX 4090 entries at T ≥ 1024K in Tables 11–12 are marked estimated, and Figure 5's caption states that the dashed lines are estimated values based on previous trends. The abstract sentence that InfiniteHiP 'enables the processing of up to 3 million tokens on a single L40s 48GB GPU' is therefore only a memory-capacity and extrapolated-throughput statement. Please either measure at least one end-to-end decoding run at 3M, or rephrase the headline claim to distinguish measured capability from extrapolation.
  2. [Abstract; §5.2, Table 7; Appendix E.1] The phrase 'without any permanent loss of context information' is a claim about accuracy preservation, but the only retrieval evidence at extreme length is the passkey table (Table 7), which stops at 1M tokens; the 3M rows in Figure 5 and Tables 11–12 report throughput and VRAM only. The phrase is also potentially misleading because the pruning mask still discards unselected chunks for the current query block; the 'no permanent loss' property refers to KV-cache retention, not to attention coverage. Please clarify the claim and add a quality benchmark or top-k recall measurement at 1M–3M.
  3. [§4, Algorithms 2–3; Figure 6a] The accuracy of the advertised length rests on the assumption that a single representative token per chunk, found by hierarchical top-1 selection, estimates the chunk's maximum attention score well enough to prune safely. At 3M tokens, Stage 1 keeps k^(1)/l_c^(1) = 32768/256 = 128 chunks out of roughly 12,000, i.e., about 1.1% of chunks, before Stages 2–3 reduce the selected set to 2K–3K tokens. A false negative in one representative token therefore discards an entire chunk in a regime where no recall or quality measurement is reported. Figure 6a reports recall only at an unspecified and apparently much shorter context. Please report representative-token recall as a function of context length up to at least 1M, or provide task accuracy at 1M–3M.
  4. [Tables 3–4] The 18.95x attention-decoding speedup at 1M tokens relies on a FlashAttention2 baseline that the paper itself says cannot run beyond 128K on the test GPU. Table 4's footnote states that FA2 does not support KV offloading and thus cannot run decoding with a context window exceeding 128K, and that FA2 results are estimated by layer-wise simulation. Table 3, however, lists a measured-looking FA2 decode latency of 4645 μs at T=1024K. The manuscript should reconcile these two statements and report which FA2 numbers are measured, which are simulated, and how the simulation was validated.
minor comments (5)
  1. [Appendix F] The relationship between the '3K' preset and the hyperparameter table is easy to miss because k^(3) is given as '2048 (4096 for l≤3)' while the default is called 3K; please add a table that maps each preset name to all hyperparameters.
  2. [Table 4, Figure 7] The tables and figure use 'Stage 0' although the paper defines N pruning stages S^(1) through S^(N); clarify whether 'Stage 0' refers to the hierarchical top-1 selection inside a stage.
  3. [Figure 5] The acronym 'SRT' is not defined in the text or caption; define it as the SGLang Runtime with FlashInfer and state the exact FlashInfer configuration used.
  4. [Table 2 caption] The caption ends with 'See the caption on Table 1 on Abs. Rel.', but Table 1 uses 'Avg. Rel.'; unify the notation between the two tables.
  5. [§5.2] The claim that the method 'processes 4x fewer key tokens' should state whether this counts tokens entering the sparse attention or the total number of tokens read during pruning, since the three-stage pipeline reads more tokens during pruning than the final window size.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper's central speed and quality claims are measured against external benchmarks, with self-citations serving as components rather than as load-bearing justifications.

full rationale

InfiniteHiP is an empirical systems paper. Its headline capability claims (3M-token context on a single L40s 48GB GPU, 18.95x attention-decoding speedup at 1M tokens) are supported by measured latency tables (Tables 3, 4, 11, 12) and by benchmark evaluations on external suites (LongBench, ∞Bench, RULER, InfiniteBench, Passkey). No reported accuracy result is obtained by construction from a fitted parameter: hyperparameters such as chunk sizes, window sizes, and refresh intervals are stated in Appendix F and described as chosen empirically, which is standard model selection rather than a circular prediction. The self-citations to the authors' prior HiP work (Lee et al., 2024b) are used to borrow a representative-token selection routine and the notion of attention locality; these are components whose usefulness is re-validated here against full-attention recall and external benchmarks, not premises that are assumed true solely because the authors cited themselves. The 3M-token throughput numbers in Table 12 and Figure 5 are explicitly labeled as estimated/extrapolated; this is an evidence limitation about unverified accuracy at the advertised length, not a circular derivation. Overall, the derivation chain is self-contained with respect to circularity concerns, and the only minor blemish is the presence of self-citations that are not load-bearing in a circular sense.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The framework is deliberately heuristic: all parameters that determine the sparse window and mask refresh cadence are chosen empirically, and the core assumptions about attention sparsity and mask temporal locality are not derived from first principles. This does not make the paper circular, but it means the headline numbers are tied to the chosen presets.

free parameters (7)
  • Sparse window size k(3) = 2048 default, 4096 for layers 1-3; 4096 for 5K preset
    Controls the number of tokens actually attended after stage 3; chosen as a latency-quality trade-off, not derived.
  • Chunk sizes l_c per stage = (256, 32, 8) for 3K preset; (64, 32, 16) for 5K preset
    Chunk granularity in each pruning stage; selected empirically for accuracy and parallelism.
  • Stage token budgets k(1), k(2) = (32K, 8K) for 3K preset; (32K, 16K) for 5K preset
    How many candidate tokens survive each stage; hand-picked based on latency and benchmark scores.
  • Mask refresh intervals n_refresh = (16, 8, 4); fast (32, 16, 8); flash (96, 24, 8)
    How often each pruning stage mask is recomputed during decoding; fast and flash variants trade accuracy for speed.
  • Sink and streaming token counts = nsink=256, nstream=1024
    Tokens always retained; inherited from StreamingLLM and HiP, but still a tunable choice.
  • RoPE layer split = Chunk-indexed RoPE for layers 1-3, Relative RoPE for layers 4+
    Selected via ablation in Table 6; not derived from a theoretical account of layer-wise attention.
  • Query block size b_q = 64
    Query chunk granularity used in all pruning stages; not systematically varied.
assumptions (4)
  • domain assumption Attention keys that matter for a query block are concentrated in a small number of contiguous chunks, and representative-token scores estimate chunk maximum scores.
    This motivates the pruning algorithm (Section 3, Figure 2a). If false, pruning will discard important tokens.
  • domain assumption The sparse attention mask for a pruning stage remains valid for n_refresh decoding steps.
    Mask caching assumes temporal locality of attention patterns; validated only indirectly through benchmark scores.
  • domain assumption RoPE adjustments such as chunk-indexed, relative, and StreamingLLM-style positions preserve the model's learned semantics outside the pretrained context window.
    Out-of-length generalization is shown empirically, not proven; the chosen layer-wise split may not generalize to other model families.
  • domain assumption Offloading KV cache to CPU memory through UVM does not change attention outputs, only latency.
    The framework assumes the page table and UVM mechanisms are correct; a systems-level assumption that the paper does not formally verify.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU." pith.science (2026). https://pith.science/paper/CMZ6LELQ

@misc{pith2026250208910,
  author       = {Pith},
  title        = {Pith review of: InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CMZ6LELQ}},
  note         = {Machine review of arXiv:2502.08910}
}
read the original abstract

In modern large language models (LLMs), handling very long context lengths presents significant challenges as it causes slower inference speeds and increased memory costs. Additionally, most existing pre-trained LLMs fail to generalize beyond their original training sequence lengths. To enable efficient and practical long-context utilization, we introduce InfiniteHiP, a novel, and practical LLM inference framework that accelerates processing by dynamically eliminating irrelevant context tokens through a modular hierarchical token pruning algorithm. Our method also allows generalization to longer sequences by selectively applying various RoPE adjustment methods according to the internal attention patterns within LLMs. Furthermore, we offload the key-value cache to host memory during inference, significantly reducing GPU memory pressure. As a result, InfiniteHiP enables the processing of up to 3 million tokens on a single L40s 48GB GPU -- 3x larger -- without any permanent loss of context information. Our framework achieves an 18.95x speedup in attention decoding for a 1 million token context without requiring additional training. We implement our method in the SGLang framework and demonstrate its effectiveness and practicality through extensive evaluations.

Figures

Figures reproduced from arXiv: 2502.08910 by the authors.

Figure 1
Figure 1. Overview of InfiniteHiP. (a) Infinitely growing KV cache: In InfiniteHiP, the context keys and values are stored in a unified memory space, where some of the keys and values are loaded on GPU memory. (b) Configurable modular pruning: Each pruning stage narrows down the candidate key indices based on the current query block. During pruning, if a cache miss is encountered, the missing tokens are dynamically loaded and… view at source ↗
Figure 2
Figure 2. Design of our Context Pruning Algorithm. 2. Related Works Previous studies have proposed dynamic token selection for efficient LLM inference for long contexts. MInfer￾ence (Jiang et al., 2024) classifies attention heads into two types to estimate the sparse attention pattern, which is used to drop less important tokens before the dot product. While this method considerably speeds up the prefill stage, it can￾not be … view at source ↗
Figure 3
Figure 3. Results with Llama3.1 8B. 0 50 100 150 200 250 30 40 50 60 70 En.MC Acc. (%) 0 50 100 150 200 250 Context Length (k) 20 30 40 En.QA Recall. (%) Gemma2 Exaone3 Exaone3.5 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Results with Short Con￾text Models. Star (★)-shaped mark￾ers indicate out-of-length generaliza￾tion results. scores are selected for the next stage, as follows: I (i) m′ = ⋃ ȷˆ∈T (i) m C (i) m,ȷˆ , (1) where T (i) m = arg top j K(i) (s (i) m,j ), (2) and m′ = { ⌈m ⋅ b …
Figure 5
Figure 5. Figure 5: SGlang Decoding Throughput Benchmark. Dashed lines are estimated values. RTX4090 has 24GB and L40s has 48GB of VRAM. We used is AWQ Llama3.1 with FP8 KV cache. 5.2. Results LongBench. In [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Analysis [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Visualization of Stage Caching During Decoding. The visualized mask refresh interval hyperparameter n (1,2,3) refresh = (16, 8, 4) for simplicity. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Visualziation of RoPE Adjustment. In [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Visualization of Each Stage of Each RoPE Adjustment Method. From left, we visualize the output of stages 1, 2, and 3. We use Llama 3.2 1B and T=256K. The model’s pretrained context length is 128K. The horizontal axis represents the key sequence dimension, and the verti…
Figure 10
Figure 10. Figure 10: Generated Mask Example. We use Llama 3.1 8B with T=64K PG19 sample without the RoPE extend mechanism. Refer [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Visualization of Approximating Sliding Window with Block Sparse Attention. (Left) Cropped generated block sparse mask from 13th layer from [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Controllably Efficient Language Models

    cs.LG 2025-11 conditional novelty 6.0 of 10

    A single transformer variant can compress past context into chunk summaries and use chunk size as a test-time knob to trade quality against speed and memory, outperforming many efficient baselines on recall benchmarks.

  2. FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression

    cs.CL 2025-07 conditional novelty 5.0 of 10

    FAEDKV compresses the KV cache by keeping the most important Fourier-frequency components of cached keys and values, reporting gains over eviction baselines on LongBench and position-flat retrieval on needle-in-a-hays...

Reference graph

Works this paper leans on

33 extracted references · 7 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023

    arXiv 2023

  3. [3]

    NTK - Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation., June 2023

    bloc97. NTK - Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation., June 2023. URL www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/

    work page 2023

  4. [4]

    Y., Ermon, S., Rudra, A., and Ré, C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAttention : Fast and memory-efficient exact attention with IO -awareness, 2022. URL http://arxiv.org/abs/2205.14135

    arXiv 2022

  5. [5]

    Flash-decoding for long-context inference, 2023

    Dao, T., Haziza, D., Massa, F., and Sizov, G. Flash-decoding for long-context inference, 2023. URL https://crfm.stanford.edu/2023/10/12/flashdecoding.html

    work page 2023

  6. [6]

    DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Xu, H., Yang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., Chen, J., Yuan, J., Qiu, J., Song,...

    arXiv 2024

  7. [7]

    DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., ...

    arXiv 2025

  8. [8]

    Lazyllm: Dynamic token pruning for efficient long context llm inference, 2024

    Fu, Q., Cho, M., Merth, T., Mehta, S., Rastegari, M., and Najibi, M. Lazyllm: Dynamic token pruning for efficient long context llm inference, 2024. URL https://arxiv.org/abs/2407.14057

    arXiv 2024

Show all 33 references
  1. [9]

    Gemma 2: Improving Open Language Models at a Practical Size , October 2024

    Gemma Team . Gemma 2: Improving Open Language Models at a Practical Size , October 2024. URL http://arxiv.org/abs/2408.00118. arXiv:2408.00118 [cs]

    2024 arXiv

  2. [10]

    LM - Infinite : Zero - Shot Extreme Length Generalization for Large Language Models , June 2024

    Han, C., Wang, Q., Peng, H., Xiong, W., Chen, Y., Ji, H., and Wang, S. LM - Infinite : Zero - Shot Extreme Length Generalization for Large Language Models , June 2024. URL http://arxiv.org/abs/2308.16137. arXiv:2308.16137 [cs]

    2024 arXiv

  3. [11]

    W., Shao, Y

    Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length llm inference with kv cache quantization, 2024. URL https://arxiv.org/abs/2401.18079

    2024 arXiv

  4. [12]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7B , October 2023. ...

    2023 arXiv

  5. [13]

    H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L

    Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. MInference 1.0: Accelerating Pre -filling for Long - Context LLMs via Dynamic Sparse Attention , October 2024. URL http://arxiv.org/abs/2407.02490. arXiv:240...

    2024 arXiv

  6. [14]

    LLM Maybe LongLM : Self - Extend LLM Context Window Without Tuning , July 2024

    Jin, H., Han, X., Yang, J., Jiang, Z., Liu, Z., Chang, C.-Y., Chen, H., and Hu, X. LLM Maybe LongLM : Self - Extend LLM Context Window Without Tuning , July 2024. URL http://arxiv.org/abs/2401.01325. arXiv:2401.01325 [cs]

    2024 arXiv

  7. [15]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient Memory Management for Large Language Model Serving with PagedAttention , September 2023. URL http://arxiv.org/abs/2309.06180. arXiv:2309.06180 [cs]

    2023 arXiv

  8. [16]

    Lee, H., Kang, M., Lee, Y., and Hwang, S. J. Sparse token transformer with attention back tracking. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=VV0hSE8AxCw

    2023

  9. [17]

    Lee, H., Kim, J., Willette, J., and Hwang, S. J. SEA : Sparse linear attention with estimated attention mask. In The Twelfth International Conference on Learning Representations, 2024 a . URL https://openreview.net/forum?id=JbcwfmYrob

    2024

  10. [18]

    Lee, H., Park, G., Lee, Y., Suh, J., Kim, J., Jeong, W., Kim, B., Lee, H., Jeon, M., and Hwang, S. J. A Training -free Sub -quadratic Cost Transformer Model Serving Framework With Hierarchically Pruned Attention , October 2024 b . URL http://arxiv.org/abs/2406.09827. arXiv:240...

    2024 arXiv

  11. [19]

    EXAONE 3.0 7

    LG AI . EXAONE 3.0 7. 8B Instruction Tuned Language Model , August 2024 a . URL http://arxiv.org/abs/2408.03541. arXiv:2408.03541 [cs]

    2024

  12. [20]

    EXAONE 3.5: Series of Large Language Models for Real -world Use Cases , December 2024 b

    LG AI . EXAONE 3.5: Series of Large Language Models for Real -world Use Cases , December 2024 b . URL http://arxiv.org/abs/2412.04862. arXiv:2412.04862 [cs]

    2024

  13. [21]

    Snapkv: Llm knows what you are looking for before generation

    Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024

    2024 arXiv

  14. [22]

    The Llama 3 Herd of Models , November 2024

    Llama Team , A. The Llama 3 Herd of Models , November 2024. URL http://arxiv.org/abs/2407.21783. arXiv:2407.21783 [cs]

    2024 arXiv

  15. [23]

    Transformers are Multi - State RNNs , June 2024

    Oren, M., Hassid, M., Yarden, N., Adi, Y., and Schwartz, R. Transformers are Multi - State RNNs , June 2024. URL http://arxiv.org/abs/2401.06104. arXiv:2401.06104 [cs]

    2024 arXiv

  16. [24]

    E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C

    Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Mart...

    2024 arXiv

  17. [25]

    RoFormer : Enhanced Transformer with Rotary Position Embedding , November 2023

    Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. RoFormer : Enhanced Transformer with Rotary Position Embedding , November 2023. URL http://arxiv.org/abs/2104.09864. arXiv:2104.09864 [cs]

    2023 arXiv

  18. [26]

    Quest: Query - Aware Sparsity for Efficient Long - Context LLM Inference , August 2024

    Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., and Han, S. Quest: Query - Aware Sparsity for Efficient Long - Context LLM Inference , August 2024. URL http://arxiv.org/abs/2406.10774. arXiv:2406.10774 [cs]

    2024 arXiv

  19. [27]

    Tillet, P., Kung, H.-T., and Cox, D. D. Triton: an intermediate language and compiler for tiled neural network computations. Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 2019. URL https://api.semanticscholar.org/Corpu...

    2019

  20. [28]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention Is All You Need , August 2017. URL http://arxiv.org/abs/1706.03762. arXiv:1706.03762 [cs]

    2017 arXiv

  21. [29]

    Willette, J., Lee, H., Lee, Y., Jeon, M., and Hwang, S. J. Training-free exponential extension of sliding window context with cascading kv cache. arXiv preprint arXiv:2406.17808, 2024

    2024 arXiv

  22. [30]

    InfLLM : Training - Free Long - Context Extrapolation for LLMs with an Efficient Context Memory , May 2024 a

    Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., and Sun, M. InfLLM : Training - Free Long - Context Extrapolation for LLMs with an Efficient Context Memory , May 2024 a . URL http://arxiv.org/abs/2402.04617. arXiv:2402.04617 [cs]

    2024 arXiv

  23. [31]

    Efficient Streaming Language Models with Attention Sinks , April 2024 b

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient Streaming Language Models with Attention Sinks , April 2024 b . URL http://arxiv.org/abs/2309.17453. arXiv:2309.17453 [cs]

    2024 arXiv

  24. [32]

    K., Han, X., Thai, Z

    Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M. K., Han, X., Thai, Z. L., Wang, S., Liu, Z., and Sun, M. \ infty\ Bench : Extending Long Context Evaluation Beyond 100K Tokens , February 2024. URL http://arxiv.org/abs/2402.13718. arXiv:2402.13718 [cs]

    2024 arXiv

  25. [33]

    H\ \_2\ O : Heavy - Hitter Oracle for Efficient Generative Inference of Large Language Models , December 2023

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H\ \_2\ O : Heavy - Hitter Oracle for Efficient Generative Inference of Large Language Models , December 2023. URL http://arxiv.org/abs/2306.14048. arX...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.