Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

RedKnot claims LLM serving can cut prefill compute by up to 79.5% and raise concurrent sessions per GPU by 4.7–7.8× by managing the KV cache head-by-head instead of as a monolithic block, without retraining or fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 12:16 UTC pith:SG6DEQXH

load-bearing objection Head-aware KV reuse is a real idea with a sharp diagnosis; the evaluation is disclaimed, unreproduced, and partly circular — send it out, but expect heavy revision. the 5 major comments →

arxiv 2606.06256 v3 pith:SG6DEQXH submitted 2026-06-04 cs.AI

RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

Yang Liu , Zhaokai Luo , Huayi Jin , Zhiyong Wang , Ruozhou He , Boyu Wang , Guanjie Chen , Tao Xie
show 1 more author
Junhao Hu
This is my paper
classification cs.AI
keywords head-aware KV cacheposition-independent KV reuseglobal/local attention headsSegPagedAttentionsparse FFN executionlong-context LLM servingprefix compressionKV cache lifecycle
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

RedKnot tries to establish that the key-value cache of a large language model should be managed at the granularity of attention heads, not tokens or whole layers. It claims that each (layer, head) has a stable identity—mostly 'local' heads that only need a sink plus a recent window, and a few 'global' heads that need full context—so most cached KV can be reused verbatim when a chunk appears after a new prefix. Because the FFN, not attention, dominates short-context prefill, the paper adds token-level sparse FFN execution. On top of these, a head-segmented paged KV layout makes the sparsity physical, avoiding the mask penalty that otherwise cancels algorithmic savings. If correct, this unifies position-independent reuse, prefix compression, hot/cold separation, and distributed placement in one substrate with accuracy equal to or better than dense serving.

Core claim

The paper's central claim is that the KV cache's utility is structured by head: offline profiling can classify each (layer, head) pair as global or local, and this classification transfers across prefixes, datasets, and context lengths. On reuse, only global heads are recomputed under the new prefix; local heads are reused with a lightweight repair confined to sink tokens and a sliding window, and low-attention token states skip the FFN update. The claimed result is a quality–efficiency frontier that dominates dense prefill: up to 3.54× TTFT speedup, 4.7–7.8× more concurrent sessions per GPU, 67–79.5% fewer prefill FLOPs, with end-to-end accuracy matching or exceeding the dense baseline, all

What carries the argument

Head-class sparsification: an offline map assigning each KV head to global (12–15% of heads, recomputed on reuse) or local (85–88%, reused within a visible set of sink tokens plus a sliding window). SegPagedAttention: storage indexed by (layer, head, segment) with virtual-page indirection, so each head keeps only the pages it needs and runs through a fused variable-length attention kernel with no attention mask, converting byte savings into bandwidth savings. Sparse FFN: after head-aware recovery, only top-attention tokens execute the FFN; unselected tokens follow the residual identity, attacking the FFN-dominated short-context regime.

Load-bearing premise

The load-bearing premise is that each head's classification as global or local, assigned once during offline calibration, stays correct in deployment; if a 'local' head becomes prefix-sensitive for some query, its stale reused cache is never corrected because the described edge-mass restore is not specified or evaluated.

What would settle it

Run the system on a distribution-shifted workload, such as agent tool-call traces where the same chunk follows many different prefixes, and for each local head measure the attention mass falling outside its visible window on a sample of queries, then compare final accuracy against dense recompute. If that out-of-window mass exceeds the calibration threshold while accuracy drops below roughly 95% of dense, the central claim fails; alternatively, an ablation that removes the global-head recomputation path should show a sharp accuracy cliff if the classification is truly load-bearing.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Position-independent KV reuse shifts from token-level to head-level recovery, escaping the union-of-token-sets problem that makes token-level schemes recompute nearly everything.
  • The same head classification supplies a unified mechanism for prefix compression (local heads evict the middle), hot/cold separation, and prefill–decode transfer reduction of 4.3–6.3× in KV bytes.
  • Because sparse FFN is structurally independent of context length, the approach accelerates short-context agent workloads that attention-side optimization cannot reach.
  • With SegPagedAttention, local heads occupy constant-length windows, enabling 4.7–7.8× more concurrent sessions per GPU and shifting the memory bottleneck from full dense KV caches to compact per-head caches.
  • Longer contexts make attention sparser, so sparse execution can act as denoising; the paper reports RedKnot crossing dense accuracy at 64K–128K contexts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If head classes are stable, a serving engine should expose per-head KV as a first-class scheduling object, making cache admission and eviction head-aware rather than merely chunk-aware; the paper gestures at this but leaves it as future work.
  • A testable extension is an online head-class monitor: randomize chunk order per query and measure whether local-head attention mass outside the visible window stays below calibration threshold; this would validate the transferability assumption under distribution shift.
  • The sparse-FFN zeroing claim is the least mechanistically justified approximation; a direct probe comparing hidden states with and without FFN skipping on individual tokens would locate where fidelity loss actually concentrates.
  • The long-context quality crossover suggests a broader noise-aware scheduling principle: intentionally skipping low-value tokens can improve quality even where no KV reuse occurs, a generalization the paper describes qualitatively but does not formalize.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. RedKnot proposes a head-aware KV cache management system that decomposes the KV cache along attention heads, classifies heads offline as global or local, reuses local-head KV with RoPE realignment and bounded-window repair, applies token-level sparse FFN, and introduces SegPagedAttention, a per-(layer,head) paged KV layout with a fused varlen attention kernel. The paper claims 1.6–3.54× TTFT speedup (up to 5.16× in §5.2), 4.7–7.8× concurrency gains, 67–79.5% prefill FLOP reduction, and accuracy matching or exceeding the dense baseline across three model families, six QA datasets, and context lengths 8K–128K, all without model retraining.

Significance. If the accuracy-preservation claim survives replication, the head-level decomposition is a genuinely useful systems abstraction: it unifies position-independent reuse, prefix compression, hot/cold separation, and distributed KV placement under one substrate. The paper has real strengths: the RoPE realignment is the standard identity, the LSE softmax fusion claim in §4.4 is stated correctly, and the SegPagedAttention microbenchmarks in §5.3 isolate the kernel and report numerical equivalence (cos>0.99998). The public code release is also a plus. However, the load-bearing empirical claim — dense-matching accuracy at 67–79% FLOP reduction — currently rests on an internally inconsistent algorithm listing, an unvalidated stability assumption, and configuration selection that partially predetermines the reported success metric. These issues are substantive but addressable, so the paper is viable in principle with major revision.

major comments (5)
  1. [§4.2, Algorithm 1, lines 20–23] As printed, Algorithm 1 is internally inconsistent: line 21 computes Z[S] = FFN(Y[S]), and line 22 immediately sets Z[S] = 0, so no FFN update survives. The surrounding text says unselected tokens should follow the residual path, meaning line 22 should zero the complement of S. As printed, the deep-layer sparse FFN is completely skipped, and all Sparse-FFN savings and accuracy results attached to Fig. 3 and §5.2 are not reproducible from the algorithm. This must be corrected and the implementation must be described precisely.
  2. [§1 vs. §4.2, Algorithm 1, §5] The abstract and introduction promise an 'adaptive runtime restore' that promotes local heads to full attention when an edge-mass signal detects misclassification. This safeguard is never specified in Algorithms 1–2, never implemented in the evaluation, and never tested. The accuracy claims rely on the offline head-class map M transferring to every new prefix, length, and dataset; §3.2/Fig. 4 only report average global/local proportions, not prefix-to-prefix stability. If a profiled local head becomes prefix-sensitive under distribution shift, its stale KV is reused without correction and the error propagates through deeper layers. This is load-bearing: the claimed 'strictly-better-than-dense' frontier depends on stability that is asserted, not demonstrated.
  3. [§5.1 and §5.4] Part of the accuracy-preservation claim is circular with configuration selection. §5.1 states the Qwen3.5-397B configuration is the 'sweet-spot... selected on the 397B sweep: it is lossless on the 32K TriviaQA calibration setting.' §5.4 states trim<32 was chosen 'as the accuracy-safe operating point' and then reports that first-decode-step cosine stays above the 0.99 pass threshold — the same threshold used for selection. The headline result that accuracy 'matches or exceeds the dense baseline' is therefore partly a consequence of picking configurations that satisfy the evaluation criterion. Please report the full configuration sweep, select thresholds a priori or on a held-out calibration set, and show results for configurations that are not tuned to the reported metric.
  4. [Abstract, §5.2, Fig. 8] The paper reports inconsistent maximum speedups: the abstract and conclusion state 'up to 3.54× TTFT speedup,' but §5.2 and Fig. 8(i)–(l) report RedKnot reaching 3.51× at 16K and 5.16× at 128K on DeepSeek-V4-Flash. If the 5.16× number is valid, the abstract understates the result; if it is not comparable (e.g., a different baseline or measurement protocol), the discrepancy must be explained. Either way, the headline claim should match the reported measurements.
  5. [§5.7 and Fig. 13] The 'sparse denoising' claim — that RedKnot overtakes dense accuracy at long context — is presented without statistical significance, error bars, or ablations separating the effect of head-class KV reuse from the effect of sparse FFN token selection. Given that the sparse FFN path zeroes the FFN update for many tokens, the observed accuracy crossover could be sensitive to the specific thresholds (mass_thresh=0.7, etc.). Please provide confidence intervals or repeated-run variance, and an ablation that isolates whether the gain comes from KV reuse, sparse FFN, or their interaction.
minor comments (6)
  1. [§4.4] Typo: 'Disgussed' should be 'Discussed'.
  2. [Fig. 5 caption / §5.2 text] 'Retieval' and 'achives' are typos; please proofread figure captions and captions in Fig. 8.
  3. [§1, footnote] The disclaimer 'The experimental results in the paper are for reference only, the test results from the open-source community code shall prevail' is unusual for a refereed paper and undermines reproducibility. It should be removed or replaced with a clear statement of how to reproduce the reported numbers from the released code and exact environment.
  4. [References] Several references are to models, datasets, and systems with future or very recent dates (Qwen3.5, DeepSeek-V4, ProphetKV, CacheSlide). Please verify all citations are publicly available and add arXiv identifiers or DOIs where possible.
  5. [§5.6] The PD-disaggregation experiments 'do not yet enable SegPagedAttention'; this is a significant caveat and should be stated in the main text near Fig. 12, not only in the setup, so readers do not attribute the transfer savings to the new kernel.
  6. [Algorithm 2] The notation H*ℓ, T*seg, and the packed varlen metadata are used without formal definition; a small example of segment-to-page mapping would improve clarity.

Circularity Check

2 steps flagged

Two evaluation configs are selected on the metric later reported as success; the central head-aware accuracy claim is independently benchmarked.

specific steps
  1. fitted input called prediction [Section 5.4 (Prefix Compression), Qwen3-32B trim<32 configuration and Fig. 10(a)]
    "Because Qwen3-32B is a dense model with no native sliding-window mask, aggressive all-layer trimming collapses; the accuracy-safe operating point is trim<32, i.e. trimming local heads only in the first 32 of 64 layers, with window W=4096 and sink=128. All results use this single configuration. ... the first-decode-step logit cosine against the full-KV baseline stays above the 0.99 pass threshold at every prefix length (0.9911 at 8K, 0.9988 at 16K, 0.9987 at 32K)"

    The 'accuracy-safe operating point' was selected precisely because it cleared the 0.99 cosine bar; Fig. 10(a) then reports the same bar as the success criterion ('stays above the 0.99 pass threshold'). Thus the primary pass/fail result of the prefix-compression experiment is the selection rule restated, not an out-of-sample prediction. The cross-dataset cosine/top-match results in Fig. 10(c) are not so forced, so this is a partial, section-local circularity.

  2. fitted input called prediction [Section 5.1 (Qwen3.5-397B-A17B configuration) and Fig. 8(e)-(g), Section 5.2]
    "This is the sweet-spot configuration selected on the 397B sweep: it is lossless on the 32K TriviaQA calibration setting while saving about 52% total compute and giving about 2.07×TTFT speedup."

    The 397B configuration was chosen by requiring losslessness at the 32K TriviaQA calibration setting, and the same 32K TriviaQA accuracy is then reported as evidence that RedKnot 'preserves accuracy close to full recompute' (Fig. 8). For that cell, matching dense accuracy is effectively guaranteed by the selection criterion; the 16K/64K and other-dataset panels retain independent content.

full rationale

The core RedKnot claim — head-level global/local KV recovery preserves dense-comparable QA accuracy while cutting FLOPs — does not reduce to its inputs. End-to-end accuracy is benchmarked against dense recompute and token-level PIC baselines across multiple datasets and lengths, with no fitted parameter renamed as the reported F1/EM. The two flagged steps are local to calibration-selected cells: the trim<32 prefix-compression result reports the same 0.99 cosine threshold used to pick the configuration, and the 397B 'sweet-spot' config was chosen to be lossless at 32K TriviaQA before that cell is reported as a quality result. These are genuine partial reductions, but they do not make the central multi-dataset comparison circular. Self-citations (EPIC [26], CacheSlide [39]) are contextual background and are not load-bearing. Separately, the intro's promised 'adaptive runtime restore' never appears in Algorithm 1 or Section 5; that is an unsupported robustness claim, relevant to correctness and reproducibility, but not a circularity. Algorithm 1's printed FFN lines (21-22) are internally inconsistent (line 22 zeros the just-written selected-token updates), but this is a reproducibility defect rather than a definitional circularity. Because the reductions are confined to specific calibration-selected evaluations while the central claim has independent content, the score is 4 rather than higher.

Axiom & Free-Parameter Ledger

6 free parameters · 7 axioms · 0 invented entities

Everything the central claims rest on is either (a) standard identities (RoPE alignment, LSE fusion — solid), (b) prior head-sparsity findings (DuoAttention et al. — external support), or (c) per-model calibration-chosen numbers (ρg, W, sink, thresholds, trim<32 boundary — fitted, not predicted). The accuracy-preservation claim is the product of a configuration search whose objective was accuracy preservation, so the ledger is dominated by fitted operating points. No new physical or model-level entities are introduced; the 'reusable state object' and head-class map are software abstractions, not postulated entities.

free parameters (6)
  • effective global-head budget ρg = 9.4% / 10.0% / 9.4% / 4.3% / 4.6% for Mistral, Llama-3.3-70B, Qwen3-32B, Qwen3.5-397B, DeepSeek-V4-Flash
    Per-model fraction of (layer,head) entries classified global/prefix-sensitive; profiled offline and directly sets how much KV must be recomputed on reuse — the main knob behind the headline savings (§5.1, Fig. 4).
  • local window size W = 256 (Mistral/Llama); 4096 (Qwen3-32B); 2048 (Qwen3.5-397B); 128 (DeepSeek-V4-Flash, no sink)
    Per-model attention window for local heads; determines KV retained, repair scope, and FLOP savings. Qwen3's 4096 window over the last 16 layers is 25% of a 16K context, limiting 'local' savings there (§5.1, §4.2 Eq. W(i)).
  • sink size S_sink = 4 tokens (GQA models); 128 (Qwen3 prefix compression); 0 (DeepSeek-V4)
    Hand-set attention-sink budget in the local visible set (§5.1, §5.4).
  • Sparse-FFN token-selection policy (mass_thresh, dense-layer boundary, recent-keep) = 0.2→0.05 (Llama/Qwen3), 0.7 (Qwen3.5 MoE), 0.6 (DeepSeek-V4); boundaries 20/5/48/9; recent-keep 512/128/256
    Per-model token-selection policy for skipping FFN updates, tuned on calibration prompts to preserve QA accuracy (§5.1); FFN-skip is the largest accuracy-risk approximation in the stack.
  • prefix-compression operating point = 'trim<32' with W=4096, sink=128, 0.99 cosine pass threshold
    Post-hoc selected configuration in §5.4: 'aggressive all-layer trimming collapses,' so only the first 32 of 64 layers are trimmed; the 0.99 cosine is both the selection criterion and the reported success metric.
  • edge-mass restore threshold = unspecified
    Runtime promotion of local heads to full attention when an edge-mass signal detects misclassification (Fig. 1, §4.2); no threshold given and never evaluated.
axioms (7)
  • standard math RoPE rotation alignment: K(p_on) = R(p_on)R(p_off)^{-1} K(p_off)
    Standard rotary-position identity; correct as stated (§4.2).
  • standard math LSE softmax fusion over split key sets is exact for a fixed query
    Correct: merging offline/online key sets by log-sum-exp reproduces softmax attention over the union (§4.4); the most airtight piece of the paper.
  • domain assumption (layer,head) global/local class is request-invariant and portable across datasets and lengths
    The entire recovery policy uses a static class map profiled on calibration prompts (§3.2, Algorithm 1 lines 4–5); external support from DuoAttention/StreamingLLM, but no online verification in the evaluation.
  • domain assumption Local-head attention mass concentrates inside W(i)=S_sink∪[i−w,i]
    Out-of-window cached KV is reused unmodified; the claim that this preserves fidelity is the core accuracy assumption (§4.2 Eq. W(i), §5.2).
  • domain assumption Zeroing FFN updates for low-attention-mass tokens preserves (and at long context improves) output quality
    Algorithm 1 lines 21–23 set Z=0 on non-selected tokens; §5.7 claims sparse beats dense at 64K–128K. Least mechanistically justified approximation; no ablation isolates its contribution.
  • domain assumption Materialized attn_mask forces a 4.9–7.6× slower SDPA path; mask-free varlen FlashAttention is the fast path
    System premise for SegPagedAttention's benefit (§3.4, §5.3); true on current PyTorch/CUDA stacks but not a mathematical law.
  • domain assumption MuSiQue stream reuse statistics generalize to other workloads
    Lifecycle-management advice (§5.5) derives from one request-stream replay.

pith-pipeline@v1.3.0-alltime-deepseek · 34019 in / 25404 out tokens · 236927 ms · 2026-08-02T12:16:17.652743+00:00 · methodology

0 comments
read the original abstract

As the input length of large language model (LLM) serving continues to grow, the KV cache has become a dominant bottleneck in AI infrastructure. It limits GPU memory capacity, serving concurrency, cache reuse, and distributed scalability. Multiple important problems, including position-independent KV cache, prefix KV cache compression, hot/cold KV cache separation, and distributed KV cache management, all depend on how the KV cache is represented and managed. However, existing serving systems largely rely on a monolithic KV cache abstraction, where the KV cache is treated as a homogeneous sequence of token-level memory blocks and managed with similar policies across attention heads and serving scenarios. We observe that KV cache utility is highly structured across KV heads: different heads exhibit different functional roles, attention distances, and runtime importance. Therefore, a full KV cache is not always necessary for every head, token range, or serving scenario. We present RedKnot, a head-aware KV cache management system for LLM serving. RedKnot breaks the conventional monolithic KV cache abstraction by decomposing the KV cache along KV heads, whose importance and effective attention ranges vary significantly across serving scenarios. This head-level decomposition turns the KV cache from a monolithic tensor abstraction into a structured memory object, enabling RedKnot to uniformly support position-independent KV reuse, prefix KV compression, hot/cold KV separation, and distributed KV placement while preserving output fidelity and improving resource efficiency, without requiring model retraining or fine-tuning. RedKnot establishes a new foundation for AI infrastructure by transforming the KV cache from a monolithic, passive runtime artifact into a dynamic, model-aware runtime substrate for scalable LLM serving.

Figures

Figures reproduced from arXiv: 2606.06256 by Boyu Wang, Guanjie Chen, HuaYi Jin, Junhao Hu, RuoZhou He, Tao Xie, Yang Liu, ZhaoKai Luo, Zhiyong Wang.

Figure 1
Figure 1. Figure 1: RedKnot decouples the KV cache along the head dimen￾sion, classifies heads into global and local classes, and co-optimizes sparse attention, sparse FFN execution with selected tokens and Seg￾PagedAttention. The combined design yields 1.6–3.5× lower TTFT, 4.7–7.8× higher concurrency, and 67–79% fewer FLOPs compared with dense attention. generation (RAG) [11,21] routinely concatenates tens of thou￾sands of r… view at source ↗
Figure 2
Figure 2. Figure 2: For short contexts, prefill TTFT is dominated by FFN computation rather than KV-cache construction. heads of a selected token together, the effective recomputa￾tion set becomes the union of head-specific important tokens. This union can cover a large portion of the chunk, forcing the system to recompute many tokens to recover accuracy. This creates a fundamental limitation for token-level PIC recovery. Eve… view at source ↗
Figure 3
Figure 3. Figure 3: For short contexts, prefill TTFT is dominated by FFN computation rather than KV-cache construction. unfavorable trade-off: selecting fewer tokens may leave some head-specific errors uncorrected and hurt output quality, while selecting more tokens improves fidelity but quickly reduces the TTFT benefit of KV reuse. Beyond the attention-level bottleneck, existing PIC systems also overlook a complemen￾tary cha… view at source ↗
Figure 4
Figure 4. Figure 4: Overview of RedKnot. 4.1 Overview of RedKnot RedKnot consists of two core components: (i) an Elastic Sparsity module, and (ii) a module that stores data at the gran￾ularity of KV-cache heads. We next describe the end-to-end workflow of RedKnot, highlighting how these modules in￾teract during inference. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: WorkFlow of Elastic Sparsity. recovery. RoPE-based positional alignment. When a reusable chunk is cached offline and later placed after a different prefix, its token positions change. Since modern LLMs commonly use RoPE, the cached keys contain position-dependent rotations. Before applying recovery, Elastic Sparsity first aligns the cached keys to their online positions using the rotational structure of Ro… view at source ↗
Figure 5
Figure 5. Figure 5: Overview of RedKnot. HBM bandwidth utilization and throughput. 4.1 Overview of RedKnot RedKnot consists of two core components: (i) an Elastic Sparsity module, and (ii) a module that stores data at the gran￾ularity of KV-cache heads. We next describe the end-to-end workflow of RedKnot, highlighting how these modules in￾teract during inference. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Overview of SegPagedAttention. Lines 18–19 merge the recovered heads and form the inter￾mediate hidden states. Lines 20–23 then apply partial sparse FFN recovery. Elastic Sparsity selects important tokens ac￾cording to the recovered attention signal, executes the dense FFN only on these selected tokens, and sets the FFN update of unselected tokens to zero so that they follow the residual iden￾tity path. Fi… view at source ↗
Figure 6
Figure 6. Figure 6: WorkFlow of Elastic Sparsity. positions using the rotational structure of RoPE. For a cached key originally encoded at offline position poff and reused at online position pon, Elastic Sparsity applies the relative rota￾tion K(pon) = R(pon)R(poff) −1K(poff), where R(·) denotes the RoPE rotation matrix. This step re￾moves the deterministic position mismatch caused by moving the chunk to a new location. The r… view at source ↗
Figure 7
Figure 7. Figure 7: End-to-end comparison across latency, answer quality, and KV matching metrics. RedKnot consistently improves TTFT while preserving higher accuracy and stronger Top-K KV matching than position-independent cache baselines. scaling is better than that of CacheBlend and ProphetKV. Why RedKnot achieves a better quality–latency trade￾off. The key difference is that RedKnot aligns its recovery granularity with th… view at source ↗
Figure 7
Figure 7. Figure 7: Overview of SegPagedAttention. Sparsity uses the recovered attention signal to estimate token importance. Tokens with high importance execute the dense FFN, while other tokens follow the residual identity path. In this way, Elastic Sparsity spends FFN computation only where correction is likely to affect the final hidden states. The two sparsity dimensions are complementary. Head￾aware attention recovery r… view at source ↗
Figure 8
Figure 8. Figure 8: Prefill compute comparison across six workload settings. RedKnot substantially reduces prefill FLOPs compared with dense recomputation, CacheBlend, and ProphetKV. Panel titles encode model.dataset.context length: M = Mistral-7B, Q = Qwen3-32B, and L70 = Llama-3.3-70B; TQA = TriviaQA, MFQA = MultiFieldQA, and HQA = HotpotQA; 16K/24K/32K/64K denote the total prompt context length in tokens. Each panel uses a… view at source ↗
Figure 8
Figure 8. Figure 8: End-to-end accuracy and TTFT comparison across three model families. RedKnot achives the per-panel TTFT speedup ranging from 3.51× to 5.16×. Overall, RedKnot preserves accuracy close to full recompute (typically ≥ 95% of the dense F1) while delivering 1.4–5.2× TTFT speedup, and consistently dominates the token-level PIC baselines on the quality–latency trade-off. tokens. Datasets. We draw RAG prompts from … view at source ↗
Figure 9
Figure 9. Figure 9: Single-layer decode kernel latency under two SDPA back￾ends on Qwen3-32B shapes (Hq = 32, D = 128, bf16, qlen = 1). FlashAttention requires a null attn_mask. explain why RedKnot is not merely faster than dense prefill, but also substantially cheaper in total prefill compute than token-level PIC baselines. 5.4 SegPagedAttention Micro-Benchmarks We evaluate SegPagedAttention with kernel-isolated micro￾benchm… view at source ↗
Figure 9
Figure 9. Figure 9: Throughput and attention-kernel efficiency of RedKnot. Top row (a)–(d): serving throughput (QPS/GPU, log scale) vs. context length on (a) Qwen3-32B (TP=2), (b) Llama-3.3-70B (TP=4), (c) Qwen3.5-397B (TP=8), and (d) DeepSeek-V4-Flash (PP=8), comparing RedKnot with dense recompute, CacheBlend (r = 15%), and ProphetKV (r = 20%). Bottom row (e)–(h): kernel-isolated latency of SegPagedAttention vs. masked/dense… view at source ↗
Figure 12
Figure 12. Figure 12: 64-layer prefill throughput (tok/s) at batch 1. SDPA+mask retains only 25% of 8 K throughput at 32 K, while SegPagedAttention retains 46% and maintains much higher absolute throughput. 32K, and 128K, respectively. This gap is not an algorithmic property of head sparsity; it is a dispatch artifact. PyTorch SDPA can use the FlashAttention backend only when the mask is null. Once RedKnotmaterializes local/gl… view at source ↗
Figure 10
Figure 10. Figure 10: Prefix multi-head KV compression on Qwen3-32B under PD disaggregation. (a) first-decode-step logit cosine vs. the full-KV baseline (left axis) and KV-transfer saving (right axis) vs. prefix length; the dashed line marks the 0.99 pass threshold. (b) aggregate decode throughput (QPS/GPU) under a fixed KV-memory budget, full-KV baseline vs. trim<32, with the per-point speedup annotated. (c) per-dataset logit… view at source ↗
Figure 11
Figure 11. Figure 11: 64-layer decode and prefill latency with fused varlen SegPagedAttention. All paths are numerically equivalent (cos > 0.99998). Labels above the fused bars indicate the speedup over SDPA+mask. and GQA-4. We sweep context lengths of 8K, 32K, and 128K tokens. The sparsity pattern follows the head-class layout used by RedKnot: half of the KV heads are retrieval/global heads that read the full context, and hal… view at source ↗
Figure 11
Figure 11. Figure 11: Chunk-level KV reuse on the MuSiQue stream (2417 questions, 48,315 chunk accesses, 17,629 unique passages). (a) reuse-count distribution per chunk (log–log). (b) fraction of each chunk’s reuse that comes from non-prefix positions, with mean 0.95. (c) reuse count vs. residency (the request span over which a chunk stays live), colored by log value density. (d) recompute saved vs. the KV memory needed to cac… view at source ↗
Figure 13
Figure 13. Figure 13: Additional system metrics beyond quality, TTFT, and compute. (a) RedKnot reduces KV transfer volume by 4.3–6.3× and transfer time by up to 4.1× under prefill–decode disaggregation. (b) Burst-mode throughput improves by 15–43% on the current dense+mask backend. (c) When SegPagedAttention materializes KV savings as physical memory savings, concurrent session capacity per GPU increases by 4.7–7.8×, which pro… view at source ↗
Figure 12
Figure 12. Figure 12: System-level effects of head-class KV sparsity. (a) KV-cache transfer saving over dense PD disaggregation, separated into transferred bytes and wall-clock transfer time, across Llama-3.3-70B and Qwen3-32B at 8K–24K. (b) burst-mode throughput (req/s) of dense vs. RedKnot for bursts of N concurrent requests, annotated with the relative gain. (c) concurrent sessions per GPU under dense vLLM-style KV storage … view at source ↗
Figure 13
Figure 13. Figure 13: Sparse denoising becomes more useful as context grows. (a) On DeepSeek-V4-Flash, the fraction of tokens needed to cover 99% attention mass drops with context length across HotpotQA, 2WikiMQA, MultiFieldQA, and GovReport. (b) On Qwen3.5-397B and DeepSeek-V4-Flash, dense accuracy degrades under long-context noise, while RedKnot stays stable and overtakes dense at longer contexts. become the sparsest because… view at source ↗

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Akashic: A Low-Overhead LLM Inference Service with MemAttention

    cs.AI 2026-07 conditional novelty 5.5

    Akashic’s MemAttention plus locality-aware placement improves agent task accuracy by up to 10.2 points and throughput by up to 1.21× over prior memory systems across four long-horizon workloads.

Reference graph

Works this paper leans on

73 extracted references · 10 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Gulavani, Alexey Tu- manov, and Ramachandran Ramjee

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tu- manov, and Ramachandran Ramjee. Taming throughput- latency tradeoff in LLM inference with Sarathi-Serve. In18th USENIX Symposium on Operating Systems De- sign and Implementation (OSDI), pages 117–134, 2024

    2024

  2. [2]

    GQA: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901. Association for Computational Linguistics, 2023

    2023

  3. [3]

    Claude Code

    Anthropic. Claude Code. https://www.anthropic.com/ claude-code, 2025. Accessed: 2026-06-04. 21

    2025

  4. [4]

    LongBench: A bilingual, multitask bench- mark for long context understanding.arXiv preprint arXiv:2308.14508, 2024

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask bench- mark for long context understanding.arXiv preprint arXiv:2308.14508, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    LongBench v2: Towards deeper understanding and reasoning on realis- tic long-context multitasks

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xi- aozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench v2: Towards deeper understanding and reasoning on realis- tic long-context multitasks. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2025

  6. [6]

    Extending context window of large language models via positional interpolation, 2023

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023

    2023

  7. [7]

    NVIDIA Hopper H100 GPU: Scaling performance.IEEE Micro, 43(3):9–17, 2023

    Jack Choquette. NVIDIA Hopper H100 GPU: Scaling performance.IEEE Micro, 43(3):9–17, 2023

    2023

  8. [8]

    Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y . Wu, Zhenda Xie, Y . K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wen- feng Liang. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models, 2024

    2024

  9. [9]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  10. [10]

    Smith, and Matt Gardner

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 4599–4610, 2021

    2021

  11. [11]

    DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model, 2024

    DeepSeek-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model, 2024

    2024

  12. [12]

    DeepSeek-V3 technical report, 2024

    DeepSeek-AI. DeepSeek-V3 technical report, 2024

    2024

  13. [13]

    FlashMLA: Efficient multi-head latent attention kernels

    DeepSeek-AI. FlashMLA: Efficient multi-head latent attention kernels. https://github.com/deepseek-ai/Flas hMLA, 2025. Accessed: 2026-06-14

    2025

  14. [14]

    DeepSeek-V4-Flash model card

    DeepSeek-AI. DeepSeek-V4-Flash model card. https: //huggingface.co/deepseek-ai/DeepSeek-V4-Flash,

  15. [15]

    DeepSeek-V4-Pro model card

    DeepSeek-AI. DeepSeek-V4-Pro model card. https: //huggingface.co/deepseek-ai/DeepSeek-V4-Pro, 2026. Accessed: 2026-06-14

    2026

  16. [16]

    Huerta, and Hao Peng

    Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, Aram Galstyan, Az- ton Wells, Roy Schwartz, Eliu A. Huerta, and Hao Peng. Context length alone hurts LLM performance despite perfect retrieval. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 23281–23298, Suzhou, China, 2025. Association for C...

    2025

  17. [18]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

    2022

  18. [19]

    MoA: Mixture of sparse attention for automatic large language model compres- sion.arXiv preprint arXiv:2406.14909, 2024

    Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zix- iao Huang, Shiyao Li, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. MoA: Mixture of sparse attention for automatic large language model compres- sion.arXiv preprint arXiv:2406.14909, 2024

    arXiv 2024

  19. [20]

    Not all heads matter: A head- level KV cache compression method with integrated retrieval and reasoning

    Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head- level KV cache compression method with integrated retrieval and reasoning. InInternational Conference on Learning Representations, 2025

    2025

  20. [21]

    Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2024

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2024

    Pith/arXiv arXiv 2024

  21. [22]

    Prompt Cache: Modular attention reuse for low-latency inference

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt Cache: Modular attention reuse for low-latency inference. In Proceedings of Machine Learning and Systems (MLSys), 2024

    2024

  22. [23]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  23. [24]

    Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. InProceedings of the 28th International Conference on Computational Linguistics (COLING), pages 6609– 6625, 2020. 22

    2020

  24. [25]

    RULER: What’s the real context size of your long-context language models?, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shan- tanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models?, 2024. COLM 2024

    2024

  25. [26]

    EPIC: Efficient position-independent caching for serving large language models

    Junhao Hu, Wenrui Huang, Weidong Wang, Haoyi Wang, Tiancheng Hu, Zhang Qin, Hao Feng, Xusheng Chen, Yizhou Shan, and Tao Xie. EPIC: Efficient position-independent caching for serving large language models. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofPro- ceedings of Machine Learning Research, pages 24391– 24402. PMLR, 2025

    2025

  26. [27]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b.arXiv preprint arXi...

    Pith/arXiv arXiv 2023

  27. [28]

    Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse at- tention. InAdvances in Neural Information Processing Systems, 2024

    2024

  28. [29]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly su- pervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the As- sociation for Computational Linguistics (ACL), pages 1601–1611, 2017

    2017

  29. [30]

    Gonza- lez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonza- lez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with Page- dAttention. InProceedings of the 29th Symposium on Operating Systems Principles (SOSP), pages 611–626, 2023

    2023

  30. [31]

    CATS: Contextually- aware thresholding for sparsity in large language models

    Donghyun Lee, Je-Yong Lee, Genghan Zhang, Mo Ti- wari, and Azalia Mirhoseini. CATS: Contextually- aware thresholding for sparsity in large language models. InConference on Language Modeling, 2024

    2024

  31. [32]

    GShard: Scaling gi- ant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, De- hao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling gi- ant models with conditional computation and automatic sharding. InInternational Conference on Learning Representations, 2021

    2021

  32. [33]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Se- bastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 9459–9474, 2020

    2020

  33. [34]

    NeedleBench: Evaluating LLM retrieval and reasoning across varying information densities, 2024

    Mo Li, Songyang Zhang, Taolin Zhang, Haodong Duan, Yunxin Liu, and Kai Chen. NeedleBench: Evaluating LLM retrieval and reasoning across varying information densities, 2024

    2024

  34. [35]

    CompressKV: Seman- tic retrieval heads know what tokens are not important before generation.arXiv preprint arXiv:2508.02401, 2025

    Xiaolin Lin, Jingcun Wang, Olga Kondrateva, Yiyu Shi, Bing Li, and Grace Li Zhang. CompressKV: Seman- tic retrieval heads know what tokens are not important before generation.arXiv preprint arXiv:2508.02401, 2025

    Pith/arXiv arXiv 2025

  35. [36]

    Rethinking RoPE: A mathematical blueprint for n-dimensional positional encoding, 2025

    Haiping Liu and Hongpeng Zhou. Rethinking RoPE: A mathematical blueprint for n-dimensional positional encoding, 2025

    2025

  36. [37]

    TEAL: Training- free activation sparsity in large language models

    James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. TEAL: Training- free activation sparsity in large language models. In International Conference on Learning Representations, 2025

    2025

  37. [38]

    Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    2024

  38. [39]

    CacheSlide: Unlocking cross position-aware KV cache reuse for accelerating LLM serving

    Yang Liu, Yunfei Gu, Liqiang Zhang, Chentao Wu, Guangtao Xue, Jie Li, Minyi Guo, Junhao Hu, and Jie Meng. CacheSlide: Unlocking cross position-aware KV cache reuse for accelerating LLM serving. InPro- ceedings of the 24th USENIX Conference on File and Storage Technologies, FAST ’26. USENIX Association, 2026

    2026

  39. [40]

    Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  40. [41]

    Deja Vu: Contextual sparsity for efficient LLMs at inference 23 time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. Deja Vu: Contextual sparsity for efficient LLMs at inference 23 time. InProceedings of the 40th International Confer- ence on Machine Learning (ICML), pages 22137–22176, 2023

    2023

  41. [42]

    Rossi, Seunghyun Yoon, and Hinrich Sch"utze

    Ali Modarressi, Hanieh Deilamsalehy, Franck Dernon- court, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Sch"utze. NoLiMa: Long-context evaluation beyond literal matching, 2025. ICML 2025

    2025

  42. [43]

    NVIDIA H100 Tensor Core GPU architecture

    NVIDIA. NVIDIA H100 Tensor Core GPU architecture. Whitepaper, NVIDIA Corporation, 2023

    2023

  43. [44]

    Introducing Codex: A cloud-based software engineering agent

    OpenAI. Introducing Codex: A cloud-based software engineering agent. https://openai.com/index/introduci ng-codex/, 2025. Accessed: 2025-05-20

    2025

  44. [45]

    OpenClaw, 2025

    OpenClaw Contributors. OpenClaw, 2025

    2025

  45. [46]

    Efficiently scal- ing transformer inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scal- ing transformer inference. InProceedings of Machine Learning and Systems (MLSys), 2023

    2023

  46. [47]

    Qwen3.5-35B-A3B model card

    Qwen Team. Qwen3.5-35B-A3B model card. https: //huggingface.co/Qwen/Qwen3.5-35B-A3B, 2026. Accessed: 2026-06-14

    2026

  47. [48]

    Qwen3.5-397B-A17B model card

    Qwen Team. Qwen3.5-397B-A17B model card. https: //huggingface.co/Qwen/Qwen3.5-397B-A17B, 2026. Accessed: 2026-06-14

    2026

  48. [49]

    Qwen3.5 model collection

    Qwen Team. Qwen3.5 model collection. https://huggin gface.co/collections/Qwen/qwen35, 2026. Accessed: 2026-06-14

    2026

  49. [50]

    Qwen3.5: Towards native multimodal agents

    Qwen Team. Qwen3.5: Towards native multimodal agents. https://qwen.ai/blog?id=qwen3.5, 2026. Accessed: 2026-06-14

    2026

  50. [51]

    HiCache system design and optimiza- tion

    SGLang Team. HiCache system design and optimiza- tion. https://docs.sglang.ai/advanced_features/hicache _design.html, 2025. Accessed: 2026-05-31

    2025

  51. [53]

    FlashAttention- 3: Fast and accurate attention with asynchrony and low- precision.arXiv preprint arXiv:2407.08608, 2024

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. FlashAttention- 3: Fast and accurate attention with asynchrony and low- precision.arXiv preprint arXiv:2407.08608, 2024

    Pith/arXiv arXiv 2024

  52. [54]

    Fast transformer decoding: One write- head is all you need.arXiv preprint arXiv:1911.02150, 2019

    Noam Shazeer. Fast transformer decoding: One write- head is all you need.arXiv preprint arXiv:1911.02150, 2019

    Pith/arXiv arXiv 1911

  53. [55]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations, 2017

    2017

  54. [56]

    ProSparse: Intro- ducing and enhancing intrinsic activation sparsity within large language models

    Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li, Chen Chen, Zhiyuan Liu, Guan- gli Li, Tao Yang, and Maosong Sun. ProSparse: Intro- ducing and enhancing intrinsic activation sparsity within large language models. InProceedings of the 31st In- ternational Conference on Computational Linguistics, 2025

    2025

  55. [57]

    PowerInfer: Fast large language model serving with a consumer-grade GPU

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. PowerInfer: Fast large language model serving with a consumer-grade GPU. InProceedings of the 30th Symposium on Operating Systems Principles (SOSP), 2024

    2024

  56. [58]

    RoFormer: Enhanced trans- former with rotary position embedding.Neurocomput- ing, 568:127063, 2024

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced trans- former with rotary position embedding.Neurocomput- ing, 568:127063, 2024

    2024

  57. [59]

    Ra- zorAttention: Efficient KV cache compression through retrieval heads

    Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Danning Ke, Shikuan Hong, Yiwu Yao, and Gongyi Wang. Ra- zorAttention: Efficient KV cache compression through retrieval heads. InInternational Conference on Learn- ing Representations, 2025

    2025

  58. [60]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  59. [61]

    MuSiQue: Multihop questions via single hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    2022

  60. [62]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, 2017

    2017

  61. [63]

    Prophetkv: User- query-driven selective recomputation for efficient kv cache reuse in retrieval-augmented generation.arXiv preprint arXiv:2602.02579, 2026

    Shihao Wang, Jiahao Chen, Yanqi Pan, Hao Huang, Yichen Hao, Xiangyu Zou, Wen Xia, Wentao Zhang, Chongyang Qiu, and Pengfei Wang. Prophetkv: User- query-driven selective recomputation for efficient kv cache reuse in retrieval-augmented generation.arXiv preprint arXiv:2602.02579, 2026. 24

    arXiv 2026

  62. [64]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art...

    2020

  63. [65]

    LongGenBench: Benchmarking long- form generation in long context LLMs

    Yuhao Wu, Ming Shan Hee, Zhiqing Hu, and Roy Ka-Wei Lee. LongGenBench: Benchmarking long- form generation in long context LLMs. InInterna- tional Conference on Learning Representations, 2025. arXiv:2409.02076

    Pith/arXiv arXiv 2025

  64. [66]

    DuoAttention: Efficient long-context LLM inference with retrieval and streaming heads

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. DuoAttention: Efficient long-context LLM inference with retrieval and streaming heads. InInternational Conference on Learning Representations, 2025

    2025

  65. [67]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InInternational Confer- ence on Learning Representations, 2024

    2024

  66. [68]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chen- gen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jian- wei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Ya...

    Pith/arXiv arXiv 2025

  67. [69]

    Gated delta networks: Improving Mamba2 with delta rule,

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving Mamba2 with delta rule,

  68. [70]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 2369–2380, 2018

    2018

  69. [71]

    CacheBlend: Fast large language model serving for RAG with cached knowledge fusion

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Du, Xie Han, Shan Cao, and Junchen Jiang. CacheBlend: Fast large language model serving for RAG with cached knowledge fusion. InProceedings of the 19th European Conference on Computer Systems (EuroSys), 2025

    2025

  70. [72]

    H2O: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuan- dong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: Heavy-hitter oracle for efficient generative inference of large language models. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  71. [73]

    Gonzalez, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kober, Cedric Shi, Kefan Xiao, Ion Stoica, Hao Zhang, Joseph E. Gonzalez, and Ying Sheng. SGLang: Effi- cient execution of structured language model programs. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  72. [74]

    Dist- Serve: Disaggregating prefill and decoding for goodput- optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Dist- Serve: Disaggregating prefill and decoding for goodput- optimized large language model serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 193–210, 2024. 25

    2024

  73. [2026]

    Accessed: 2026-06-14

    2026