{"id":"e674d228-0dd5-45f7-b8a4-f50b03e9df34","arxiv_id":"2412.02252","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PoD cuts KV-cache memory by up to 35% by keeping full key-value states for proximal tokens and sharing one set of keys across layers for distant tokens, with no substantial accuracy loss.","lead":"A new LLM inference method, PoD, cuts key-value cache memory by about 35% by keeping detailed memory for nearby tokens and letting layers share one approximate memory for distant tokens. The method keeps long-context accuracy close to the uncompressed model and can be stacked with token-selection compression.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (3) shares distant-token logits from the block's base layer while Eq. (4) and Appendix C.1 derive gating assuming the current layer's own Q and K; this mismatch is the load-bearing risk to the no-loss claim.","rationale":"The reader's weakest assumption—that inter-layer attention similarity must survive for higher-layer queries after adaptation—is exactly the point where the method's correctness rests. Eq. (3) bypasses the current layer's query and key for distant tokens, and the gating derivation in Appendix C.1 does not match this computation, so the theoretical justification is not sound as written. The empirical results are broad (Needle in a Haystack, LongBench, LEval, InfiniteBench, standard benchmarks) and consistently show PoD near the dense baseline, which is real evidence that the approximation works in practice, even if the paper's derivation is imprecise. The concern is therefore not that the method is obviously broken, but that the no-loss claim is supported only by the specific evaluations shown; the shared-Q design could fail on inputs where a higher layer needs query-specific distant attention. The proposed diagnostic test would settle this directly. Secondary issues (the JS-divergence formula in Eq. (2) defining 'similarity' with a threshold of 0.5, and the slight tension between 'no loss' and the 2.8% average degradation) are worth fixing but do not change the core assessment. Since the concern aligns with the reader's conditional verdict and does not move it, the appropriate recommendation remains the same: conditional acceptance pending release of code/models and the diagnostic test.","tokens_in":23916,"tokens_out":6442,"duration_ms":70269,"concrete_test":"On the final PoD model, instrument a forward pass over LongBench/LEval: for each layer ℓ above the block base, recompute distant logits with Q_ℓ and the stored block keys K_ℓa (a no-training oracle diagnostic), and replace the shared logits with these oracle logits. Compare task accuracy and the average JS divergence between shared and oracle attention distributions. If accuracy improves by more than ~1 point or average JS divergence exceeds ~0.1, the shared-key assumption is load-bearing and the no-loss claim needs qualification; if accuracy stays within noise, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is defined in Eq. (3): for distant tokens, layer ℓ in a block uses aD_ℓ,i = Q_ℓa,i K_ℓa,[ns+1,n−nr]ᵀ/√d, i.e. the base layer's query and key, not the current layer's Q_ℓ and K_ℓ. Eq. (4) then combines the proximal output oP (computed with Q_ℓ, K_ℓ) and the distant output oD (computed with Q_ℓa, K_ℓa) using a gate gℓ,i. Appendix C.1 (Eqs. 5–6) derives this gate from the standard softmax normalization of the current layer's logits, assuming both aP and aD are produced by Q_ℓ and K_ℓ. Because the distant logits actually come from a different query/key pair, the gate is not the true softmax weight: oℓ,i is not a convex combination of the true per-group attention outputs even when the attention distributions are perfectly similar. The post-training adaptation can tune the model to this fixed structure, but the load-bearing assumption is subtler: the base layer's attention over distant tokens must remain an adequate substitute for every higher layer's query-specific attention across all relevant inputs. If a higher layer would need materially different attention over distant tokens for some query, that layer has no mechanism to express it, because distant logits are frozen to the base layer's Q and K. The reported 2.8% average degradation in Table 5 is consistent with this risk being partially realized, and the 'without compromising performance' claim is stronger than the evidence. This is the most load-bearing concern because it directly governs whether the shared-key approximation preserves the information flow needed for long-context accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PoD, a KV cache compression method for long-context LLM inference. PoD keeps the full KV cache for proximal tokens (a fixed number of initial and most recent tokens) and, for all other \"distant\" tokens, shares a single set of attention scores — equivalently, a single key state and query state from the lowest layer of each block — across consecutive layers whose attention patterns are similar. Blocks are discovered offline by a per-head Jensen-Shannon divergence criterion. The model is then post-trained on 5B tokens to adapt to the new attention structure. The evaluation covers Needle in a Haystack, LongBench, LEval, InfiniteBench, standard benchmarks, and memory/throughput measurements, reporting a KV cache reduction of about 35% with an average performance degradation of 2.8% relative to the dense model and better performance than token-eviction and naive layer-sharing baselines.","tokens_in":24305,"tokens_out":5883,"duration_ms":69475,"significance":"If the central claim holds, PoD is a practically useful and conceptually interesting alternative to token-eviction compression: it keeps all tokens in a compact shared form rather than discarding them, and it is orthogonal to token-selection methods such as SnapKV. The empirical work is unusually broad for a KV-cache paper, covering multiple long-context benchmarks, a second model family (LLaMA3.1-8B), and explicit memory-batch-size measurements that confirm the theoretical savings. The authors also disclose the main engineering limitations (latency overhead and post-training cost) in their Impacts and Limitations section. However, the paper's theoretical justification has a load-bearing gap: the gating derivation in Appendix C.1 assumes that both proximal and distant logits come from the current layer's query and key, while Eq. (3) actually computes distant logits from the block's lowest layer. The empirical claim \"without compromising performance\" is also stronger than the reported 2.8% average degradation. These issues are fixable with additional analysis, so I do not see a basis for rejection, but they require a revision before the claims can be accepted as stated.","major_comments":[{"comment":"The derivation of the gating mechanism is inconsistent with the attention-sharing implementation. In Eq. (3), the distant-token logits aD_ℓ,i are computed with Q_ℓa and K_ℓa, the query and key of the lowest layer in the block, not with the current layer's Q_ℓ and K_ℓ. In contrast, Appendix C.1 (Eqs. (5)-(6)) derives the gate g_ℓ,i as the exact softmax weight under the assumption that both aP and aD are produced by the same layer's Q_ℓ and K_ℓ. Consequently, Eq. (4) is not an exact decomposition of the current layer's attention output even when inter-layer attention distributions are perfectly similar, since the normalizing constant uses sums of exponentials of a different query-key pair. This is the load-bearing approximation behind the \"no performance loss\" claim, and the paper currently treats it as an exact derivation. I ask the authors to either (a) state explicitly that this is an approximation that post-training adaptation compensates for, or (b) add an ablation that isolates the effect of the shared query: for example, compare the current Eq. (3) against a variant that uses each layer's own Q_ℓ with the shared K_ℓa for distant tokens. Reporting the post-adaptation substitution error, e.g., the distance between softmax(Q_ℓ K_ℓ^T/sqrt(d)) and softmax(Q_ℓa K_ℓa^T/sqrt(d)) for distant tokens at non-base layers, would directly quantify the risk identified here.","section":"§2.2, Eq. (3); Appendix C.1"},{"comment":"The abstract and conclusion state that PoD reduces KV cache memory \"without compromising performance,\" but Table 5 reports a 2.8% average performance degradation for PoD relative to the dense model, and Table 3 shows practical memory savings of 30.8-33.3% rather than the nominal 35%. The 2.8% number may well be acceptable in practice, especially relative to the 7.4-8.0% degradations of StreamingLLM and H2O and the 31.4% degradation of CLA, but the wording of the claim should be calibrated to the evidence. I request that the authors either soften the claims to \"with negligible / minimal degradation\" or add statistical significance information (multiple seeds, confidence intervals, or per-task error bars) showing that the 2.8% average is not driven by a few outlier tasks. The distinction matters because the central contribution is exactly that compression does not hurt accuracy.","section":"§3.2, Table 5; Abstract; §5"},{"comment":"The layer grouping is computed offline on the pre-adaptation model, but the attention-sharing structure is then post-trained for 5B tokens. The paper does not verify that the inter-layer attention similarities used to form blocks survive the post-training adaptation. If post-training changes attention patterns, the chosen blocks may no longer satisfy the similarity criterion, and the shared distant logits would then be an even rougher approximation for higher layers. I recommend reporting the attention-similarity matrix of the adapted PoD model on held-out sequences, or at minimum ablating grouping choices (e.g., random grouping of equal block sizes vs. the similarity-based grouping) to show that the offline similarity signal is what makes the method work.","section":"§2.1, Algorithm 1; §3.3, Figure 3"}],"minor_comments":[{"comment":"There are typos in the headers: \"LM-Infite\" should be \"LM-Infinite\" and \"T oken-selection\" should be \"Token-selection.\" Please correct these and check all table headers for similar spacing issues.","section":"Table 1; Table 2"},{"comment":"The symbol ℓa is used in Eq. (3) for the block's lowest layer but is not defined in the main text; please define it explicitly (e.g., \"where ℓa = min B_ℓ is the lowest layer in the block\") so that the equation is self-contained.","section":"§2.2, Eq. (3)"},{"comment":"The PoD row reports its configuration as \"16+4080+28K,\" which is not explained in the table caption or surrounding text. Please clarify that 16 and 4080 are the initial and recent proximal token counts, and that 28K denotes the remaining distant tokens whose logits are shared, so the total context is 32K.","section":"§3.1, Table 1"},{"comment":"The ablation study selects the 4K proximal-token budget and the 35% saving rate based on the same LEval validation set that is later used for the headline results. Please state this explicitly and report whether the LongBench, InfiniteBench, and NIAH numbers in Tables 1, 2, and 4 were obtained with this exact configuration, so readers can assess any potential selection bias.","section":"§3.3, Figure 3"},{"comment":"The τ threshold in the computation-optimization analysis is an additional free hyperparameter, and Figure 7 shows that it trades 5% performance loss for 25% computation savings at τ=0.7. This is a useful analysis, but it should be referenced in the main text and its relationship to the 2.8% average degradation in Table 5 should be clarified, since the reported throughput numbers presumably include this computation skipping.","section":"Appendix B.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within the scope of the journal and the empirical contribution is substantial. My main concern is the gap between the exact-looking derivation in Appendix C.1 and the actual computation in Eq. (3); I believe this is addressable with a clearly labeled approximation statement, a quantitative substitution-gap measurement, and an ablation isolating the shared-query assumption. The other issue is calibrating the \"without compromising performance\" claim to the reported 2.8% degradation. Neither issue appears fatal, and the extensive benchmark coverage suggests the method may be genuinely useful. I am recommending major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read PoD with some care. The short version: it's a solid systems contribution with a genuinely new mechanism, and the empirical case is decent, but the headline claim is a bit stronger than the data.\n\nThe core idea is to keep all tokens but store distant-token keys in a shared form for layers with similar attention patterns. The head-wise grouping based on JS divergence is new, and sharing keys only for distant tokens—not values, not all tokens—is a clever middle ground between CLA and eviction. The post-training adaptation is the right way to make this work. The paper also shows the method composes with SnapKV, which is practically useful.\n\nThe experiments are broad: NIAH, LongBench, LEval, InfiniteBench, and standard benchmarks. The results are consistent with the method being close to dense. The practical memory analysis (batch size increase) is a nice touch.\n\nSoft spots: the derivation mismatch. Eq. (3) computes distant-token logits with the block's base layer queries and keys, but Appendix C.1 derives the gate from the current layer's own logits. So the gate is not the true softmax normalization, and the output isn't exactly a convex combination of per-group attention. The post-training can adapt to this structure, so this is not fatal, but it means the theoretical motivation is less clean than it appears. The claimed \"without compromising performance\" is also a bit generous: Table 5 shows a 2.8% average degradation. That's small and arguably within noise, but it's not zero. Finally, no code or models are released at the moment, so the exact numbers are hard to verify.\n\nMinor: the free parameters (ns, nr, the similarity threshold, and the gate threshold tau) are partly tuned on LEval, so the reported numbers are somewhat optimistic, but the method is not reducing to fitting benchmark labels.\n\nWho should read: anyone working on KV cache compression or efficient long-context inference. It's a useful building block. I would send it to referees; with modest revisions on the claims and a clearer statement of the derivation gap, it could be a solid paper.\n\nRecommendation: accept for peer review, request revisions to soften the no-loss claim and ideally release code.","headline":"A competent new KV-sharing mechanism with broad experiments; the derivation gap and a slightly overstated headline are addressable.","tokens_in":24849,"tokens_out":3002,"would_cite":true,"duration_ms":32456,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PoD compresses the KV cache of long-context LLMs by up to 35% without a measured loss in accuracy, by keeping the full cache for the first and most recent tokens and letting layers with similar attention share one set of keys for distant…","keywords":["KV cache compression","long-context LLM inference","inter-layer attention similarity","attention sharing","proximal tokens","post-training adaptation","layer grouping","token importance"],"falsifier":"Run the same offline similarity measurement the paper uses to form blocks, but on the post-adaptation model: for each head and block, compare each higher layer's true attention logits over distant tokens against the shared lowest-layer logits, and check whether the Jensen-Shannon divergence stays below the 0.5 threshold on inputs at the full context length. If for some head the divergence rises well above threshold, or if a needle planted in the distant region becomes retrievable only when the higher layer uses its own keys, then sharing has collapsed and the no-performance-loss claim would fail for that configuration.","tokens_in":23706,"feed_emoji":"🧠","tokens_out":7959,"duration_ms":70546,"temperature":0.7,"pith_summary":"The paper tries to establish that the main memory bottleneck of long-context LLM inference, the key-value cache that grows with every layer and every token, can be cut by about a third without discarding any token and without losing accuracy. The proposed method, PoD, rests on two measured regularities: for next-token prediction the first and most recent tokens dominate, and attention scores on distant tokens are nearly identical across consecutive layers. So PoD keeps full keys and values for proximal tokens while storing distant tokens' keys once per block of similar layers, letting every layer in the block reuse them after a short post-training step adapts the model to this shared structure. If correct, long-context models would need about 35% less cache memory, which the paper shows translates into roughly 30% larger batches on a single GPU, and the method would compose with token-selection compression such as SnapKV for even larger savings.","feed_headline":"Sharing attention across layers shrinks LLM KV cache by 35%","feed_subtitle":"PoD keeps every token in memory, storing distant keys once per layer block after light post-training.","key_machinery":"The carrying mechanism is key-sharing in layer blocks, defined by measured inter-layer attention similarity. For each head, layers that are mutually similar under Jensen-Shannon divergence (threshold $\\ge 0.5$) are merged greedily into blocks; distant-token logits $a^D_{\\ell,i}$ at each higher layer are then computed with the block's lowest-layer query and key, $Q_{\\ell_a}$ and $K_{\\ell_a}$, so only that layer's distant keys need to be stored and cached. A parameter-free gate $g_{\\ell,i} = \\sum \\exp a^P_{\\ell,i} / (\\sum \\exp a^P_{\\ell,i} + \\sum \\exp a^D_{\\ell,i})$ combines proximal and distant attention outputs, which is exactly the decomposition of the dense attention output into two softmax terms over the two token groups. The whole scheme is trained in by the lightweight post-training adaptation, and the gate also enables approximate compute savings by skipping distant-token computation whenever $g \\ge \\tau$ for a tunable threshold.","core_discovery":"PoD's central claim is that distant tokens can be compressed, not forgotten: the entire long context stays in memory at lower cost by exploiting redundancy in how layers attend to it. For each attention head, the paper measures the average Jensen-Shannon divergence between attention-score distributions of consecutive layers, groups consecutive layers whose distant-token attention agrees above a threshold of 0.5 into blocks, and then within each block keeps only the lowest layer's keys for distant tokens. Every higher layer in the block computes its distant-token logits with the block's lowest-layer query and key rather than its own, and a parameter-free gate blends the proximal and distant attention outputs. A light post-training run on 5 billion tokens adapts the model to this structure, and the resulting models match or slightly exceed the dense baseline on LongBench, LEval, InfiniteBench, and Needle in a Haystack while using up to 35% less KV cache memory.","pith_inferences":["The paper shares keys but not values for distant tokens; an immediate extension would apply the same block structure to value states or low-rank value projections, roughly doubling the savings, though values carry more content and the adaptation cost would likely rise.","Because the offline layer grouping is computed once and then fixed, a testable variant is to add a small training regularizer that actively encourages inter-layer attention similarity, which should make the block structure stable for input distributions far from the adaptation corpus.","The paper's finding that token-selection methods degrade at 128K context while PoD holds suggests a general principle, that keeping a degraded copy of every token beats keeping an exact copy of some tokens, which would predict PoD stays ahead of eviction methods on long inputs with many repeated distractors.","The gate $g_{\\ell,i}$ is a per-token, per-layer measure of dependence on distant context, computed for free by the method; it could serve as a runtime signal for detecting when the model is about to need retrieval of a distant fact."],"forward_implications":["KV cache memory drops by up to 35% with no measured accuracy loss, because distant-token keys are stored once per layer block instead of once per layer.","The saved memory shows up operationally: maximum batch size on one GPU grows by more than 30% at prompt lengths from 2K to 16K tokens.","PoD composes with token-selection methods; PoD+SnapKV reaches about 92% KV cache savings with only a small performance drop, so the two compression philosophies are additive.","Retaining every token keeps mid-context needles retrievable, unlike token-eviction methods; PoD scores 98.9 on Needle in a Haystack versus 56.8 for StreamingLLM and 55.6 for H2O.","A gate threshold $\\tau$ lets decoding skip distant-token computation for tokens whose attention is already dominated by proximal context, trading about 25% of computation for roughly 5% performance at $\\tau = 0.7$."],"supporting_citations":[{"why":"Supplies the attention-sink observation that first and recent tokens dominate next-token prediction, and serves as a token-eviction baseline that PoD outperforms.","marker":"Xiao et al. 2024"},{"why":"Defines the proximal (start plus recent) versus distant split of the context that PoD adopts.","marker":"Han et al. 2024"},{"why":"The layer-sharing baseline (CLA) that shares keys and values across layers; PoD must and does beat it.","marker":"Brandon et al. 2024"},{"why":"Early evidence that attention weights can be shared across layers, motivating Observation 2.","marker":"Xiao et al. 2019"},{"why":"Reuse Transformers, the prior report of inter-layer attention redundancy that the paper scales to modern LLMs.","marker":"Bhojanapalli et al. 2021"},{"why":"Defines the Jensen-Shannon divergence used to measure inter-layer attention similarity and form blocks.","marker":"Menéndez et al. 1997"},{"why":"The Dolma corpus from which the 5 billion adaptation tokens are sampled.","marker":"Soldaini et al. 2024"},{"why":"SnapKV, the token-selection baseline that combines with PoD for further cache compression.","marker":"Li et al. 2024"},{"why":"LongBench, the real-world benchmark used for most of the performance comparisons.","marker":"Bai et al. 2024"}],"fun_headline_variants":["PoD: Share distant-token keys across layers, drop KV cache by 35%","Keep all context, reuse keys across layers to cut KV memory 35%","KV cache 35% lighter: share keys for distant tokens across layers","KV cache down 35% with no quality drop via key sharing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that attention scores for distant tokens computed at the lowest layer of a block, using that layer's query and key states, stay accurate enough for every higher layer in the block, even though the similarity that groups those layers is measured on the original model before the adaptation training.","fun_headline_variants_meta":{"raw":{"variants":["PoD: Share distant-token keys across layers, drop KV cache by 35%","Keep all context, reuse keys across layers to cut KV memory 35%","KV cache 35% lighter: share keys for distant tokens across layers","KV cache down 35% with no quality drop via key sharing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000831,"raw_usage":{"total_tokens":3684,"prompt_tokens":1059,"completion_tokens":2625,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":2542}},"tokens_in":675,"tokens_out":2625,"duration_ms":16489,"temperature":1.0,"reasoning_tokens":2542,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:40:52.192046+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same offline similarity measurement the paper uses to form blocks, but on the post-adaptation model: for each head and block, compare each higher layer's true attention logits over distant tokens against the shared lowest-layer logits, and check whether the Jensen-Shannon divergence stays below the 0.5 threshold on inputs at the full context length. If for some head the divergence rises well above threshold, or if a needle planted in the distant region becomes retrievable only when the higher layer uses its own keys, then sharing has collapsed and the no-performance-loss claim would fail for that configuration.","supporting_citations":[],"review_version":1}