{"id":"5a847ef4-587d-45a7-a76e-6a90c1369878","arxiv_id":"2412.08521","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"EMS improves KV cache compression with a Global-Local importance score and a head-wise Evict-then-Merge strategy, outperforming prior methods on LongBench and Needle-in-a-Haystack.","lead":"This paper proposes EMS, a method to compress the key-value cache that speeds up large language model inference while reducing memory. It combines global and local attention scores to pick important tokens, then evicts or merges the rest per attention head.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The merge-destination metric in Eq. 4 is not validated against attention-output error; if cosine similarity of raw K/V is a weak proxy, the claimed advantage of Evict-then-Merge over eviction collapses.","rationale":"I agree with the reader that Eq. 4 is the weakest load-bearing assumption, but I would sharpen the objection: the issue is not merely absence of external evidence, but the absence of any internal control that separates the merge-destination rule from the overall evict-then-merge pipeline. The paper's own Table 8 only compares variants of the same cosine product and therefore cannot reveal whether the specific choice of R_{i,j} is what drives the improvement. A random-destination control and an attention-error-based upper bound would directly test whether the metric is doing causal work. The concern does not demonstrate that the central result is false; it identifies a condition that must hold for the central claim to be properly supported, and that condition is currently unverified. For this reason the reader's CONDITIONAL verdict remains appropriate, and no new verdict change is needed. The LongBench anomaly on LongChat and the lack of released code are additional verification concerns, but the redundancy metric is the more fundamental, method-internal risk because it determines whether the merge stage is well-founded at all.","tokens_in":19265,"tokens_out":9603,"duration_ms":111447,"concrete_test":"On the Llama-2-7B-Chat LongBench configuration of Table 4 (Nbudget=256, tau=0.6, gamma=4), run three variants: (a) EMS with merge destinations from Eq. 4; (b) identical pipeline except destinations chosen uniformly at random from the class-center set, preserving the same number of tokens above/below tau; (c) destinations chosen to minimize actual attention-output perturbation, estimated by sampling queries from prefilling and decoding and computing, for each TBM token, the L2 change in softmax attention output if it were replaced by each candidate center. If (b) is within about 0.3 average LongBench points of (a), cosine redundancy is not the operative cause of the merge gain. If (c) beats (a) by more than about 0.5 points, Eq. 4 is a demonstrably suboptimal proxy and the head-wise redundancy claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that merging sub-important tokens into class centers preserves attention outputs, and Section 4.2 assigns merge destinations using Eq. 4: R_{i,j} = cos(k_i,k_j) * cos(v_i,v_j), with a threshold tau for whether merging is safe. This is a static, query-independent proxy for the quantity that actually matters, namely the change in softmax attention output when token i is replaced by center d. High cosine similarity of raw K/V states does not guarantee that q·k_i and q·k_d are close for the queries attending to those tokens, and errors introduced by a few badly merged entries can propagate through layers. The paper's only direct support, Table 8, compares three members of the same cosine family (key-only, value-only, key-value) and does not compare Eq. 4 against random merge destinations, against an attention-output-error-based assignment, or against a no-merge two-level eviction baseline with the same candidate pool. Since Table 4 attributes 0.52 LongBench points to the Evict-then-Merge stage, a failure of this proxy would undermine the claimed benefit of merging over eviction-only compression at the 256 budget. Section 7 concedes that the Global-Local alignment is empirical rather than theoretical, but no similar caveat is given for the redundancy criterion, which is equally lacking in formal justification.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes EMS, a training-free KV cache compression method. It computes a Global-Local importance score by mean-aligning accumulated global and local attention weights and taking an elementwise max; it then partitions tokens into irrelevant, to-be-merged, and important sets, evicts the first set, merges the second set into class centers chosen by a product of key and value cosine similarities, and uses a zero-class center to unify eviction and merging in a head-wise parallel manner. The method is evaluated on LongBench, Needle-in-a-Haystack, PG19 perplexity, and throughput, using LLaMA-2, LLaMA-3, LongChat, and Mistral, with reported state-of-the-art results under a 256-token cache budget.","tokens_in":19571,"tokens_out":10787,"duration_ms":110617,"significance":"If the reported results are reproducible, the paper makes a practical contribution: it combines eviction and merging at the head level rather than relying on eviction alone, and it integrates the importance-score computation into FlashAttention2, enabling a measured 10.75x throughput gain in a favorable setting. The zero-class mechanism is an elegant way to keep head-wise merge and evict decisions parallel, and the evaluation spans four LLMs and three benchmark families, which is broader than many KV-cache compression papers. However, the central quantitative claim currently rests on hyperparameters selected using the same LongBench benchmark on which the headline numbers are reported, and on a merge-redundancy proxy that is not directly validated against attention-output error. These issues need to be addressed before the state-of-the-art claim can be accepted.","major_comments":[{"comment":"The hyperparameters τ, γ, ζ, Lwin, and kernel_size are selected through ablations on LongBench, and the same LongBench average is then used as the headline result in Table 1. There is no held-out validation split or nested selection procedure, and Table 1 reports the better of two EMS variants per model, which further increases the risk of fitting to the test set. Section 7 explicitly concedes that the Global-Local alignment is empirical rather than theoretical, which makes a fixed, prespecified hyperparameter choice particularly important. Please provide results with hyperparameters chosen without access to LongBench test labels, or use a separate validation set, and report a single EMS variant or a principled selection rule.","section":"§5.2 and Appendix B, Tables 6 and 7"},{"comment":"The redundancy metric used to choose merge destinations is a query-independent product of cosine similarities. The paper validates only key-only, value-only, and key-value variants of this same metric; it does not compare Eq. (4) against random merge destinations, against an attention-output-error-based assignment, or against an evict-only baseline with the same candidate pool. A token with high cosine similarity to a center can still produce different softmax contributions under the queries that actually attend to it, and errors can propagate across layers. Since Table 4 attributes part of the LongBench gain specifically to the Evict-then-Merge stage, this proxy is load-bearing. Please add a direct comparison of attention-output error induced by Eq. (4) versus alternative merge-destination rules, and an end-to-end comparison to an evict-only method with the same candidate selection.","section":"§4.1, Eq. (4), Table 8"},{"comment":"The Table 1 caption states that SnapKV is an exception and increases its budget at the decoding stage, so the comparison is not at a uniform 256-token cache budget. The title claim of improving scores under a 256 cache budget is weakened if the main baseline is allowed to exceed that budget during generation. Please report SnapKV's effective peak cache size during decoding, or enforce a hard 256 budget for all methods including newly generated tokens. In addition, the LongChat-7B row shows an unusually large gap (EMS 31.94 vs. SnapKV 14.90 at budget 256); please verify that the LongChat results are produced with the official LongBench evaluation protocol and, if the gap persists, discuss the qualitative reason.","section":"§5.2, Table 1"},{"comment":"The compression procedure is not fully specified. Equation (3) defines a head-wise sparsity rate p_m, and Figure 4b reports sparsity and redundancy, but the text does not explain how p_m or r_m determines N_irr, N_tbm, or N_imp in the token partition. In practice the number of merged tokens is controlled by the externally chosen γ = (N_budget + N_tbm) / N_budget; the connection between the observed head-wise statistics and the actual per-head evict/merge ratios is therefore not demonstrated. Please state the algorithm precisely, including how ζ and p_m are used at inference time, or clarify that p_m and r_m are only motivational observations.","section":"§4.2, Eq. (3)"}],"minor_comments":[{"comment":"The text states that Evict-then-Merge improves the score by 0.61 points and that combining global and local scores improves by 0.37 and 2.62 points, but the table shows differences of 0.52, 0.28, and 2.53 points. Please correct the text or the table so the ablation claims match the reported numbers.","section":"§5.5, Table 4"},{"comment":"The first sentence of the Merge Threshold paragraph says a smaller threshold results in more tokens being evicted and a larger threshold leads to more aggressive merging; this is reversed, because a token is merged when its redundancy exceeds τ.","section":"Appendix B, Merge Threshold"},{"comment":"The notation and the window-size statement need clarification: the sums in Eq. (2) should be defined explicitly, and the sentence that the actual window size is \"Lwin ∼ 2Lwin − 1\" should be replaced by a precise statement of the effective window size used to compute the local score.","section":"§3.3, Eq. (2)"},{"comment":"The \"mean pooling function\" applied to the Global-Local score is not defined; please specify the kernel size, stride, and whether pooling is applied along the token dimension for each head.","section":"§4.2"},{"comment":"Table 2 does not report the full-cache retrieval accuracy that is needed to interpret the claim that EMS retains 95.9% of full-cache retrieval ability. Please add the full-cache score and state the exact cache budget corresponding to the abstract's \"less than 2% of context length\" claim.","section":"§5.3, Table 2"},{"comment":"The throughput comparison is only against the full-cache FlashAttention2 implementation; reporting latency and throughput for at least one compression baseline (e.g., SnapKV or H2O) would help characterize the overhead of the modified kernel and the look-up-table expansion.","section":"§5.4, Table 3"},{"comment":"The caption refers to Mistral-7B-Instruct-v1.5, while the experiments in §5.3 use Mistral-7B-Instruct-v0.2; please correct the model name.","section":"Figure 7 caption"}],"recommendation":"major_revision","confidential_remarks":"The method is plausible and the experiments are broad, but the current evidence base is too dependent on in-benchmark hyperparameter selection and an unvalidated merge-redundancy proxy. The LongChat result should be checked for evaluation-protocol artifacts; if it stems from an artifact, the headline improvement should be recomputed. I would be willing to review a revised version that addresses the four major concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Alex,\n\nQuick take on 2412.08521 (EMS). It's a decent, incremental KV cache compression paper. The new bits are a Global-Local importance score (mean-aligned max of global and local attention accumulations) and an Evict-then-Merge strategy that works head-wise, with a zero-class trick to treat eviction as merging so you can keep everything parallel. That zero-class is a nice engineering move. The authors also integrate the score computation into the FlashAttention2 kernel, which gives a real speedup.\n\nWhat's well done: the ablation table is honest. They show that both the score and the merge stage contribute, and that key-value cosine similarity beats key-only or value-only (Table 8). They test across four LLMs, multiple budgets, LongBench, NIAH, and PG19. The throughput numbers are plausible. The paper is clearly written, and the limitations section acknowledges the Global-Local alignment is empirical.\n\nNow the soft spots. The biggest one: the merge-destination metric in Eq. 4 is a static cosine similarity between raw K and V states, used as a proxy for \"safe to merge.\" The paper never validates that this proxy actually tracks attention-output error. Table 8 only compares three members of the same cosine family; it doesn't compare against random merge destinations or an error-based assignment. Since the merge stage gains 0.52 points on LongBench (Table 4), a failure there would hurt the core claim. The stress-test note we got is right that this needs addressing.\n\nSecond: hyperparameters (τ, γ, Lwin, kernel_size) are tuned on LongBench and then LongBench is the main evaluation. There is some independent grounding (NIAH, PG19 use different settings), but the paper doesn't report a validation split or error bars. The LongChat result is suspiciously good — EMS gets ~31-32 while SnapKV gets ~15 and H2O ~8. That gap is so large it makes me wonder if something else is going on (e.g., an evaluation bug or a different decoding behavior). No code is released, which makes verification harder.\n\nBottom line: it's a solid engineering contribution, not a conceptual leap. I'd send it to review, but I'd push for the authors to validate Eq. 4 against attention output error, report standard deviations, and ideally release code. The core idea — combine global/local scores with head-wise evict-then-merge — is worth a serious look. If the merge proxy holds up, this is a useful method for extreme compression.\n\nRecommendation: give it a full review, and expect major revision before acceptance.","headline":"A solid, incremental KV cache compression paper with a sensible global-local score and a neat zero-class trick, but the merge-destination proxy needs validation before I trust the LongChat numbers.","tokens_in":20091,"tokens_out":2684,"would_cite":true,"duration_ms":27114,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A head-wise evict-then-merge strategy with a Global-Local importance score compresses KV cache to a 256-token budget while improving LongBench scores by at least 1.28 points over baselines on four LLMs.","keywords":["KV cache compression","long-context inference","token importance","evict-then-merge","global-local attention","redundancy","LongBench","needle-in-a-haystack"],"falsifier":"Run a trained LLM on a held-out prompt, compute the redundancy of Eq. 4 for all token pairs in a layer, merge only those pairs whose redundancy is just above the threshold, and compare the model's next-token distribution to the unmerged model; if these high-redundancy merges cause larger output changes than simply discarding the same number of tokens, then the product-of-cosine-similarities criterion is not a faithful measure of merge safety.","tokens_in":19067,"feed_emoji":"🧠","tokens_out":7105,"duration_ms":66327,"temperature":0.7,"pith_summary":"This paper argues that KV cache compression under extreme budgets fails less because important tokens are hard to find than because token selection is biased and compression ignores head-wise redundancy. It proposes EMS, which scores tokens with a Global-Local importance measure, the element-wise maximum of aligned global and local accumulated attention, and then, per head, evicts the least important tokens and merges sub-important tokens into a small set of class centers, using a zero-class entry so eviction is just another merge. Across four LLMs on LongBench, EMS reports the best average scores at a 256-token cache budget, improving over existing baselines by at least 1.28 points, and it preserves about 95% of full-cache retrieval accuracy in Needle-in-a-Haystack with a budget under 2% of context length. The practical payoff is that an LLM can hold a much smaller KV cache, support larger batches, and run long contexts without the usual quality collapse.","feed_headline":"Evict then merge: KV cache shrinks while scores rise","feed_subtitle":"A global-local score plus head-wise merging lifts LongBench by 1.28 points or more at a 256-token cache.","key_machinery":"The load-bearing object is the combination of a Global-Local importance score and a head-wise Evict-then-Merge strategy that treats eviction as merging into a zero class. The score is $\\mathbf{s}_{\\mathrm{Glo{-}Loc}} = \\max(\\mathbf{s}_{\\mathrm{Glo}} \\cdot (\\sum \\mathbf{s}_{\\mathrm{Loc}}/N)/(\\sum \\mathbf{s}_{\\mathrm{Glo}}/N), \\mathbf{s}_{\\mathrm{Loc}})$, where $\\mathbf{s}_{\\mathrm{Glo}}$ and $\\mathbf{s}_{\\mathrm{Loc}}$ are the accumulated attention weights from all queries and from a local window respectively; the mean-alignment term removes the scale gap that otherwise pushes selection to the front or the end of the context. The merge criterion is the redundancy $R_{i,j} = \\cos(\\mathbf{k}_i, \\mathbf{k}_j) \\cdot \\cos(\\mathbf{v}_i, \\mathbf{v}_j)$, which is high only when both the key and the value of a candidate token resemble those of a class center; tokens above threshold $\\tau$ are merged, tokens below it are sent to the zero-class token, and a position look-up table expands the small stored cache back to $\\gamma N_{\\mathrm{budget}}$ entries at computation time. This construction is what lets each head choose a different evict/merge ratio while still running in parallel.","core_discovery":"On the paper's own terms, the central discovery is that extreme KV-cache compression is limited less by how many tokens one keeps than by two solvable defects: importance scores that are biased toward early or recent tokens, and compression strategies that ignore how much redundancy varies from head to head. EMS fixes the first with a Global-Local score that aligns the mean magnitudes of the globally accumulated attention vector and the locally accumulated attention vector and takes their element-wise maximum, so retained tokens are drawn from both perspectives. It fixes the second with an adaptive Evict-then-Merge procedure: within each head, the least important tokens are evicted, the most important tokens serve as class centers, and sub-important tokens are merged into the nearest center only when the product of their key and value cosine similarities exceeds a threshold; a zero-class center turns eviction into a special case of merging so all heads can be compressed in parallel. The paper reports that this combination holds up precisely where prior methods break down: at a 256-cache budget across four LLMs on LongBench, under 2% context-length budgets on Needle-in-a-Haystack, and with up to 10.75x throughput gains over a fully cached FlashAttention2 model.","pith_inferences":["If the cosine-product redundancy faithfully predicts merge safety, the same merge machinery could be applied to other stateful transformer caches, such as cross-attention memories or recurrent state summaries, not just autoregressive KV caches.","The mean-alignment step is a heuristic; a model- or task-dependent calibration of global versus local importance could make token selection more robust, especially for retrieval-heavy prompts where one perspective dominates.","The equal-per-head budget suggests a natural combination with layer-wise or head-wise budget-allocation methods; EMS's own numbers at 1024-token budgets indicate the method would likely benefit from allocating more entries to low-sparsity, low-redundancy heads.","A testable extension is to run EMS on tasks with deliberate distractors or on models with grouped-query attention to see whether the head-wise merge decisions transfer when heads share KV projections."],"forward_implications":["At the fixed 256-token-per-head budget, EMS reports the lowest perplexity and the best LongBench average over StreamingLLM, H2O, CAM, and SnapKV on all four tested LLMs, improving the average by at least 1.28 points and by 17.64 points on LongChat.","In Needle-in-a-Haystack, EMS preserves around 95% of full-cache retrieval accuracy using a cache budget below 2% of context length, tracking the performance of much larger caches.","The per-head eviction and merging is parallelizable through the zero-class mechanism, so the method is not just a quality improvement: with a 256-token budget it supports up to 16 batches and reaches 10.75x higher throughput than a fully cached FlashAttention2 model on 4k prompt plus 8k generation.","Because eviction is treated as merging into a zero-class entry, the same code path handles heads with different merge/evict ratios, and the position look-up table keeps the stored cache at a constant size per head.","Ablations attribute the gain to both components: Global-Local scoring over global or local alone, and Evict-then-Merge over evict-only, so the claim is that each design choice contributes separately."],"supporting_citations":[{"why":"Baseline that selects KV tokens by globally accumulated attention; supplies the global-bias point EMS corrects.","marker":"Zhang et al., 2023"},{"why":"Baseline that selects tokens by local-window attention; supplies the local-bias point and a retrieval-oriented comparison.","marker":"Li et al., 2024b"},{"why":"Merge-based compression baseline whose value-token merging EMS extends to joint key-value redundancy.","marker":"Zhang et al., 2024"},{"why":"The FlashAttention2 kernel EMS modifies to compute global and local attention scores efficiently.","marker":"Dao, 2024"},{"why":"Establishes attention-sink and local-token retention, motivating the local component of the score.","marker":"Xiao et al., 2024"},{"why":"The LongBench benchmark used to measure task performance under the fixed 256-token budget.","marker":"Bai et al., 2023"}],"fun_headline_variants":["EMS: Global-local scores rescue extreme KV cache compression","Head-wise evict-then-merge beats extreme KV cache limits","Extreme KV cache: evict and merge, not just keep important tokens","EMS: 1.28+ LongBench gain using 256-token cache","Global-local attention scores fix biased KV cache pruning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on assuming that high cosine similarity between a token's key and value and a class center's key and value means the two tokens are interchangeable enough that merging them does not change what the model generates.","fun_headline_variants_meta":{"raw":{"variants":["EMS: Global-local scores rescue extreme KV cache compression","Head-wise evict-then-merge beats extreme KV cache limits","Extreme KV cache: evict and merge, not just keep important tokens","EMS: 1.28+ LongBench gain using 256-token cache","Global-local attention scores fix biased KV cache pruning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001089,"raw_usage":{"total_tokens":4622,"prompt_tokens":1092,"completion_tokens":3530,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":708,"completion_tokens_details":{"reasoning_tokens":3441}},"tokens_in":708,"tokens_out":3530,"duration_ms":28115,"temperature":1.0,"reasoning_tokens":3441,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:44:10.072570+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a trained LLM on a held-out prompt, compute the redundancy of Eq. 4 for all token pairs in a layer, merge only those pairs whose redundancy is just above the threshold, and compare the model's next-token distribution to the unmerged model; if these high-redundancy merges cause larger output changes than simply discarding the same number of tokens, then the product-of-cosine-similarities criterion is not a faithful measure of merge safety.","supporting_citations":[],"review_version":1}