Pith. sign in

REVIEW 4 major objections 10 minor 60 references

RCStat: A Statistical Framework for using Relative Contextualization in Transformers

T0 review · 4 major / 10 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that pre-softmax attention logits encode token relevance that softmax normalization obscures, and that a single statistic built from them — Relative Contextualization — can drive both KV-cache compression and attribution…

desk verdict A correct copula bound on pre-softmax logits is the real contribution; the KV-compression claims are plausible but rest on an unvalidated prompt-window proxy. read the letter →

arxiv 2506.19549 v1 pith:4VM6WDTN submitted 2025-06-24 cs.CL cs.AIcs.LG

Debabrata Mahapatra , Shubham Agarwal , Apoorv Saxena , Subrata Mitra This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords relativecontextualizationpre-softmaxattentionlogitsKVcachecompressiontokenattributionheadselectiontransformerinterpretabilityLLMinferenceefficiencysinks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that softmax-normalized attention weights flatten and distort the token-relevance information that lives in raw pre-softmax query–key logits, and that this lost information matters for two practical tasks. The authors define Relative Contextualization (RC), a random variable that compares how strongly prompt tokens align with generated tokens versus how strongly generated tokens align among themselves, and derive a near-linear-time upper bound for its expectation. RC then serves as a per-head relevance score: KV-cache eviction is driven by a per-token threshold comparing that token's RC with the prompt's overall RC, and attribution is improved by keeping only the top-RC heads. On LLaMA-3.2-3B and LLaMA-3.1-8B across QA and summarization datasets, the paper reports the lowest Value Error Rate at every compression ratio studied and better RougeL-F1 frontiers than post-softmax baselines, plus 2–3% higher attribution accuracy from about 2% of heads, all without retraining. A sympathetic reader should take the central claim as: raw logit-space statistics are a general-purpose, training-free relevance signal for transformer internals.

What carries the argument

The central object is the Relative Contextualization random variable, $Z_s(p_1,g_1)=\max(X_s(p_1,g)-Y_s(g_1,g),0)$, where $X$ is the distribution of $\langle q,k\rangle$ logits between a prompt subset and the generated tokens and $Y$ is the distribution of logits among generated tokens; large $Z$ means the prompt part is actively grounding the generation in that head. Since $X$ and $Y$ are dependent and their joint law is intractable, the paper proves the bound $\mathbb{E}[Z]\le A=\int_{-\infty}^{\infty}\min(F_Y(t),1-F_X(t))\,dt$ using copula concentration bounds, and computes $A$ by a Lebesgue-style sweep over sorted logit samples. In the KV-compression application the mechanism is an eviction rule: evict token $t_i$ when $\mathbb{E}[Z_p(\{t_i\},\hat g)]\le c\,\mathbb{E}[Z_p(p\setminus\hat g,\hat g)]$, with $\hat g$ a window of the last few prompt tokens standing in for future generation, so each head's cache keeps only tokens whose expected contextual contribution clears the threshold.

What would settle it

Re-run the compression experiments with the reference window $\hat g$ drawn from the beginning or middle of the prompt, or from the actual generated continuation, and compare Value Error Rate and RougeL-F1 at matched compression ratios; if RCStat no longer dominates SnapKV and TOVA, the claimed benefit is an artifact of the recency proxy rather than of logit-space contextualization.

Watch

Extended reading notes

Core claim

RCStat's central discovery is that the expected surplus of cross-contextualization over self-contextualization—RC, in short—ranks attention heads and prompt tokens more faithfully than post-softmax attention. Concretely, pre-softmax logits from prompt tokens to generated tokens (cross-contextualization) and among generated tokens (self-contextualization) are treated as two distributions; their degree of separation, measured by the overlap area between the self-contextualization CDF and the cross-contextualization survival function, upper-bounds the expected RC and is computable in $~O(|p||g|+|g|^2)$ time. The authors use exact expected RC for eviction decisions and the upper bound for long-context attribution, and report that heads with high RC are anti-correlated with per-head compression ratios, that high-RC heads cluster in middle layers, and that keeping only those heads improves attribution while bottom-ranked heads degrade it sharply.

Load-bearing premise

The compression claim stands on treating the last few prompt tokens as a proxy $\hat g$ for the tokens the model will actually generate; if those window tokens are not representative of future generation, the RC scores are computed against the wrong reference distribution and the threshold $c$ no longer has a principled meaning.

Editorial extensions

If this is right

  • The same scalar $c$, without retraining or per-layer schedules, produces head-specific cache budgets: heads with higher RC retain more keys, which the paper verifies by showing per-head compression ratio is anti-correlated with RC.
  • Attention attribution can be reduced to a head-selection step: a compact top-RC subset of heads (roughly 2% of all heads) outperforms using all heads on QuoteSum and VERI-GRAN, while the bottom-RC subset performs near chance.
  • RC scores are sensitive to task difficulty and context relevance: a prompt unrelated to the question activates far fewer high-RC heads than a relevant prompt, and QMSum summaries recruit about nine times more high-RC heads than SQuAD single-hop answers.
  • Because the framework only touches logits during inference, it is applicable to any auto-regressive transformer with exposed attention logits and can be composed with quantization or low-rank KV methods, which the paper positions as future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The clearest untested consequence is RC as a hallucination or grounding detector: if low RC measures weak prompt-to-generation alignment, then generations with low RC against their context should be the ones that fabricate content; the authors name this as a future direction, and it is directly checkable on an existing hallucination benchmark.
  • A stress test on the proxy window would separate contextual signal from recency artifact: re-running eviction with $\hat g$ taken from the beginning or middle of the prompt, or from the true continuation, should preserve the reported ordering; if it does not, the gains depend on the proxy rather than on RC itself.
  • The theory suggests an architectural probe: since the upper bound is tight only for continuous CDFs and the paper notes that the discrete case does not guarantee tightness, measuring the gap between exact expected RC and its overlap-area bound per head would show which heads' rankings are most sensitive to the copula bound, and could motivate a sharper estimator.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 10 minor

Summary. RCStat formalizes pre-softmax attention logits as random variables for an autoregressive transformer. For a sequence s = p ⊕ g, the paper defines cross-contextualization X_s(p1, g') and self-contextualization Y_s(g1, g') as empirical distributions over logits between specified token subsets, then defines relative contextualization Z_s = max(X_s − Y_s, 0). Theorem 3.6 proves that E[Z] is bounded above by A = ∫ min(F_Y(t), 1 − F_X(t)) dt and below by a = ∫ max(F_Y(t) − F_X(t), 0) dt, using Sklar's theorem and the Fréchet–Hoeffding bounds; Algorithm 1 computes A in Õ(|p||g| + |g|²) time. Two applications are presented. In KV-cache compression (§4.1), a prompt token t_i is evicted when its expected RC against a window ĝ of the last w prompt tokens is at most c times the prompt-level expected RC against ĝ, with ĝ used as a proxy for the unknown future generation. In attribution (§4.2), heads are ranked by sequence-level RC, the top-k are kept, and spans are scored by normalized RC. Experiments on LLaMA-3.2-3B and LLaMA-3.1-8B Instruct report lower Value Error Rate and better RougeL-F1 frontiers than KNorm, SnapKV, StreamingLLM, and TOVA on QMSum, 2WikiMultiHop, and SQuAD v2.0, and 2–3% higher chunk-level attribution accuracy on QuoteSum and VERI-GRAN from selecting the top-20 heads.

Significance. The theoretical core is the paper's strongest part and it is correct: in Appendix B, the proof of Theorem 3.6 validly applies the Fréchet–Hoeffding bounds through Sklar's theorem, and the authors appropriately note that with discrete marginals the copula is not unique and the bounds need not be tight; the same argument yields both the upper and lower bound. Algorithm 1 is a concrete, efficient realization of the bound, and the framework's ability to address both compression and attribution from one statistic is attractive. The paper also includes honest internal controls: Table 2 shows that the independence (IOT) approximation degrades VER, and Appendix Table 3 shows that applying the identical RC-ranking procedure to post-softmax weights loses the attribution gain, which is the key evidence for the pre- vs. post-softmax thesis.

major comments (4)
  1. [§4.1, Eq. (9)] The compression claim stands or falls on the proxy ĝ. Eq. (9) evicts token t_i when E[max(X_p({t_i}, ĝ) − Y_p(ĝ, ĝ), 0)] ≤ c E[Z_p(p\ĝ, ĝ)], where ĝ is the last w prompt tokens used in place of the future generation g. Each prompt token is therefore scored against the last few prompt tokens rather than against the tokens the model will actually generate, and the paper offers no evidence that the window's query distribution matches the generation distribution: it only reports that w=8 beats w=16. On long or multi-hop prompts (including 2WikiMultiHop, which is in the benchmark suite), relevant tokens may not be well represented by the final window, and in that case the threshold c has no demonstrated connection to the value-fidelity objective in Eq. (8); the improved VER and RougeL-F1 frontiers in Figures 5 and 6 could then reflect a favorable proxy choice rather than the superiority of pre-softmax RC. I ask for (i) a direct comparison between window-based scores and scores computed against truly generated tokens (e.g., the first few decoded tokens) to validate the proxy, and (ii) an ablation with alternative proxies (random prompt window, question-only tokens, full prompt) to show that the gains are attributable to RC rather than to the proxy.
  2. [§5.2 and §2] The 'state-of-the-art' claim is not supported by the baseline selection. In Section 2 the paper identifies QFilter [22], PyramidKV [6], and Ada-KV [20] as recent variable-budget methods, but none of them appears in the compression experiments of Section 5.2, which compare only KNorm, SnapKV, StreamingLLM, and TOVA. Likewise, the attribution discussion in Section 2 centers on the trained explainer of Cohen-Wang et al. [11], but Table 1 includes no comparison against it. Consequently the abstract's quantitative claims — 'improves generation quality by 15–40%' and 'achieves 2–5% higher compression than prior SOTA' — cannot be traced to any reported number or figure; the figures show VER and RougeL-F1 at matched compression ratios, not a percentage quality improvement or a compression gain at matched quality. Please either add the missing recent baselines or restrict the claims to the methods actually evaluated, and make the abstract numbers derivable from the included experiments.
  3. [§5.1, Figure 4a] The claim of a 'clear anti-correlation' between head-wise compression ratios and RC scores is contradicted by the paper's own data: Example 2 in Figure 4a reports a correlation of −0.01, and the claim rests on only two examples. A correlation indistinguishable from zero in one of two illustrative cases does not support the adaptive head-wise eviction narrative. Please report the distribution of head-wise compression-ratio/RC correlations over a sizable sample of examples (e.g., all 2000 SQuAD or QMSum examples) with mean and spread, and discuss the fraction of examples for which the anti-correlation actually holds.
  4. [§5.3, Table 1] The attribution gains are small (93.91 vs. 90.54 on QuoteSum, 79.37 vs. 77.91 on VERI-GRAN) and are reported without error bars, standard deviations, or significance tests, so it is not possible to exclude sampling noise at these margins. In addition, only k=20 is reported in Table 1, and although Figure 7 shows the accuracy-versus-k curve, it carries no uncertainty information; the choice of k should be justified or swept with confidence intervals. The conclusion that RC 'boosts' attribution accuracy should be based on a test (e.g., paired bootstrap over examples) that shows the 2–3% difference is not noise.
minor comments (10)
  1. [§3.3, Definition 3.3] The sentence 'We define conditional SC similar to conditional CC in Theorem 3.2' should refer to Definition 3.2, not Theorem 3.2.
  2. [§4.1, Eq. (9)] The max(·) in Eq. (9) is missing its second argument; it should read max(X_p(p_i, ĝ) − Y_p(ĝ, ĝ), 0).
  3. [§5.2, baseline list] The citation 'Knorm [33]' points to MiniCache [33]; the K-norm baseline is Devoto et al. [12], which is cited correctly in Section 2. Please fix the inconsistent citation.
  4. [§5.2, last paragraph] The sentence beginning 'In fact, even at 80~90% compression an the LLM generates answers...' is grammatically broken and appears to introduce a claim about hallucination ('internal model weights learnt during pre-training') that is not connected to the RougeL ordering discussion. Rewrite or remove it.
  5. [§4.2] The text refers to 'the efficient upper bound (Eq. 11, visualized in Figure 3b)', but Eq. (11) is numbered in the appendix; within the main text this bound appears as Eq. (4). Use a single numbering scheme.
  6. [§5.1] The 'p<0.05' in the task-complexity comparison is reported without any description of the statistical test, and the threshold τ=1.5 used for the '130 vs. 12 heads' count appears to be chosen ad hoc. Please specify the test and the choice of τ.
  7. [Appendix A and G] Appendix G refers to csv files with 'mean and standard deviations' and Appendix A states that code 'is to be made available', but no code or data link is actually included in the manuscript. Since the empirical claims depend on these artifacts, a link to a public repository should be provided.
  8. [§3.3] The complexity notation 'Õ(|p||g|+|g|²) computations, where Õ(n)=O(n log n)' is imprecise: Õ denotes suppression of polylogarithmic factors, not a single function evaluation. Please state the intended meaning.
  9. [§3.3, i.i.d. approximation] The formula for the i.i.d. approximation of E[Z|s] contains unbalanced parentheses and brackets; as written it does not parse.
  10. [Figure 4b] The x-axis labels such as '(22,0) (13,13)' are not explained in the caption; the caption should state that these are (layer, head) indices of the selected heads.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central RC bound is a standalone inequality and the empirical claims are evaluated against external benchmarks.

full rationale

The paper's central derivation is self-contained. Theorem 3.6 bounds E[max(X-Y,0)] by integrals of the marginal CDFs via Sklar's theorem and the Frechet-Hoeffding bounds; the inequality holds for any joint distribution and its statement and proof do not use the downstream metrics, the task labels, or any fitted parameter. In KV compression, Eq. (9) compares each prompt token's expected RC against a window proxy g-hat with a scalar threshold c; this is a heuristic eviction rule, not a quantity that is forced by construction to equal the value-fidelity objective in Eq. (8), and the reported VER and RougeL results are external evaluations. The window proxy is an acknowledged approximation taken from SnapKV, not a self-cited theorem, so its representativeness is an empirical robustness concern rather than a circular step. In attribution, RC scores are computed from the model's logits alone, with ground-truth labels entering only at evaluation time, so the claimed accuracy gains are independent evidence. The self-citations that exist (for example ref. [2], and ref. [41] with a co-author) appear in related-work or baseline contexts and are not load-bearing; no cited uniqueness result is invoked to force the choice of RC. The tuned values (c, w, k, tau) are application hyperparameters and are not parameters of the derived bound. I therefore find no prediction that reduces by construction to its input. The main caveat is the g-hat proxy assumption, which is a validity risk for the compression application, not a circular derivation. The KV-compression eviction rule is scored against a prompt-window proxy for future generations; if the proxy is unrepresentative, the compression gains may not establish RC as a superior relevance signal, but the method is still not circular because the proxy and threshold are not fitted to the reported metrics through the derivation. The theorem and attribution pipeline stand independent of that empirical risk. No circular step was identified.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three things: standard copula facts (safe), the SnapKV-style window proxy for future tokens (load-bearing heuristic), and a uniform-support assumption for direct RC computation in Eq. (6). The free parameters are tuning knobs for the two applications. No new physical or ontological entities are postulated; RC, CC, and SC are observational statistics defined on existing pre-softmax logits.

free parameters (4)
  • compression threshold c = 0.2, 0.7, 0.8, 1.0, 1.2, 1.3, 1.8 (default 1.0)
    Controls the eviction cutoff in Eq. (9); larger c evicts more aggressively. The sweep over c defines the reported compression frontier, so the headline tradeoff curve depends on this scalar.
  • window size w = 8 and 16
    Number of last prompt tokens used as proxy g-hat in Section 4.1; the method is evaluated at both, and results differ by configuration.
  • attribution head count k = 20 (2% of 1024 heads)
    Used for the main attribution results in Table 1; the paper does not state a validation procedure for this choice, though Figure 7 sweeps k.
  • high-RC head threshold tau = 1.5
    Used only in Section 5.1 to count contextualizing heads (130 vs 12); the number is post hoc and not tied to downstream performance.
assumptions (3)
  • standard math Sklar's theorem and Frechet-Hoeffding copula bounds
    Used in the proof of Theorem 3.6 to bound the joint CDF term in E[max(X-Y,0)]; standard results from copula theory [17].
  • domain assumption Last-window proxy for future generation
    Section 4.1 approximates the unknown future tokens g by the last few prompt tokens, following SnapKV; if this proxy is poor, eviction scores are misaligned with actual generation.
  • domain assumption Uniform distribution over discrete logit pairs for expected RC
    Eq. (6) computes E[Z] by assuming the joint distribution of Xs and Ys is uniform over its discrete support; the paper notes this is a simplifying assumption and that an i.i.d. approximation performs worse.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RCStat: A Statistical Framework for using Relative Contextualization in Transformers." pith.science (2026). https://pith.science/paper/4VM6WDTN

@misc{pith2026250619549,
  author       = {Pith},
  title        = {Pith review of: RCStat: A Statistical Framework for using Relative Contextualization in Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4VM6WDTN}},
  note         = {Machine review of arXiv:2506.19549}
}
read the original abstract

Prior work on input-token importance in auto-regressive transformers has relied on Softmax-normalized attention weights, which obscure the richer structure of pre-Softmax query-key logits. We introduce RCStat, a statistical framework that harnesses raw attention logits via Relative Contextualization (RC), a random variable measuring contextual alignment between token segments, and derive an efficient upper bound for RC. We demonstrate two applications: (i) Key-Value compression, where RC-based thresholds drive adaptive key-value eviction for substantial cache reduction with minimal quality loss; and (ii) Attribution, where RC yields higher-fidelity token-, sentence-, and chunk-level explanations than post-Softmax methods. Across question answering, summarization, and attribution benchmarks, RCStat achieves significant empirical gains, delivering state-of-the-art compression and attribution performance without any model retraining.

Figures

Figures reproduced from arXiv: 2506.19549 by the authors.

Figure 1
Figure 1. Attribution between generated text (right) and input prompt (left) is analyzed using pre [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the pre-softmax attention logits [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the upper and lower bounds stated in Theorem 3.6 for two types of relative [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: All plots are generated using the mentioned datasets with LLaMA-3B model: 28 layers, 24 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: KV-cache compression performance on QMSum using LLaMA-3.2-3B and LLaMA-3.1- [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: KV-cache compression results on 2WikiMultiHop and SQuAD v2.0 using LLaMA-3.1-8B. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Attribution accuracy as we change number of attention heads, ranked by their RC scores output text overlap. In fact, even at 80 ∼ 90% compression the an LLM generates answers, not from grounding in the context, but internal model weights learnt during pre-training [10,…
Figure 8
Figure 8. Figure 8: Example 1: Relevant context 15 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Example 2: Irrelevant context D Appendix: Plots This appendix presents all evaluation plots across datasets (QMSum, 2WikiMultiHop, SQuAD) and model sizes (3B, 8B). Each plot compares performance metrics–VER, ROUGE-1 F1, and ROUGE-L F1– across different configurations. …
Figure 10
Figure 10. Figure 10: VER scores for 3B model on QMSum, 2WikiMultiHop, and SQuAD datasets. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: VER scores for 8B model on QMSum, 2WikiMultiHop, and SQuAD datasets. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: ROUGE-1 F1 scores for 3B model on QMSum, 2WikiMultiHop, and SQuAD datasets. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: ROUGE-L F1 scores for 3B model on QMSum, 2WikiMultiHop, and SQuAD datasets. [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: ROUGE-1 F1 scores for 8B model on QMSum, 2WikiMultiHop, and SQuAD datasets. [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: ROUGE-L F1 scores for 8B model on QMSum, 2WikiMultiHop, and SQuAD datasets. [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: (see on screen) The distribution of RC upper bound (overlap area) for QmSum dataset. [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: (see on screen) The distribution of RC upper bound (overlap area) for Squad v2 dataset. [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: (see on screen) The distribution of RC upper bound (overlap area) for 2WikiMultiHop [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: (see on screen) Percentiles of RC upper bound (overlap area) across the QmSum dataset [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: (see on screen) Percentiles of RC upper bound (overlap area) across the Squad dataset [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]
Figure 21
Figure 21. Figure 21: (see on screen) Percentiles of RC upper bound (overlap area) across the 2WikiMultiHop [PITH_FULL_IMAGE:figures/full_fig_p025_21.png]
Figure 22
Figure 22. Figure 22: Attention heatmaps from Layer 15 using three attribution strategies. The top head yields [PITH_FULL_IMAGE:figures/full_fig_p026_22.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 22 canonical work pages

  1. [22]

    Q-filters: Leveraging qk geometry for efficient kv cache compres- sion.arXiv preprint arXiv:2503.02812, 2025

    Nathan Godey, Alessio Devoto, Yu Zhao, Simone Scardapane, Pasquale Minervini, ´Eric de la Clergerie, and Benoˆıt Sagot. Q-filters: Leveraging qk geometry for efficient kv cache compres- sion.arXiv preprint arXiv:2503.02812, 2025

    arXiv 2025

  2. [6]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069, 2024

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069, 2024

    arXiv 2024

  3. [20]

    Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference.arXiv preprint arXiv:2407.11550, 2024

    Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference.arXiv preprint arXiv:2407.11550, 2024

    arXiv 2024

  4. [11]

    Learning to attribute with attention.arXiv preprint arXiv:2504.13752, 2025

    Benjamin Cohen-Wang, Yung-Sung Chuang, and Aleksander Madry. Learning to attribute with attention.arXiv preprint arXiv:2504.13752, 2025

    arXiv 2025

  5. [1]

    Quantifying attention flow in transformers.arXiv preprint arXiv:2005.00928, 2020

    Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers.arXiv preprint arXiv:2005.00928, 2020

    arXiv 2005

  6. [2]

    Cache-craft: Managing chunk-caches for efficient retrieval-augmented generation.arXiv preprint arXiv:2502.15734, 2025

    Shubham Agarwal, Sai Sundaresan, Subrata Mitra, Debabrata Mahapatra, Archit Gupta, Rounak Sharma, Nirmal Joshua Kapu, Tong Yu, and Shiv Saini. Cache-craft: Managing chunk-caches for efficient retrieval-augmented generation.arXiv preprint arXiv:2502.15734, 2025

    arXiv 2025

  7. [3]

    Circuit tracing: Revealing computational graphs in language models.Transformer Circuits Thread, 2025

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, et al. Circuit tracing: Revealing computational graphs in language models.Transformer Circuits Thread, 2025

    work page 2025

  8. [4]

    Why the 1-wasserstein distance is the area between the two marginal cdfs, 2021

    Marco De Angelis and Ander Gray. Why the 1-wasserstein distance is the area between the two marginal cdfs, 2021

    work page 2021

Show all 60 references
  1. [5]

    Mechanistic interpretability meets vision language models: Insights and challenges

    Anonymous. Mechanistic interpretability meets vision language models: Insights and challenges. VLM Understanding Blog, 2024

    2024

  2. [7]

    Palu: Compressing kv-cache with low-rank projection.arXiv preprint arXiv:2407.21118, 2024

    Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection.arXiv preprint arXiv:2407.21118, 2024

    2024 arXiv

  3. [8]

    Identifying linear relational concepts in large language models.arXiv preprint arXiv:2311.08968, 2023

    David Chanin, Anthony Hunter, and Oana-Maria Camburu. Identifying linear relational concepts in large language models.arXiv preprint arXiv:2311.08968, 2023

    2023 arXiv

  4. [9]

    Transformer interpretability beyond attention visualization

    Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 782–791, 2021

    2021

  5. [10]

    Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps.arXiv preprint arXiv:2407.07071, 2024

    Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, and James Glass. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps.arXiv preprint arXiv:2407.07071, 2024

    2024 arXiv

  6. [12]

    A simple and effective l 2norm-based strategy for kv cache compression.arXiv preprint arXiv:2406.11430, 2024

    Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective l 2norm-based strategy for kv cache compression.arXiv preprint arXiv:2406.11430, 2024

    2024 arXiv

  7. [13]

    Get more with less: Synthesizing recurrence with kv cache compression for efficient llm inference

    Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. Get more with less: Synthesizing recurrence with kv cache compression for efficient llm inference. arXiv preprint arXiv:2402.09398, 2024

    2024 arXiv

  8. [14]

    How gpt learns layer by layer.arXiv preprint arXiv:2501.07108, 2025

    Jason Du, Kelly Hong, Alishba Imran, Erfan Jahanparast, Mehdi Khfifi, and Kaichun Qiao. How gpt learns layer by layer.arXiv preprint arXiv:2501.07108, 2025

    2025 arXiv

  9. [15]

    Position information in transformers: An overview.Computational Linguistics, 48(3):733–763, 2022

    Philipp Dufter, Martin Schmitt, and Hinrich Sch¨utze. Position information in transformers: An overview.Computational Linguistics, 48(3):733–763, 2022

    2022

  10. [16]

    Transcoders find interpretable llm feature circuits.arXiv preprint arXiv:2406.11944, 2024

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm feature circuits.arXiv preprint arXiv:2406.11944, 2024. 10

    2024 arXiv

  11. [17]

    Copula theory: An introduction

    Fabrizio Durante and Carlo Sempi. Copula theory: An introduction. In Piotr Jaworski, Fabrizio Durante, Wolfgang Karl H¨ardle, and Tomasz Rychlik, editors,Copula Theory and Its Applications, pages 3–31, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg

    2010

  12. [18]

    On the biology of a large language model,

    Nelson Elhage, Catherine Olsson, Neel Nanda, et al. On the biology of a large language model,

  13. [19]

    Trapping llm hallucinations using tagged context prompts.arXiv preprint arXiv:2306.06085, 2023

    Philip Feldman, James R Foulds, and Shimei Pan. Trapping llm hallucinations using tagged context prompts.arXiv preprint arXiv:2306.06085, 2023

    2023 arXiv

  14. [21]

    Model tells you what to discard: Adaptive kv cache compression for llms

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. InThe Twelfth International Conference on Learning Representations

  15. [23]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  16. [24]

    When attention sink emerges in language models: An empirical view.arXiv preprint arXiv:2410.10781, 2024

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view.arXiv preprint arXiv:2410.10781, 2024

    2024 arXiv

  17. [25]

    Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. International Committee on Computational Linguistics

  18. [26]

    Kvquant: Towards 10 million context length llm inference with kv cache quantization.Advances in Neural Information Processing Systems, 37:1270–1303, 2024

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization.Advances in Neural Information Processing Systems, 37:1270–1303, 2024

    2024

  19. [27]

    Simon Jegou, Maximilian Jeblick, and David Austin. kvpress. https://github.com/ NVIDIA/kvpress, 2024. Version released 2024-11-13

    2024

  20. [28]

    A survey on large language model acceleration based on kv cache management.arXiv preprint arXiv:2412.19442, 2024

    Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. A survey on large language model acceleration based on kv cache management.arXiv preprint arXiv:2412.19442, 2024

    2024 arXiv

  21. [29]

    Attributionbench: How hard is automatic attribution evaluation?, 2024

    Yifei Li, Xiang Yue, Zeyi Liao, and Huan Sun. Attributionbench: How hard is automatic attribution evaluation?, 2024

    2024

  22. [30]

    Snapkv: Llm knows what you are looking for before generation.Advances in Neural Information Processing Systems, 37:22947–22970, 2024

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation.Advances in Neural Information Processing Systems, 37:22947–22970, 2024

    2024

  23. [31]

    Matryoshkakv: Adaptive kv compression via trainable orthogonal projection

    Bokai Lin, Zihao Zeng, Zipeng Xiao, Siqi Kou, Tianqi Hou, Xiaofeng Gao, Hao Zhang, and Zhijie Deng. Matryoshkakv: Adaptive kv compression via trainable orthogonal projection. arXiv preprint arXiv:2410.14731, 2024

    2024 arXiv

  24. [32]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  25. [33]

    Minicache: Kv cache compression in depth dimension for large language models.Advances in Neural Information Processing Systems, 37:139997–140031, 2024

    Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Reza Haffari, and Bohan Zhuang. Minicache: Kv cache compression in depth dimension for large language models.Advances in Neural Information Processing Systems, 37:139997–140031, 2024. 11

    2024

  26. [34]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of impor- tance hypothesis for llm kv cache compression at test time.Advances in Neural Information Proces...

    2023

  27. [35]

    Kivi: a tuning-free asymmetric 2bit quantization for kv cache

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: a tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of the 41st International Conference on Machine Learning, pages 32332–32344, 2024

    2024

  28. [36]

    A unified approach to interpreting model predictions

    Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017

    2017

  29. [37]

    Copy suppression: Comprehensively understanding an attention head.arXiv preprint arXiv:2310.04625, 2023

    Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding an attention head.arXiv preprint arXiv:2310.04625, 2023

    2023 arXiv

  30. [38]

    Using captum to explain generative language models.arXiv preprint arXiv:2312.05491, 2023

    Vivek Miglani, Aobo Yang, Aram H Markosyan, Diego Garcia-Olano, and Narine Kokhlikyan. Using captum to explain generative language models.arXiv preprint arXiv:2312.05491, 2023

    2023 arXiv

  31. [39]

    In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  32. [40]

    Transformers are multi-state rnns.arXiv preprint arXiv:2401.06104, 2024

    Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi-state rnns.arXiv preprint arXiv:2401.06104, 2024

    2024 arXiv

  33. [41]

    Peering into the mind of language models: An approach for attribution in contextual question answering

    Anirudh Phukan, Shwetha Somasundaram, Apoorv Saxena, Koustava Goswami, and Bal- aji Vasan Srinivasan. Peering into the mind of language models: An approach for attribution in contextual question answering. InFindings of the Association for Computational Linguistics ACL 2024, p...

    2024

  34. [42]

    Explanations of deep language models explain language representations in the brain.arXiv e-prints, pages arXiv–2502, 2025

    Maryam Rahimi, Yadollah Yaghoobzadeh, and Mohammad Reza Daliri. Explanations of deep language models explain language representations in the brain.arXiv e-prints, pages arXiv–2502, 2025

    2025

  35. [43]

    Know what you don’t know: Unanswerable questions for squad.arXiv preprint arXiv:1806.03822, 2018

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad.arXiv preprint arXiv:1806.03822, 2018

    2018 arXiv

  36. [44]

    On the efficacy of eviction policy for key-value constrained generative language model inference.CoRR, 2024

    Siyu Ren and Kenny Q Zhu. On the efficacy of eviction policy for key-value constrained generative language model inference.CoRR, 2024

    2024

  37. [45]

    ” why should i trust you?” explaining the predictions of any classifier

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ” why should i trust you?” explaining the predictions of any classifier. InProceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016

    2016

  38. [46]

    A primer in bertology: What we know about how bert works.Transactions of the association for computational linguistics, 8:842–866, 2021

    Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works.Transactions of the association for computational linguistics, 8:842–866, 2021

    2021

  39. [47]

    Occam’s laser: Occlusion-based attribution maps for 3d object detectors on lidar data

    David Schinagl, Georg Krispel, Horst Possegger, Peter M Roth, and Horst Bischof. Occam’s laser: Occlusion-based attribution maps for 3d object detectors on lidar data. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1141–1150, 2022

    2022

  40. [48]

    Semqa: Semi-extractive multi-source question answering.arXiv preprint arXiv:2311.04886, 2023

    Tal Schuster, Adam D Lelkes, Haitian Sun, Jai Gupta, Jonathan Berant, William W Cohen, and Donald Metzler. Semqa: Semi-extractive multi-source question answering.arXiv preprint arXiv:2311.04886, 2023

    2023 arXiv

  41. [49]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR, 2017

    2017

  42. [50]

    S. S. Vallender. Calculation of the wasserstein distance between probability distributions on the line.Theory of Probability & Its Applications, 18(4):784–786, 1974. 12

    1974

  43. [51]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InProceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17

  44. [52]

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.arXiv preprint arXiv:1905.09418, 2019

    Elena V oita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.arXiv preprint arXiv:1905.09418, 2019

    1905 arXiv

  45. [53]

    Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks.arXiv preprint arXiv:2407.08454, 2024

    Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks.arXiv preprint arXiv:2407.08454, 2024

    2024 arXiv

  46. [54]

    Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  47. [55]

    Hallucination is inevitable: An innate limitation of large language models.arXiv preprint arXiv:2401.11817, 2024

    Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models.arXiv preprint arXiv:2401.11817, 2024

    2024 arXiv

  48. [56]

    Automatic evaluation of attribution by large language models

    Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 4615–4635, 2023

    2023

  49. [57]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R´e, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Syste...

    2023

  50. [58]

    Explainability for large language models: A survey.ACM Transactions on Intelligent Systems and Technology, 15(2):1–38, 2024

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey.ACM Transactions on Intelligent Systems and Technology, 15(2):1–38, 2024

    2024

  51. [59]

    anchorpersonalstoriestotheland

    Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. QMSum: A new benchmark for query-based multi-domain meeting summarization. Association for Computa- tional Linguistics. 13 A...

    2025

  52. [2025]

    Transformer Circuits

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.