{"id":"1f67f310-5d1f-4927-b283-d323a5dae765","arxiv_id":"2505.17630","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"GIM modifies softmax gradients with a temperature adjustment, layer-norm freeze, and gradient normalization to counter attention self-repair, improving the faithfulness of gradient-based LLM attributions.","lead":"The paper identifies attention self-repair, where softmax renormalization hides the importance of attention scores, and proposes GIM, a backpropagation modification that improves attribution faithfulness. It reports state-of-the-art circuit localization and feature attribution results across seven LLMs and six datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The theoretical proof of vanishing gradients (Eq. 7) reduces to the standard softmax constant-shift invariance, not a novel 'softmax redistribution' mechanism; TSG's link to that proof is therefore unestablished, and the paper's own Limitations concede the causal gap.","rationale":"The reader's weakest_assumption correctly targets TSG's causal role, and the paper's own Limitations support that concern. My analysis goes further: the proof in Section 3.1 is not as novel or mechanism-specific as presented, because Eq. (7) follows from the standard softmax Jacobian and the uniform-contribution condition alone, independent of softmax renormalization dynamics. This weakens the theoretical foundation of GIM, but does not invalidate the substantial empirical results across seven models and six datasets, nor the reproducible code release. Since the paper explicitly disclaims the causal link and the empirical gains are credible, CONDITIONAL remains the right verdict: acceptance pending a more rigorous derivation or direct causal test of TSG's mechanism. I partially agree with the reader because we identify the same overall weak spot (TSG's unestablished mechanism), but I locate the issue one level deeper: the proof itself is a standard softmax property, and the proposed TSG mechanism is not formally connected to breaking that property. If the concrete test shows TSG's effect is independent of the proof's condition, the paper should either reframe its claims as purely empirical or provide a revised theory.","tokens_in":23752,"tokens_out":2304,"duration_ms":19322,"concrete_test":"Construct a minimal two-position attention module with values [1, 1], scores [a1, a2], and upstream gradient such that ∂z/∂s1 = ∂z/∂s2 = c (the paper's Eq. 5 condition). Compute ∂z/∂a1 for T=1, 2, 10 using the paper's TSG backward rule. If the gradient remains exactly zero for all temperatures, then TSG does not 'break the uniformity condition' in the manner the paper claims, and the theoretical motivation is incomplete. If it becomes nonzero, identify which term in the TSG-modified backward pass causes the change and check whether that term corresponds to the paper's stated mechanism of expanding I_ϵ.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central theoretical claim is that under the self-repair condition Eq. (5), ∂z/∂a_j ≈ 0. But Eq. (3) is the standard softmax Jacobian: ∂z/∂a_j = s_j(∂z/∂s_j − Σ_k s_k ∂z/∂s_k). If all ∂z/∂s_k ≈ c, then ∂z/∂a_j ≈ s_j(c − c·Σ_k s_k) = 0 automatically, because Σ_k s_k = 1. This cancellation is a property of softmax being invariant to adding a constant to all logits; it does not depend on 'softmax redistribution' or on the weights s_k. The proof therefore does not establish a mechanism unique to attention self-repair. Moreover, TSG changes the temperature T, which changes s_k but does not affect the cancellation when gradients are exactly uniform. The paper argues TSG 'expands I_ϵ' to break uniformity, but the contributions of newly included positions are weighted by their softmax weights s_k; if those weights are small, the effect on the gradient is proportionally small, and the argument is informal. The authors explicitly state in Limitations that they cannot conclusively establish a causal link between TSG and improved faithfulness. This is the load-bearing weakness: the theoretical motivation for the headline method rests on a standard softmax identity, and the actual mechanism of TSG is not derived from the proof. The empirical results may be real, but the explanation of why GIM works is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies a failure mode it calls \"attention self-repair\": when several attention weights are large and the corresponding value vectors contribute similarly to the output, the gradient of the final logit with respect to each attention score is near zero, so gradient-based attribution methods underestimate the importance of those attention scores. The authors prove this in Section 3.1 (Eqs. 3–7) and propose GIM, a backpropagation method that combines temperature-adjusted softmax gradients (TSG), layer-normalization freezing, and gradient normalization. GIM is evaluated on feature attribution across seven LLMs and six datasets, on layer-wise circuit identification faithfulness, and on the Mechanistic Interpretability Benchmark (MIB) circuit-localization track, where it reports state-of-the-art CPR scores. An ablation study attributes part of the improvement to each modification, and code is released.","tokens_in":24002,"tokens_out":3949,"duration_ms":40335,"significance":"The empirical contribution is potentially valuable: GIM is a simple, cheap modification to gradient-based attribution that reports strong faithfulness gains over established baselines across a wide range of models and datasets, plus competitive MIB results. The paper also honestly discusses residual weaknesses, including per-model variance and the lack of a proven causal mechanism for TSG. However, the theoretical novelty is modest: the zero-gradient result in Eq. (7) is a direct consequence of the standard softmax shift-invariance property, and the paper's central explanation of why TSG works is explicitly conceded to be unproven. If the empirical results are reproducible, the method deserves attention, but the manuscript currently overstates the theoretical contribution and the causal explanation. The broad evaluation, released code, and the inclusion of confidence intervals in the circuit-identification figures are strengths.","major_comments":[{"comment":"The derivation in Eqs. (3)–(7) is mathematically correct, but it reduces to the standard softmax constant-shift invariance: for any set of logits with exactly uniform upstream gradients c, the softmax Jacobian yields ∂z/∂a_j = s_j(c − c Σ_k s_k) = 0 because the weights sum to 1. The result does not depend on the value vectors being similar, on attention-specific structure, or on the particular definition of I_ϵ beyond the uniform-gradient assumption. The paper should either provide a theorem or empirical test that distinguishes attention self-repair from generic softmax behavior, or substantially soften the claim that this is a \"novel form of self-repair\" (Abstract, Section 3).","section":"3.1, Eq. (7)"},{"comment":"The causal link between TSG and the theoretical mechanism is not established, and the paper itself states in the Limitations that \"we cannot conclusively establish a causal link between these observations.\" Specifically, under exactly uniform gradients, increasing the temperature changes s_k but does not break the cancellation in Eq. (7); the argument that expanding I_ϵ to include smaller-weight positions breaks uniformity is informal and unquantified, and newly included positions have proportionally small softmax weights. To support the central claim that TSG works by counteracting attention self-repair, the authors should add an experiment that compares TSG on attention heads with detected self-repair against heads without self-repair, and ideally compare TSG against a control modification (e.g., adding isotropic noise to the softmax gradients) to rule out unrelated effects.","section":"4.1, Limitations"},{"comment":"Table 1 reports point estimates without confidence intervals or significance tests, yet the text makes statements such as \"GIM significantly outperforms\" and \"GIM achieves the highest scores for the majority of combinations.\" Given that many differences between adjacent entries in Table 1 are small (e.g., 0.68 vs. 0.67 in several cells), the absence of uncertainty quantification undermines the comparative claim. The authors should provide bootstrap confidence intervals or pairwise significance tests, at least for the main feature-attribution table.","section":"Table 1, Section 5.2"},{"comment":"The TSG temperature T=2 is chosen based on results on Gemma-2 2B on the FEVER and HateXplain datasets, which are also part of the main evaluation. This is a form of selection on the evaluation data for those dataset–model pairs. Although Table 7 shows that the results are reasonably stable across temperatures, the main claims would be cleaner if T were selected on a separate validation set not used in the reported table, or if the headline results were averaged over a range of temperatures.","section":"Section 5.2 (temperature selection)"}],"minor_comments":[{"comment":"The Abstract gives the code URL as https://github.com/corticph/gim, while Section 9 gives https://github.com/JoakimEdin/gim; these should be reconciled.","section":"Abstract / Reproducibility statement"},{"comment":"There are two separate references with identical titles for Kramár et al. (2024a, 2024b); this looks like a duplicate entry and should be collapsed into one.","section":"References"},{"comment":"Equation (3) is derived for the standard softmax without temperature, but Eq. (2) defines softmax with a temperature parameter τ. The default τ=1 should be stated explicitly before Eq. (3).","section":"Eq. (2)"},{"comment":"AttnLRP is listed as \"-\" for Gemma models; the authors should explain in the caption or text whether the method is not applicable to those models or was not run.","section":"Table 1"},{"comment":"The caption refers to \"layer norm freeze and gradient normalization\" in blue and \"also including TSG\" in orange, but the figure and text do not clearly report the baseline for the relative improvement; please clarify that the baseline is GradientXInput.","section":"Figure 4"},{"comment":"The text says \"We used a temperature of 2 for TSG, which we chose based on the results on Gemma-2 2B on the FEVER and HateXplain datasets,\" but Table 7 was run after this selection; this ordering should be stated more transparently to avoid the appearance of post-hoc selection.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The empirical results are likely to be of interest to the mechanistic interpretability community even if the theoretical framing is overstated. The main risk is that the paper's headline explanation (softmax self-repair and its mitigation by TSG) is not backed by a causal experiment; the authors' own Limitations paragraph concedes this. If the authors can add a head-level or layer-level analysis tying TSG's improvement to the presence of self-repair, and report uncertainty in Table 1, the paper could become a strong contribution. I would not recommend rejection, because the evaluation breadth and the MIB results make the method plausibly useful regardless of the theoretical narrative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"If you read one gradient-attribution paper this quarter, this is worth your time, but take the theory section with salt. The empirical core is genuinely solid: the authors identify attention self-repair, show it occurs repeatedly across seven LLMs, and demonstrate that their GIM recipe (temperature-adjusted softmax gradients, layernorm freeze, grad norm) improves faithfulness on standard feature-attribution benchmarks and, more impressively, on the MIB circuit localization track. The MIB CPR numbers are strong, and the ablation study in Table 4 gives a clear picture of what each modification buys. Credit where due: the work is broad, the code is released, and the Limitations section is unusually candid.\n\nThe soft spots are real but not fatal. The stress-test note is right: Eq. 7 is basically the standard softmax constant-shift invariance, not a novel 'redistribution' mechanism. The proof shows that when all relevant output-gradients are equal, the attention-score gradient vanishes — which is a clean way to state a sufficient condition for self-repair, but it is not a new mathematical result. And TSG's connection to that proof is hand-waved. The paper admits it cannot establish a causal link between temperature adjustment and the improved faithfulness; TSG may work through some other mechanism. That is fine for an empirical heuristic, but the abstract and contribution list oversell the theory.\n\nThere are also smaller issues. The abstract says GIM 'outperforms existing gradient-based methods,' yet in Table 1 IG and AttnLRP win a non-trivial number of cells; 'state-of-the-art' is carried by the MIB track. Table 1 has no confidence intervals or significance tests, although Figure 3 does include 95% CIs for circuit identification. The temperature hyperparameter was chosen on two evaluation datasets, and while Table 7 shows stability across temperatures, that is still tuning on the test bed. The self-repair classifier uses a coefficient-of-variation threshold of 0.1 that is reasonable but somewhat arbitrary.\n\nOverall, the empirical phenomenon and the method deserve serious attention. The paper would benefit from a revision that recalibrates the claims, adds uncertainty estimates to Table 1, and either strengthens the theoretical story or explicitly frames TSG as a heuristic with empirical support. I would send it to peer review, and if I were working on attribution methods I would cite the empirical results. It is not a 'yes' as is, but it is a solid conditional.\n\nRecommendation: engage with it, but push the authors to separate the solid empirical contribution from the speculative framing.","headline":"A broad, well-executed empirical study of a real attention self-repair phenomenon, wrapped in a theory that is thinner than advertised and a method (TSG) whose causal link to that theory is explicitly unproven.","tokens_in":24622,"tokens_out":1422,"would_cite":true,"duration_ms":13507,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Attention self-repair erases gradients; a temperature fix restores them","keywords":["attention self-repair","gradient-based attribution","mechanistic interpretability","circuit localization","feature attribution","softmax gradients","layer normalization","large language models"],"falsifier":"Take any input where the paper's self-repair detector fires (multiple high attention weights on similarly contributing values) and compute the joint effect of ablating the top two attention scores together. If GIM's attributions do not rank those jointly ablating scores substantially above the scores whose individual ablations already change the output, or if a model with zero such self-repair cases still shows GIM's full faithfulness gain, then the proposed mechanism is not what drives the improvement.","tokens_in":23495,"feed_emoji":"🧠","tokens_out":7054,"duration_ms":52994,"temperature":0.7,"pith_summary":"The paper's central claim is that attention self-repair—softmax redistributing weight to similar-value positions—makes both perturbation-based and gradient-based interpretability methods systematically underestimate how much attention scores matter. Building on a proof that the gradient of the output logit with respect to an attention score vanishes when all high-weight positions contribute similarly to the output, the authors introduce Gradient Interaction Modifications (GIM), which changes backpropagation in three ways: temperature-adjusted softmax gradients, layernorm freeze, and gradient normalization. GIM is reported to reach state-of-the-art faithfulness on the circuit localization track of the Mechanistic Interpretability Benchmark and to outperform five existing gradient-based feature attribution methods across seven LLMs and six datasets. If the claim is right, existing mechanistic analyses of transformers may have missed or mis-ranked exactly the attention components that matter most.","feed_headline":"Attention self-repair erases gradients; a temperature fix restores them","feed_subtitle":"Warming softmax gradients during backprop gives more faithful circuit and feature attributions across seven LLMs.","key_machinery":"The load-bearing object is the softmax gradient identity for attention scores, Equation (3), together with the cancellation that occurs under Equation (5). Attention self-repair is named for the situation where several high-weight attention positions carry similar value information, so the softmax renormalizes to compensate when any one score is perturbed; the corresponding gradient $\\partial z/\\partial a_j$ is shown to be near zero. The proposed countermeasure, temperature-adjusted softmax gradients (TSG), replaces the softmax used in the backward pass with a higher-temperature version ($\\tau > 1$), which flattens the attention distribution, enlarges the set $I_\\epsilon$ of positions that receive meaningful gradient weight, and breaks the uniformity that causes cancellation. Two further modifications—layernorm freeze, which treats the normalization scale as constant during backpropagation to block a previously identified layer-norm self-repair, and grad norm, which divides gradients by the number of multiplied inputs—handle other interaction effects. The three modifications together define GIM, and the paper isolates each one's contribution in an ablation study.","core_discovery":"The discovery is a previously unidentified failure mode inside the attention mechanism, formalized as follows. Let $a_j$ be an attention score and $s_j$ its softmax weight; when the value vectors at all positions with substantial weights contribute approximately equally to the output, so $\\partial z/\\partial s_j \\approx c$ for all $j$ in $I_\\epsilon$, the softmax gradient identity $\\partial z/\\partial a_j = s_j(\\partial z/\\partial s_j(1-s_j)-\\sum_{k\\neq j}\\partial z/\\partial s_k s_k)$ collapses to zero because $1-s_j \\approx \\sum_{k\\neq j}s_k$. The paper calls this attention self-repair and shows empirically that jointly ablating the largest attention scores produces much larger output changes than the sum of individual ablations. GIM counteracts the cancellation by recomputing the softmax at a higher temperature during backpropagation, which widens the set of positions with non-negligible weights and lets the two canceling terms separate, and it adds layernorm freeze and gradient normalization for other interaction-driven distortions. The paper's claim is that these modifications make GIM the most consistently faithful gradient-based method for both circuit identification and feature attribution across the tested models.","pith_inferences":["The paper does not claim this, but if attention self-repair is as frequent as reported (65–1200 occurrences per input), published circuit analyses built on single-component activation patching may have systematically omitted redundant-but-important attention edges; GIM offers a way to re-screen those circuits.","The paper's own limitations note that TSG's improved faithfulness is not causally linked to its self-repair mechanism. An unclaimed corollary is that TSG's empirical value may survive even if the firing-squad explanation is wrong, and other attribution methods could adopt higher-temperature backward softmax without endorsing that rationale.","The joint-versus-individual perturbation question the paper raises is general: any gradient method on a model with redundant, interacting substructures faces the same cancellation, so the temperature idea could transfer beyond attention to other softmax-based routing modules such as mixture-of-experts gates.","If modified gradients are better causal estimates, they could be used not only for interpretation but for training signal or structured pruning; the paper only speculates about training, so this remains an inference."],"forward_implications":["Existing gradient-based circuit maps likely downgrade any attention head whose attended positions carry redundant values; re-ranking with GIM should change those maps on exactly the inputs where joint ablation differs from individual ablation.","Faithfulness gains concentrate in early layers, consistent with evidence that LLMs use early-layer attention heads most; circuit localization should weight early-layer attributions accordingly.","A temperature sweet spot exists per model and dataset, so practical deployment of GIM requires tuning $\\tau$; the paper's default of 2 was chosen on two datasets and performance degrades at high temperatures.","Because GIM is a backpropagation modification, it can be dropped into any gradient-based attribution baseline, not only GradientXInput, and the ablations show grad norm plus layernorm freeze already give most of the gain.","Joint perturbation, not single-component perturbation, is the right causal target for importance when self-repair is present; evaluation protocols that only ablate one component will understate importance."],"supporting_citations":[{"why":"Supplies the 'firing squad' analogy for OR-gate causality that motivates joint perturbation and TSG.","marker":"Pearl (2009)"},{"why":"Established layer-normalization self-repair and provides the layernorm-freeze component with the conceptual framework for self-repair.","marker":"Rushing & Nanda (2024)"},{"why":"Introduced the gradient-normalization rule (uniform rule) that GIM adopts as its third modification, with Taylor/Shapley justification.","marker":"Achtibat et al. (2024)"},{"why":"Proposed the ATP* softmax backpropagation fix that GIM's TSG is designed against and that serves as a circuit-identification baseline.","marker":"Kramár et al. (2024b)"},{"why":"Defines the Mechanistic Interpretability Benchmark and its circuit localization track, GIM's headline evaluation.","marker":"Mueller et al. (2025)"},{"why":"Defined comprehensiveness and sufficiency, the faithfulness metrics used in all evaluations.","marker":"DeYoung et al. (2020)"},{"why":"Provides Integrated Gradients, the main gradient-based baseline and the formalism of gradient attributions that GIM extends.","marker":"Sundararajan et al. (2017)"},{"why":"Provides attribution patching (ATP), a circuit-identification baseline and the basis for ATP* and the joint-ablation comparisons.","marker":"Syed et al. (2023)"}],"fun_headline_variants":["Softmax self-repair kills gradients; GIM fixes it with temperature","GIM: taming attention self-repair to restore lost gradients","Gradient vanishing from attention self-repair? GIM heats up softmax","Interaction-aware backprop (GIM) corrects circuit localization gradients"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that running softmax at a higher temperature during backpropagation really approximates jointly ablating several attention scores; the authors state they developed TSG empirically and cannot conclusively establish the causal link.","fun_headline_variants_meta":{"raw":{"variants":["Softmax self-repair kills gradients; GIM fixes it with temperature","GIM: taming attention self-repair to restore lost gradients","Gradient vanishing from attention self-repair? GIM heats up softmax","Interaction-aware backprop (GIM) corrects circuit localization gradients"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000631,"raw_usage":{"total_tokens":2927,"prompt_tokens":974,"completion_tokens":1953,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":1875}},"tokens_in":590,"tokens_out":1953,"duration_ms":13059,"temperature":1.0,"reasoning_tokens":1875,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:43:11.327489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any input where the paper's self-repair detector fires (multiple high attention weights on similarly contributing values) and compute the joint effect of ablating the top two attention scores together. If GIM's attributions do not rank those jointly ablating scores substantially above the scores whose individual ablations already change the output, or if a model with zero such self-repair cases still shows GIM's full faithfulness gain, then the proposed mechanism is not what drives the improvement.","supporting_citations":[{"cited_title":"Causality","cited_arxiv_id":null,"evidence_quote":"Supplies the 'firing squad' analogy for OR-gate causality that motivates joint perturbation and TSG."},{"cited_title":"Explorations of Self-Repair in Language Models , May 2024","cited_arxiv_id":null,"evidence_quote":"Established layer-normalization self-repair and provides the layernorm-freeze component with the conceptual framework for self-repair."},{"cited_title":"AttnLRP : Attention-Aware Layer-Wise Relevance Propagation for Transformers","cited_arxiv_id":null,"evidence_quote":"Introduced the gradient-normalization rule (uniform rule) that GIM adopts as its third modification, with Taylor/Shapley justification."},{"cited_title":"Axiomatic Attribution for Deep Networks","cited_arxiv_id":null,"evidence_quote":"Provides Integrated Gradients, the main gradient-based baseline and the formalism of gradient attributions that GIM extends."},{"cited_title":"Attribution Patching Outperforms Automated Circuit Discovery , November 2023","cited_arxiv_id":null,"evidence_quote":"Provides attribution patching (ATP), a circuit-identification baseline and the basis for ATP* and the joint-ablation comparisons."}],"review_version":1}