Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

The paper proves that, when tokens are sums of linearly represented features, the top singular vectors of an attention head's QK matrix align with the features that head is paying attention to, and shows this alignment produces sparse atten

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:29 UTC pith:7JQDGT4V

load-bearing objection First systematic theory-plus-toy-model case for singular-vector/feature alignment in attention heads; conditional and overclaimed at the real-model end, but a solid, referee-worthy advance. the 3 major comments →

arxiv 2602.13524 v2 pith:7JQDGT4V submitted 2026-02-13 cs.LG cs.AI

Singular Vectors of Attention Heads Align with Features

Gabriel Franco , Carson Loughridge , Mark Crovella This is my paper
classification cs.LG cs.AI
keywords singular vector-feature alignmentQK matrixsparse attention decompositionlinear representationstokens as sums of featuresattention headstoy modelinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks why and when the singular vectors of an attention head's QK matrix point at the features that head cares about—a regularity several interpretability studies have used without proof. It shows in a toy model where features are directly observable that alignment is consistent across many settings, adds theorems proving that, under linear feature sums and a bilinear attention target, the unique minimizer of cross-entropy is a rank-one matrix whose singular vectors are the features (exactly for isotropic features, approximately otherwise), and proves an orthogonalization theorem that allows multiple features to align to distinct singular vectors. It then converts alignment into a testable prediction—sparse attention decomposition—and reports that this sparsity emerges during training in both the toy model and real language models on an indirect-object-identification circuit. If correct, the result supplies a principled basis for identifying features by decomposing activations in the SVD basis of attention weights.

Core claim

The central claim is that singular vector–feature (SVF) alignment is not an accident but a consequence of how attention heads are trained: when tokens are sums of linearly represented features and the head must attend to specific feature pairs, cross-entropy minimization forces the learned QK matrix into a rank-one form whose top left and right singular vectors are the whitened feature directions of interest. With isotropic features this is exact alignment; with near-isotropy, alignment holds up to a bound on feature interference. The same training objective, combined with reconstruction loss, pushes non-attended features orthogonal to aligned singular-vector directions, so multiple feature

What carries the argument

The central object is the QK matrix Ω = W_Q^T W_K of an attention head. The key identity is that, under the student-teacher setting where tokens are independent Bernoulli sums of features, the unique minimizer of population cross-entropy to a teacher attention target is Ω* = α(Σ_X^{-1} x_1)(Σ_Y^{-1} y_1)^T, a rank-one matrix whose singular vectors are exactly the whitened feature directions. When feature sets are isotropic (Σ_X = aI, Σ_Y = bI), singular vectors become exactly the features x_1, y_1; when nearly isotropic, a singular-vector perturbation bound gives an explicit angle estimate. The paper's Theorem 4 shows that a reconstruction-loss penalty drives features orthogonal to already-a

Load-bearing premise

The theorems assume the attention target the head is trained to match is a bilinear form in whitened feature-presence statistics, so the teacher already has its singular vectors along (whitened) features; if real attention is not bilinear in feature indicators or features are strongly anisotropic, the proof does not transfer.

What would settle it

Train the paper's toy model with a teacher that is not bilinear—for example, attention logits set by a conjunction or threshold of feature-presence indicators—and measure whether the learned QK's top singular vectors still align with the attended features. If alignment largely persists, the bilinear-teacher assumption is not the operative mechanism; if it fails, the explanation offered here is falsified. A real-model check would compare a head's top singular directions against feature directions obtained by an independent activation-based method on the same token pairs.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Feature identification can proceed by projecting activations onto a head's singular vectors; when alignment holds, these projections give candidate features in a single forward pass per prompt.
  • Multiple features of interest are assigned to distinct singular directions up to head dimension, because features orthogonalize to keep reconstruction loss low—so a head's top singular directions form a bounded search space for its features.
  • Sparse attention decomposition—relative attention concentrated on few SVD components—is a reliable predictor that a limited set of attended features is present; high sparsity correlates with fewer features.
  • The number of singular vectors needed to reconstruct relative attention estimates how many features a head is attending to; in the real-model circuit studied, typical counts are 1-4 and decline during training.
  • Because sparsity emerges during training in both toy and real models, monitoring decomposition sparsity offers a training-time diagnostic for when heads commit to feature-specific computations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the results suggest a cheap, training-free diagnostic—measure SVD-basis sparsity of relative attention across many prompts for every head; heads with persistently sparse decompositions are likely to be interpretable as feature-detectors, while persistently diffuse heads may be mixing many features. This is a testable extension beyond the paper.
  • Editorial inference: since the teacher target is a whitened bilinear form, the singular vectors that align are whitened feature directions, not raw feature directions; when features are correlated, interpreting singular vectors as the features themselves may be subtly wrong—the right reading is that singular vectors point along decorrelated feature contrasts. This distinction is explicit in the pa
  • Editorial inference: if real attention targets are non-bilinear (e.g., attention depends on feature conjunctions or on relative feature strengths), the alignment theorem's premises fail; a direct test would be to induce a head to implement a non-bilinear function in the same toy setup and check whether alignment breaks. This would mark the boundary of when the result transfers.
  • Editorial inference: the over-capacity experiments (more feature pairs than head dimensions) predict that the smallest singular vector of a head accumulates low-priority features in superposition; a real-model corollary is that the smallest singular directions are the least likely to be monosemantic, and should be treated cautiously in interpretability pipelines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that the singular vectors of an attention head's QK matrix align with linearly represented features under a token-as-sum-of-features model. It supports this claim with a toy model, a set of theorems in Appendices B.2--B.3, and a testable prediction called sparse attention decomposition (SAD), which the authors report observing in GPT-2 and Pythia on an IOI prompt. Theorems 1--3 show that, when the teacher logits are a bilinear form in whitened feature directions, the unique minimizer of population cross-entropy is rank-one with singular vectors in those whitened directions; near-isotropy yields approximate alignment. Theorem 4 gives a conditional orthogonality result under an explicit interference penalty. The paper concludes that SVF alignment has strong empirical and theoretical justification as a basis for feature identification.

Significance. If the central claim were established, SVF alignment would offer a fast, weight-based route to feature identification that is far cheaper than SAEs or activation patching. The toy-model evidence is a genuine strength: the paper provides systematic sweeps over feature probability, loss weight, head dimension, number of features, context length, and random seeds, and the rotation controls in Section 5 rule out the low-rank/spectrum explanation of the real-model sparsity. Theorems 1--3 are internally consistent and provide a rigorous recovery guarantee for a particular bilinear teacher. However, the teacher is explicitly constructed so that its singular vectors are whitened feature directions, making the theoretical results sufficiency theorems for a class of target functions rather than a derivation that real attention heads must align. The real-model SAD evidence is suggestive but indirect, and the paper's abstract and conclusion claim more than the evidence supports. The contribution is a useful conditional result plus a candidate empirical signature, rather than the strong justification claimed.

major comments (3)
  1. [Appendix B.2.1 / Theorems 1–3] The teacher in Appendix B.2.1 is defined as Ω_T = α(Σ_X^{-1} x1)(Σ_Y^{-1} y1)^⊤. By construction, its nonzero singular vectors are proportional to the whitened feature directions. Theorem 1 proves that the unique population minimizer of the cross-entropy loss is exactly this Ω_T; Theorems 2 and 3 then read off alignment exactly under isotropy and approximately near it. This is a consistency/recovery result for a teacher that was built to align; it does not derive alignment from 'tokens are sums of features' and 'heads attend to features'. The Section 4 claim that 'SVF alignment provably occurs under fairly general conditions' is therefore overstated. Please restate Theorems 1--3 explicitly as sufficiency results for a bilinear teacher, and discuss what would need to be true of real attention logits (e.g., approximate bilinearity in whitened feature-presence statistics) for the conclusion
  2. [Section 5, Figs. 6–8] Sparse attention decomposition is a plausible necessary condition for SVF alignment, but it is not unique to alignment. The rotation control in Figs. 6(b) and 7 rules out low rank and the spectrum, but the authors themselves acknowledge in Section 6 that anisotropy/cone directions can create spurious structure in the SVD basis. The empirical test is also narrow: a single IOI prompt and a preselected set of circuit heads (Appendix F, Table 2). Thus the statement that SVF alignment has 'strong empirical justification' in real models is not supported by the presented evidence. To make this load-bearing claim convincing, I suggest (i) measuring SAD on multiple prompts and on heads outside the hand-selected circuit, (ii) constructing null token distributions matched in anisotropy but without feature-aligned singular vectors, and (iii) connecting SAD to actual features by causal interventions
  3. [Appendix B.3, Eq. (16)] Theorem 4 is conditional in two ways that are not reflected in the main text. The objective in Eq. (16) contains an explicit interference penalty (λ/2)(y_2^⊤ y_1)^2; the theorem does not show that the cross-entropy loss alone induces orthogonality. Second, the conclusion that y_2^⊤ y_1 = 0 requires the existence of a feasible (x′, y′) that simultaneously attains the infimum of the cross-entropy terms and satisfies y′_2^⊤ y_1 = 0; no such existence argument is given. The theorem also assumes the first feature pair is already aligned with singular vectors of Ω. The main-text bullet 'Features orthogonalize to minimize interference' should therefore be stated as a property of a regularized objective under an existence assumption, not as a demonstrated outcome of the unregularized training dynamics. The toy experiments do show orthogonalization, so the intended phenomenon is plausible; the th
minor comments (6)
  1. [Abstract / Appendix A.1 / Appendix F.1] There are typos: 'identifysparse' in the abstract, 'Alighment' in Appendix A.1, and 'Identifiation' in Appendix F.1.
  2. [Eqs. (1)–(2)] The bracketed terms in Eq. (2) split the feature strengths f_i and f_j across the two factors, making the 'feature pair matched to a singular vector pair' statement harder to verify. Including f_i^{(r)} and f_j^{(s)} explicitly inside the respective brackets would improve readability.
  3. [Figure 3 caption] The caption is ambiguous: it refers to 'Above' and 'Below' and 'top' and 'bottom' without clearly indicating which panels show singular-vector evolution, which show feature evolution, and which show their alignment.
  4. [References] Franco & Crovella (2025) is listed with 'doi: TBD' and 'URL TBD'; these should be completed before publication.
  5. [Appendix D, Figure 18] The captions in Figure 18 list feature ranges such as 'f6 f11' and 'f7 f13' that do not obviously match the pair definitions used in the main text (e.g., (i, i+20) or the linearly declining logit schedule). Please clarify the indexing.
  6. [Reproducibility] The paper does not state whether training code and data are available. Given the empirical dependence on exact sweep details and the Pythia/GPT-2 prompt, a code/data availability statement would strengthen reproducibility.

Circularity Check

1 steps flagged

Theorems 1-3 inherit 'alignment' from the teacher matrix, which is defined to have whitened-feature singular vectors; the real-model SAD test adds independent but non-unique evidence.

specific steps
  1. self definitional [Appendix B.2.1 (Setting), Theorems 1-2 (B.2.3)]
    "Fix a scale α>0 and define the teacher matrix Ω_T := αuv^⊤ = α(Σ_X^{-1} x1),(Σ_Y^{-1} y1)^⊤. ... Then the unique population minimizer satisfies Ω⋆ = Ω_T = α/(ab) x1y1^⊤ ... u1(Ω⋆)∥x1, v1(Ω⋆)∥y1."

    Ω_T is defined as the outer product of the whitened feature directions Σ_X^{-1}x1 and Σ_Y^{-1}y1. For a rank-1 outer product, the singular vectors are exactly its two constituent vectors, so 'SVF alignment' of the teacher is true by construction. Theorem 1 shows the nontrivial fact that the population CE minimizer uniquely equals Ω_T; Theorem 2 then substitutes isotropy (Σ_X=aI, Σ_Y=bI) to conclude Ω⋆=(α/ab)x1y1^⊤, hence u1∥x1, v1∥y1. Thus the theoretical prediction of alignment is a corollary of the teacher's defining property, not a derivation from feature geometry or from generic attention targets. Theorem 3 only quantifies how much this constructed alignment degrades under anisotropy.

full rationale

The central circularity is localized to the teacher construction. The paper claims to show 'theoretically that such alignment is expected under a range of conditions,' but the conditions in B.2.1 build the alignment into the target: Ω_T is explicitly chosen as the outer product of the whitened feature directions, so its singular vectors align with features by definition. Theorems 1-2 then prove that minimizing cross-entropy recovers this teacher; this is a consistency / identifiability result, not an independent derivation of SVF alignment. The toy experiments are a legitimate simulation of this same construction, so they validate the training dynamics rather than independently testing the alignment hypothesis. The real-model evidence via SAD is more independent: it is derived from the alignment hypothesis, tested on an external model (Pythia/GPT-2) with a rotation control that rules out low-rank/spectrum explanations, and it uses a pre-existing circuit (Tigges et al.). However, SAD is not uniquely diagnostic of feature alignment, as the paper itself acknowledges with the cone-directions/anisotropy caveat. The SAD and relative-attention tools come from the authors' own prior work, but they are re-derived here and tested in the toy model, so the self-citation is not itself load-bearing. On balance, the theoretical 'prediction' of alignment reduces by construction to the teacher's definition, giving a partial circularity (score 6), while the empirical SAD component preserves some independent content.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The central claim rests on the linear-representation/additive-token assumption and on a teacher whose logits are bilinear in feature strengths. The theory then adds invertibility and near-isotropy, which are reasonable in the toy model but are not established for real models. The toy target-logit schedules and the Theorem 4 interference penalty are hand-selected elements that push the result in the desired direction.

free parameters (3)
  • Toy target-logit schedule Tij = T01=1; Ti,i+20=26-i for i=0..19; linearly declining schedules elsewhere
    Hand-selected by the authors to impose an attention hierarchy. The resulting singular-value ordering in Figure 2(b) mirrors this schedule, so part of the observed alignment is designed in.
  • Attention-loss weight lambda = 4 (default; swept 0.0625-16)
    Controls the balance between reconstruction and attention losses. Robustness is shown across values, but the value determines whether attention pressure is strong enough to induce alignment.
  • Theorem 4 interference penalty lambda/2 * (y2^T y1)^2 = unspecified lambda > 0
    Added explicitly to the objective so that the proof has a term pushing y2 orthogonal to y1. The theorem's conclusion is conditional on this penalty being present.
axioms (6)
  • domain assumption Features are linearly represented and tokens are sums of features: r = W f, s_j = W f_j (Section 3).
    The entire framework, including the QK logit r^T Omega s, depends on this additive linear representation.
  • domain assumption Teacher attention logits are bilinear in feature strengths: ell_T = sum_ij T_ij f_i^r f_j^s (Section 3).
    The target attention pattern is assumed to depend only on pairwise feature co-occurrence, which is what lets Omega act as a feature-pair detector.
  • domain assumption Feature covariance matrices Sigma_X and Sigma_Y are invertible, and features are at least near-isotropic for Theorems 2-3 (Appendix B.2.1-B.2.3).
    Invertibility is used to prove uniqueness of the minimizer; near-isotropy is what makes whitened features coincide with raw features.
  • standard math Softmax equality implies logits differ by an additive constant (Lemma 1, Appendix B.2.2).
    Standard property of the softmax map, used in the proof of Theorem 1.
  • standard math Wedin's sin-theta theorem and Weyl's inequality for singular values (Appendix B.2.3, Theorem 3 proof).
    Classical matrix perturbation results used to bound singular-vector deviation from features.
  • ad hoc to paper The Theorem 4 objective includes an explicit interference penalty lambda/2 (y2^T y1)^2 (Equation 16).
    This penalty is not part of the earlier toy loss; it is introduced specifically to make orthogonalization provable.

pith-pipeline@v1.3.0-alltime-deepseek · 26900 in / 16022 out tokens · 150018 ms · 2026-08-02T23:29:37.864754+00:00 · methodology

0 comments
read the original abstract

Identifying feature representations in language models is a central task in mechanistic interpretability. Several recent studies have made the observation that feature representations can be inferred in some cases from singular vectors of attention matrices. However, sound justification for this phenomenon is lacking. In this paper we address that question, asking: why and when do singular vectors align with features? First, we demonstrate that singular vectors robustly align with features in a model where features can be directly observed. We then show theoretically that such alignment is expected under a range of conditions. We close by asking how, operationally, alignment may be recognized in real models where feature representations are not directly observable. We identify sparse attention decomposition as a testable prediction of alignment, and show evidence that it emerges in real models in a manner consistent with predictions. Together these results suggest that alignment of singular vectors with features can be a sound and theoretically justified basis for feature identification in language models.

Figures

Figures reproduced from arXiv: 2602.13524 by Carson Loughridge, Gabriel Franco, Mark Crovella.

Figure 1
Figure 1. Figure 1: The geometry of features as illustrated via cosine similar￾ities. (a) Without the attention head, features arrange isotropically. (b) With 20 features of which w0, w1 are of interest, features of interest orthogonalize against the others. (c) With 100 features in dimension 50, and 40 of those features are of interest (20 pairs), features of interest also orthogonalize against the others. 4. Singular Vector… view at source ↗
Figure 2
Figure 2. Figure 2: Singular vectors align with features (shown by green boxes). Cosine similarities of singular vectors and features, and magnitudes of singular values. (a) 20 Features of which w0, w1 are of interest; w0 aligns only with u0 and w1 aligns only with v0. (b) 100 features of which w0 . . . w39 are of interest. For clarity, only an initial subset of features is shown; full figures are in Appendix A. ie, w1 is app… view at source ↗
Figure 3
Figure 3. Figure 3: Both singular vectors and features evolve during train￾ing, and alignment occurs for highest-logit features first. Above: Cosine similarities showing evolution of singular vectors (top) and features (bottom). Below: Cosine similarities showing evolution of alignment of singular vectors with features. arises consistently over variations in relative loss weight λ, number of features N, context length m, head… view at source ↗
Figure 4
Figure 4. Figure 4: Relative attention decomposition is sparse when a single feature pair is present. Top: Early in training; Bottom: Late in training. if attention is sparsely decomposed in the SVD basis, then only a small set of dimensions in the tokens contain features important for the attention computation. This ‘reduces the search space’ of causal features, a concept that is important in (Merullo et al., 2024; Franco & … view at source ↗
Figure 5
Figure 5. Figure 5: Sparse attention decomposition identifies feature pres￾ence. Top: Decomposition of relative attention across all 10 sin￾gular vectors for five token pairs (r, s). Bottom: Feature strength (f (r) i f (s) i+4) for the four corresponding feature pairs wi, wi+4. 0 5000 10000 Training Step 0.1 0.2 0.3 0.4 0.5 Decomp. Sparseness No Feats. Two Feats. One Feat. 0 50 100 Checkpoint/1000 0.1 0.2 0.3 0.4 Decomp. Spar… view at source ↗
Figure 8
Figure 8. Figure 8: No. of singular vectors to reconstruct relative attention. (a) GPT-2, after training; (b) Pythia, during training, 95% CI. matrix spectrum or rank [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Singular vectors align with features. For Ω = UΣV ⊤, cosine similarities of W and U, W and V , and spectrum (diagonal of Σ). This is an expanded view of [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: SVF Alignment is robust to feature probability. 0 5000 10000 15000 20000 25000 30000 35000 40000 Training Step 0.0 0.2 0.4 0.6 0.8 1.0 Average Alignment (over 8 SV-feature pairs) SV-Feature Alignment During Training Feature Prob S=1.000 S=0.521 S=0.271 S=0.141 S=0.074 S=0.038 S=0.020 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: SVF Alignment arises earlier when features occur more frequently [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: SVF Alignment is robust to λ. U0 f0 U1 f1 U2 f2 U3 f3 V0 f4 V1 f5 V2 f6 V3 f7 Singular Vector Feature Alignment n=10 n=15 n=20 n=25 n=30 Number of Features 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.99 0.99 1.00 1.00 0.99 0.99 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 SV-Feature Alignment vs n_features ( =4) (sparsity l… view at source ↗
Figure 13
Figure 13. Figure 13: SVF Alignment is robust to varying number of features (compared to D = 10). 14 [PITH_FULL_IMAGE:figures/full_fig_p014_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: SVF Alignment is robust to varying head dimension (compared to D = 10). 6, and 4 [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: SVF Alignment is robust to varying context length. U0 f0 U1 f1 U2 f2 U3 f3 V0 f4 V1 f5 V2 f6 V3 f7 Singular Vector Feature Alignment Mean Std Dev 0.999 0.997 0.998 0.999 0.999 0.997 0.998 0.999 0.001 0.005 0.004 0.001 0.001 0.005 0.004 0.001 Mean and Std Dev of SV-Feature Alignment ( =4, n_features=20) (across 5 seeds, sparsity level S=0.521) 0.0 0.2 0.4 0.6 0.8 1.0 Value [PITH_FULL_IMAGE:figures/full_fi… view at source ↗
Figure 16
Figure 16. Figure 16: SVF Alignment is robust across a set of random seeds. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: WW⊤ matrices showing isotropic arrangement of features (a) without attention, and (b) when two features are of interest to the head. Rearranging yields the finite-λ bound λ 2 (y λ⊤ 2 y1) 2 ≤ LCE(x ′ 2 , y′ 2 ) − LCE(x λ 2 , yλ 2 ) ≤ LCE(x ′ 2 , y′ 2 ) − inf x2,y2 LCE(x2, y2) which establishes the result. C. Isotropy Here we show evidence of the isotropy of features in the toy model. We consider two cases,… view at source ↗
Figure 18
Figure 18. Figure 18: When there are more features of interest than head capacity, features are superposed primarily on the smallest singular vector, and most singular vectors still map to a single feature. Absolute cosine similarity of features and singular vectors. We show absolute value of cosine similarity for clarity due to sign ambiguity of SVD. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: When features of interest are not present, attention is not sparsely decomposable. Top: Token pairs without features of interest, early in training; Lower: Late in training. Compare to [PITH_FULL_IMAGE:figures/full_fig_p025_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Sparse decomposition in Pythia generally increases during training. Each plot shows decomposition of relative attention at the start and end of training. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_20.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Complexity-Guided Component-wise Initialization for Language Model Pretraining

    cs.CL 2026-07 conditional novelty 5.5

    Coarse component-wise spectral matching of pretrained GPT-2 weights changes structure but does not beat standard initialization, while direct weight reuse remains competitive.

Reference graph

Works this paper leans on

6 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [2]

    URL https://arxiv.org/abs/2308.0 9124. Horn, R. A. and Johnson, C. R.Matrix Analysis. Cambridge University Press, 2 edition, 2012. Huben, R., Cunningham, H., Smith, L. R., Ewart, A., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. InThe Twelfth International Conference on Learning Representations, 2024. URL https...

    Pith/arXiv arXiv 2012

  2. [4]

    In particular, if there exists a feasible(x′ 2, y′

    with ∥x′ 2∥=∥y ′ 2∥= 1. In particular, if there exists a feasible(x′ 2, y′

  3. [5]

    achieving minimal CE while also havingy ′⊤ 2 y1 = 0, then the optimizer must satisfyy λ⊤ 2 y1 = 0 Proof.Fix anyλ >0. For any(x 2, y2)we can decompose the objective as J(x 2, y2) = CE(δ⋆ A, δA(x2, y2)) + CE(δ⋆ B, δB(x2, y2))| {z } =:L CE(x2,y2) + λ 2 (y⊤ 2 y1)2 Let(x λ 2 , yλ 2 )be a global minimizer. Now choose any comparison pair(x ′ 2, y′ 2)withy ′⊤ 2 y...

  4. [6]

    + λ 2 (y′⊤ 2 y1)2 | {z } =0 . 22 Singular Vectors of Attention Heads Align with Features 0 1 2 3 4 5 6 7 8 9 0123456789 0 1 2 0 1 2 3 4 5 6 7 8 9 0123456789 0.0 0.5 1.0 1.5 2.0 (a) (b) Figure 17.W W⊤ matrices showing isotropic arrangement of features (a) without attention, and (b) when two features are of interest to the head. Rearranging yields the finit...

    2024

  5. [1972]

    attend iff(x 1, y1)is present

    URL https://link.springer.com/ar ticle/10.1007/BF01932678. Pan, X., Philip, A., Xie, Z., and Schwartz, O. Dissecting query-key interaction in vision transformers. InProceed- ings of NeurIPS, 2024. Park, K., Choe, Y . J., and Veitch, V . The linear represen- tation hypothesis and the geometry of large language models. InCausal Representation Learning Works...

    Pith/arXiv arXiv 2024

  6. [2024]

    Franco, G

    URL https://arxiv.org/abs/2410.0 0340. Franco, G. and Crovella, M. Pinpointing attention-causal communication in language models. InProceedings of NeurIPS, San Diego, CA, 2025. doi: TBD. URLTBD. Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders, 2024. URL ...

    Pith/arXiv arXiv 2025