Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces layer collapse and claims that residual self-attention networks with small weights are always well approximated by a single-layer network.

desk verdict Layer collapse is a real idea worth engaging with, but the main theorem is not proven: the proof relies on a false norm inequality and on bounded-input assumptions that never appear in the theorem statement. read the letter →

arxiv 2505.16284 v1 pith:5P4JL4V3 submitted 2025-05-22 cs.LG

Josh Alman , Zhao Song This is my paper · ORCID
classification cs.LG
keywords layercollapserankself-attentionnetworkresidualconnectionssmallweightstransformerexpressivitysoftmaxattentioncomplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that small weights, not skip connections, are what determine whether a deep self-attention network is genuinely deep in expressive terms. It defines layer collapse: an $L$-layer residual self-attention network collapses when a one-layer network approximates its output on every input within error $O(\eta)\|X\|_\infty$, where $\eta$ bounds every attention weight entry in $\ell_\infty$ norm. The paper argues this collapse is unavoidable whenever $\eta$ is small, even if every layer keeps its residual connection. If true, this overturns the standard reading of earlier rank-collapse results, which were often cited as showing that residual connections are what protect transformer expressivity. It also sharpens the cost story for transformers: large weights are the price of depth, and hence of avoiding layer collapse.

What carries the argument

The load-bearing object is the Res function, which subtracts from each row of a token-representation matrix the best constant row and so measures how far the matrix is from having identical rows across all tokens; identical rows are the hallmark of rank collapse. The proof shows that a small-weight attention layer shrinks Res by a factor proportional to the weight bound (Lemma 5.1), because softmax shift-invariance makes attention depend only on Res and a balancedness condition keeps the softmax close to uniform. A perturbation lemma (Lemma 5.2, with a multi-head version in Lemma C.1) then shows that removing such a layer shifts the next layer's output by only $O(\eta)\|X\|_\infty$; a per-layer Lipschitz bound keeps this error from exploding as the remaining layers are processed. The recursion $\epsilon_\ell = 2\eta\phi_0(1+H\eta)^\ell$ controls how large layer outputs can grow, and the proof needs this quantity to stay below 1.

What would settle it

Take the 2-by-2 all-ones matrix $A$; $\|A^2\|_\infty = 2$ while $\|A\|_\infty^2 = 1$, so the matrix norm inequality used in the proof fails as stated. A direct test of the theorem: numerically search two-layer residual self-attention networks with all weight entries bounded by a small $\eta$ and inputs $X$ of increasing $\|X\|_\infty$, and check whether any instance violates $\|S(X)-S'(X)\|_\infty \le C\eta\|X\|_\infty$ for the best single-layer approximation; one such instance would refute the universal claim.

Watch

Extended reading notes

Core claim

The discovery the paper argues for is Theorem 5.3: if every weight matrix of every attention head of a residual self-attention network has entries bounded by $\eta$ in $\ell_\infty$ norm, then there exists a one-layer residual self-attention network $S'$ such that $\|S(X)-S'(X)\|_\infty \le O(\eta)\|X\|_\infty$ for every input $X$. The mechanism is that each low-weight attention layer behaves like a small perturbation of the identity: the attention output is close to a matrix with identical rows across tokens, so it can be folded into the residual stream without changing later layers by more than $O(\eta)$ times the input scale. Repeating this layer-removal $L-1$ times leaves a single-layer network. The paper presents this as layer collapse, a new form of representational collapse distinct from, but inspired by, rank collapse.

Load-bearing premise

The proof assumes that the largest-entry norm of a product of matrices is at most the product of their largest-entry norms, which is not true in general because matrix size enters, and it also assumes every layer's input stays small enough for a per-layer error parameter to stay below 1; the theorem claims to cover all inputs without either assumption.

Editorial extensions

If this is right

  • If Theorem 5.3 is correct, an $L$-layer residual self-attention network with all weight entries bounded by a small $\eta$ is expressively no stronger than a single-layer network up to error $O(\eta)\|X\|_\infty$.
  • The standard conclusion from earlier rank-collapse work—that skip connections are what save transformer expressivity—would need revision; on this account, large weights are the decisive ingredient.
  • For small-weight transformers, the quadratic-time barrier for attention is not an implementation detail: depth cannot substitute for large weights, so expressive transformers must either pay the quadratic cost or use large entries.
  • The argument extends to networks with MLP layers (Remark D.1), so the layer-collapse phenomenon is not an artifact of pure self-attention stacks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper, this suggests a quantitative notion of effective depth: the smallest number of layers needed to approximate a network's output, which could be measured by probing how much each added layer actually changes the function.
  • A testable extension is that a transformer initialized with very small weights should show almost no performance gain as depth grows, while the same depth with larger-norm initializations should show clear gains.
  • If the norm inequality used in the proof is repaired with its missing dimension factor, the bound $O(\eta)\|X\|_\infty$ may acquire a polynomial dependence on sequence length or hidden dimension, so the practically relevant small-weight regime could be narrower than the theorem's statement suggests.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the notion of "layer collapse" for residual self-attention networks: an L-layer network is said to have layer collapse if it can be approximated, on every input, by a one-layer self-attention network. The main result (Theorem 5.3) claims that if all attention weight entries are bounded in absolute value by η, then for every input X there exists a one-layer network S′ with ‖S(X)−S′(X)‖∞ ≤ O(η)‖X‖∞. The authors interpret this as showing that skip connections do not by themselves prevent the representational weakness of small-weight transformers, and that large weights are needed for depth to add expressive power. The proof builds on the rank-collapse framework of Dong, Cordonnier, and Loukas (DCL21) and introduces perturbation bounds for softmax and the Res operator. The central technical claim, however, depends on several steps that appear to be incorrect or insufficiently justified.

Significance. If the main theorem were correct, the paper would substantially strengthen the interpretation of DCL21: it would show that residual connections are not sufficient to preserve representational depth under small weights, and it would connect the expressivity question to the line of work on fast attention algorithms for bounded weights. The definition of layer collapse is a natural and potentially useful strengthening of rank collapse, and the single-head perturbation lemmas (Section 4) are clean and likely reusable. On the other hand, the paper does not provide machine-checked proofs or code, and the central theorem is not supported by the current proof, as detailed below. The significance of the paper therefore rests on whether the proof can be repaired.

major comments (4)
  1. [§3.1, Fact 3.3; §5.1, Eq. (8)] Fact 3.3 asserts that ‖AB‖∞ ≤ ‖A‖∞‖B‖∞ for the entrywise maximum norm. This is false: for A = [1,1] and B = [1,1]^T, the product AB has entries of magnitude 2, while ‖A‖∞‖B‖∞ = 1. The correct bound requires a dimension factor. This false inequality is used in Lemma 5.1, Eq. (8), to bound ‖softm(r)^T R Wv‖∞ ≤ ‖R‖∞‖Wv‖∞, and the same Fact is used in Lemmas D.5 and D.6 in the appendix. With the dimension factor restored, the constant K in Lemma 5.1 becomes at least (e^θ − 1)·d·‖Wv‖∞ instead of (e^θ − 1)‖Wv‖∞. Consequently, under the paper's own proof, the error in Theorem 5.3 would be O(dη)‖X‖∞, not O(η)‖X‖∞. Since the O(η) rate is the central quantitative claim, this is a load-bearing error.
  2. [Theorem 5.3; Lemma C.2; Remark D.2] Theorem 5.3 is stated for every X ∈ R^{n×d}, but the proof relies on Lemma C.2, whose hypotheses include ǫℓ < 1 and ‖XℓWv‖∞ ≤ 1. Remark D.2 shows that these conditions force either the input norm φ0 to be exponentially small in L or the weight bound η to satisfy η ≤ 1/(φ0HL). Neither of these restrictions appears in the statement of Theorem 5.3. Thus the proof cannot be instantiated for inputs of large norm, and the universal quantification over all X is not supported by the argument in the appendix.
  3. [Appendix D.1] The proof of Theorem 5.3 in Appendix D.1 does not construct the promised one-layer network S′. The objects B^{ℓ0}_ℓ are defined piecewise with a residual structure that is not a single-layer self-attention network, and the final inequality compares two networks with different residual connections rather than approximating the original network by a one-layer network. Moreover, under the parameter choices in Remark D.2, δ = O(1) and C = O(η), so the displayed bound ‖softmv_L(B^0_L) − softmv_L(B^L_L)‖∞ ≤ (C^L + ⋯ + 1)δ is O(1) (or, with the extra factor, O(1)(1+O(η))) and does not scale as O(η)‖X‖∞. The claimed rate in Theorem 5.3 therefore does not follow from the appendix.
  4. [Lemma D.4; Lemma D.5] Lemma D.4 as stated is false. For example, take a = (100, 0) and b = (0.1, 0). Then softm(a+b) ≈ (1, 0) while softm(b) ≈ (0.525, 0.475), so ‖softm(a+b) − softm(b)‖∞ ≈ 0.47, which exceeds 4‖b‖∞ = 0.4. The intended statement is presumably ‖softm(a+b) − softm(a)‖∞ ≤ 4‖b‖∞, which is the inequality needed in Lemma D.5. Since Lemma D.5 provides the layer Lipschitz constant C used in the proof of Theorem 5.3, this error also affects the central bound.
minor comments (5)
  1. [§1, Eq. (1)] The display "O(β) 3L − 1 2" is malformed; it should read O(β)^{(3^L − 1)/2} to match the DCL21 result.
  2. [§1] The word "Roamdap" appears where "Roadmap" is intended.
  3. [§5.3, proof sketch] The proof sketch says Lemma C.1 bounds the error of the second layer by O(η·ǫ0), but in the appendix ǫ0 = 3g(2Hǫ) already contains the factor coming from ǫ = O(K‖Res(X)‖∞); the displayed product appears to introduce an extra factor of η.
  4. [Appendix D.1] The notation is inconsistent: "X^{ℓ0}_ℓ = B^{ℓ0}_ℓ" is written before B^{ℓ0}_ℓ is defined, and the piecewise definition of B^{ℓ0}_ℓ is not typeset unambiguously.
  5. [Definition A.2; Theorem 5.3] Definition A.2 introduces parameters φ0 and ǫℓ, but Theorem 5.3 does not mention these parameters; the mismatch between the theorem statement and the appendix parameterization should be resolved.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the layer-collapse theorem is derived from the paper's own perturbation lemmas with no fitted parameter or conclusion reused as an assumption.

full rationale

The paper's central result (Theorem 5.3) is an existence and approximation statement. Its proof chain is: Lemma 5.1 bounds Res(SAtt(X)) by K · Res(X) using softmax shift-invariance and the θ-balance condition; Lemma 5.2 and its multi-head version Lemma C.1 propagate this to a perturbation bound between the original network and the network with a layer removed; the appendix then iterates layer deletion and uses Lipschitz lemmas (D.5, D.6, D.7). None of these steps fits a parameter to the quantity being predicted, and none defines layer collapse in terms of the theorem's conclusion. The self-citations to AS23/AS24 motivate the 'quadratic time is unavoidable' framing, but those are independent prior results that are not used to set constants or to prove Theorem 5.3. The false norm inequality in Fact 3.3 and the unstated bounded-input conditions used by Lemma C.2 are serious correctness gaps in the derivation, but they are not a circular reduction: they do not make the conclusion equal to an input by construction. Accordingly, no circular step is identified.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No data fitting appears anywhere; η, φ0, ǫℓ are bounded-input parameters, not fitted constants. There are no invented physical entities. The load-bearing assumptions are the incorrect submultiplicativity of the entrywise ℓ∞ norm, the choice of β that makes the attention score matrix θ-balanced, and the bounded-input conditions that the main theorem does not state.

assumptions (3)
  • standard math Entrywise ℓ∞ matrix norm is submultiplicative (Fact 3.3).
    Invoked in Lemma 5.1 Eq. (8) and Lemma D.6; false as stated, needs a dimension factor, so the proof's norm bounds do not follow.
  • ad hoc to paper E=β Res(X)W Res(X)⊤ is θ-balanced with θ=1.
    Lemma B.2 enforces θ=1 by choosing β≤1/(‖Res(X)‖∞²η²), a normalization condition not stated in the main theorem and incompatible with the standard 1/√d scaling without dimension-dependent bounds.
  • domain assumption Inputs remain bounded with ǫℓ<1 and ‖XℓWv‖∞≤1 for all layers.
    Lemma C.2 requires this, but Theorem 5.3 claims the bound for every X with no norm restriction; Remark D.2 discusses regimes instead of resolving the mismatch.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse." pith.science (2026). https://pith.science/paper/5P4JL4V3

@misc{pith2026250516284,
  author       = {Pith},
  title        = {Pith review of: Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5P4JL4V3}},
  note         = {Machine review of arXiv:2505.16284}
}
read the original abstract

Attention mechanisms lie at the heart of modern large language models (LLMs). Straightforward algorithms for forward and backward (gradient) computation take quadratic time, and a line of work initiated by [Alman and Song NeurIPS 2023] and [Alman and Song NeurIPS 2024] has shown that quadratic time is necessary unless the model weights are small, in which case almost linear time algorithms are possible. In this paper, we show that large weights are necessary to avoid a strong preclusion to representational strength we call layer collapse, which means that the entire network can be approximated well by a network with only a single layer. Thus, the quadratic running time of attention is unavoidable for expressive transformers. The notion of layer collapse that we introduce is a variant on the notion of rank collapse from the work of [Dong, Cordonnier, and Loukas ICML 2021]. They showed that in Self Attention Networks with small weights and with skip connections, rank collapse must occur. This is typically interpreted as justifying the necessity of skip connections in expressive networks. However, our result shows that even with skip connections, if the weights are small, then layer collapse still occurs. Thus, only large weights, and not skip connections, can prevent these representational weaknesses.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Rank survival in Transformer blocks is governed by a branch-to-skip ratio law (βα^M√L), a mean-spike coherence c_ℓ=E[σ]²/E[σ²], and a Marchenko–Pastur width threshold m/d=1/p(σ).

  2. Attention's forward pass and Frank-Wolfe

    math.OC 2025-08 conditional novelty 6.0 of 10

    Hardmax self-attention is shown to be a Frank-Wolfe iteration; with positive-definite key-query it converges to Voronoi-cell vertices, and a Markov-chain version of soft attention is metastable there for exponential-i...

Reference graph

Works this paper leans on

66 extracted references · 26 canonical work pages · cited by 2 Pith papers

  1. [3]

    Stable video diffusion: Scaling latent video diffusion models t o large datasets

    [BDK+23] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel M endelevitch, Maciej Kil- ian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models t o large datasets. arXiv preprint arXiv:2311.15127,

    arXiv

  2. [5]

    On the dangers of stochastic parrots: Can langu age models be too big? In Proceedings of the 2021 ACM conference on fairness, accounta bility, and trans- parency, pages 610–623,

    [BGMMS21] Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can langu age models be too big? In Proceedings of the 2021 ACM conference on fairness, accounta bility, and trans- parency, pages 610–623,

    work page 2021

  3. [7]

    Longf ormer: The long-document transformer

    [BPC20] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longf ormer: The long-document transformer. arXiv preprint arXiv:2004.05150 ,

    arXiv 2004

  4. [9]

    High-order matching for one-step shor tcut diffusion models

    [CGL+25] Bo Chen, Chengyue Gong, Xiaoyu Li, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song, and Mingda Wan. High-order matching for one-step shor tcut diffusion models. arXiv preprint arXiv:2502.00688 ,

    arXiv

  5. [10]

    Generating long se- quences with sparse transformers

    [CGRS19] Rewon Child, Scott Gray, Alec Radford, and Ilya Sut skever. Generating long se- quences with sparse transformers. arXiv preprint arXiv:1904.10509 ,

    arXiv 1904

  6. [12]

    Fast gradient computation for rope attention in almost line ar time

    [CHL+24b] Yifang Chen, Jiayan Huo, Xiaoyu Li, Yingyu Liang, Zhenm ei Shi, and Zhao Song. Fast gradient computation for rope attention in almost line ar time. arXiv preprint arXiv:2412.17316,

    arXiv

  7. [13]

    Kernel den- sity estimation through density constrained near neighbor search

    [CKNS20] Moses Charikar, Michael Kapralov, Navid Nouri, an d Paris Siminelakis. Kernel den- sity estimation through density constrained near neighbor search. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages 172–183. IEEE,

    work page 2020

  8. [15]

    Hsr-enhanced sparse attention acceleration

    [CLS+24] Bo Chen, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, and Zhao Song. Hsr-enhanced sparse attention acceleration. arXiv preprint arXiv:2410.10165 ,

    arXiv

Show all 66 references
  1. [16]

    Bert: Pre- training of deep bidirectional transformers for language understanding

    [DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kris tina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,

    arXiv

  2. [18]

    Faster robus t tensor power method for arbitrary order

    [DSY23] Yichuan Deng, Zhao Song, and Junze Yin. Faster robus t tensor power method for arbitrary order. arXiv preprint arXiv:2306.00406 ,

    arXiv

  3. [19]

    Superiori ty of softmax: Unveiling the performance edge over linear attention

    [DSZ23] Yichuan Deng, Zhao Song, and Tianyi Zhou. Superiori ty of softmax: Unveiling the performance edge over linear attention. arXiv preprint arXiv:2310.11685 ,

  4. [20]

    One step diffusion via shortcut models

    [FHLA24] Kevin Frans, Danijar Hafner, Sergey Levine, and Pi eter Abbeel. One step diffusion via shortcut models. arXiv preprint arXiv:2410.12557 ,

    arXiv

  5. [21]

    Sagn: semantic adaptive gr aph network for skeleton-based human action recognition

    24 [FLZ+21] Ziwang Fu, Feng Liu, Jiahao Zhang, Hanyang Wang, Chengyi Yang, Qing Xu, Jiayin Qi, Xiangling Fu, and Aimin Zhou. Sagn: semantic adaptive gr aph network for skeleton-based human action recognition. In Proceedings of the 2021 International Conference on Multimedia Re...

    2021

  6. [22]

    Can you count to nine? a human evaluation bench mark for counting limits in modern text-to-video models

    [GHH+25] Xuyang Guo, Zekai Huang, Jiayan Huo, Yingyu Liang, Zhenm ei Shi, Zhao Song, and Jiahao Zhang. Can you count to nine? a human evaluation bench mark for counting limits in modern text-to-video models. arXiv preprint arXiv:2504.04051 ,

    arXiv

  7. [23]

    T2vphysbench: A first-principles benchmark for physical co nsistency in text-to-video generation

    [GHS+25a] Xuyang Guo, Jiayan Huo, Zhenmei Shi, Zhao Song, Jiahao Z hang, and Jiale Zhao. T2vphysbench: A first-principles benchmark for physical co nsistency in text-to-video generation. arXiv preprint arXiv:2505.00337 ,

    arXiv

  8. [24]

    T2vtextbench: A human evaluation benchmark for textual control in video generation models

    [GHS+25b] Xuyang Guo, Jiayan Huo, Zhenmei Shi, Zhao Song, Jiahao Z hang, and Jiale Zhao. T2vtextbench: A human evaluation benchmark for textual control in video generation models. arXiv preprint arXiv:2505.04946 ,

    arXiv

  9. [25]

    Sub- quadratic algorithms and hardness for attention with any te mperature

    [GHS+25c] Shreya Gupta, Boyang Huang, Barna Saha, Yinzhan Xu, and Christopher Ye. Sub- quadratic algorithms and hardness for attention with any te mperature. In arXiv preprint arXiv:2505.14840,

    arXiv

  10. [26]

    Llama-adapter v2: Parameter- efficient visual instruction model

    [GHZ+23] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Gen g, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter- efficient visual instruction model. arXiv preprint arXiv:2304.15010 ,

    arXiv

  11. [27]

    On computational limits of flowar models: Express ivity and efficiency

    [GKL+25] Chengyue Gong, Yekun Ke, Xiaoyu Li, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, and Zhao Song. On computational limits of flowar models: Express ivity and efficiency. arXiv preprint arXiv:2502.16490 ,

  12. [28]

    An over- parameterized exponential regression

    [GMS23] Yeqi Gao, Sridhar Mahadevan, and Zhao Song. An over- parameterized exponential regression. arXiv preprint arXiv:2303.16504 ,

    arXiv

  13. [29]

    Fas t quantum algorithm for attention computation

    [GSYZ23] Yeqi Gao, Zhao Song, Xin Yang, and Ruizhe Zhang. Fas t quantum algorithm for attention computation. arXiv preprint arXiv:2307.08045 ,

  14. [30]

    Differe ntially private attention computation

    25 [GSYZ24a] Yeqi Gao, Zhao Song, Xin Yang, and Yufa Zhou. Differe ntially private attention computation. In Neurips Safe Generative AI Workshop 2024 ,

    2024

  15. [31]

    Contranorm: A contrastive learning perspective on oversmoothing and beyond

    [GWDW23] Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning perspective on oversmoothing and beyond. arXiv preprint arXiv:2303.06562,

    arXiv

  16. [32]

    Generalized probab ilistic attention mechanism in transformers

    [HC24] DongNyeong Heo and Heeyoul Choi. Generalized probab ilistic attention mechanism in transformers. arXiv preprint arXiv:2410.15578 ,

    arXiv

  17. [33]

    Com- putational limits of low-rank adaptation (lora) for transf ormer-based models

    [HSK+24] Jerry Yao-Chieh Hu, Maojiang Su, En-Jui Kuo, Zhao Song, a nd Han Liu. Com- putational limits of low-rank adaptation (lora) for transf ormer-based models. arXiv preprint arXiv:2406.03136,

    arXiv

  18. [34]

    On statistical rates and provably efficient criteria of latent diffusion transforme rs (dits)

    [HWSL24] Jerry Yao-Chieh Hu, Weimin Wu, Zhao Song, and Han Li u. On statistical rates and provably efficient criteria of latent diffusion transforme rs (dits). arXiv preprint arXiv:2407.01079,

    arXiv

  19. [36]

    Sparse finetuning for inference acceleration of large langu age models

    [KKF+23] Eldar Kurtic, Denis Kuznedelev, Elias Frantar, Michael Goin, and Dan Alistarh. Sparse finetuning for inference acceleration of large langu age models. arXiv preprint arXiv:2310.06927,

    arXiv

  20. [37]

    Shortened llama: A simple d epth pruning for large language models

    [KKK+24] Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castel ls, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. Shortened llama: A simple d epth pruning for large language models. arXiv preprint arXiv:2402.02834 , 11,

    arXiv

  21. [38]

    On the power of preconditioning in sparse linear regression

    [KKMR22] Jonathan A Kelner, Frederic Koehler, Raghu Meka, and Dhruv Rohatgi. On the power of preconditioning in sparse linear regression. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 550–561. IEEE,

    2021

  22. [39]

    Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp

    [KSL+22] Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hal l, Percy Liang, Christo- pher Potts, and Matei Zaharia. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp. arXiv preprint arXiv:2212.14024 ,

    arXiv

  23. [40]

    Simulation of hypergraph algorithms with looped tr ansformers

    [LLL+25] Xiaoyu Li, Yingyu Liang, Jiangxuan Long, Zhenmei Shi, Zh ao Song, and Zhen Zhuang. Simulation of hypergraph algorithms with looped tr ansformers. arXiv preprint arXiv:2501.10688,

  24. [42]

    Exploring the frontiers of softmax: Provable optimization, applications in diffusion m odel, and beyond

    [LLSS24a] Chenyang Li, Yingyu Liang, Zhenmei Shi, and Zhao S ong. Exploring the frontiers of softmax: Provable optimization, applications in diffusion m odel, and beyond. arXiv preprint arXiv:2405.03251,

  25. [43]

    A tighter complexity analysis of sparsegpt

    [LLSS24b] Xiaoyu Li, Yingyu Liang, Zhenmei Shi, and Zhao Son g. A tighter complexity analysis of sparsegpt. arXiv preprint arXiv:2408.12151 ,

    arXiv

  26. [45]

    Differential privacy mech- anisms in neural tangent kernel regression

    [LSSS24] Yingyu Liang, Zhizhou Sha, Zhenmei Shi, and Zhao So ng. Differential privacy mech- anisms in neural tangent kernel regression. arXiv preprint arXiv:2407.13621 ,

    arXiv

  27. [46]

    Differential privacy of cross- attention with provable guarantee

    [LSSZ24a] Yingyu Liang, Zhenmei Shi, Zhao Song, and Yufa Zho u. Differential privacy of cross- attention with provable guarantee. arXiv preprint arXiv:2407.14717 ,

  28. [47]

    Tensor attention train- ing: Provably efficient learning of higher-order transforme rs

    28 [LSSZ24b] Yingyu Liang, Zhenmei Shi, Zhao Song, and Yufa Zho u. Tensor attention train- ing: Provably efficient learning of higher-order transforme rs. arXiv preprint arXiv:2405.16411,

  29. [48]

    The llama 3 herd of models

    [LT24] AI @ Meta Llama Team. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  30. [49]

    Score-based gener- ative diffusion models for social recommendations

    [LZW+24] Chengyi Liu, Jiahao Zhang, Shijie Wang, Wenqi Fan, and Qing Li. Score-based gener- ative diffusion models for social recommendations. arXiv preprint arXiv:2412.15579 ,

    arXiv

  31. [50]

    Do generative video models learn physical principles from w atching videos? arXiv preprint arXiv:2501.09038,

    [MCS+25] Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini , and Robert Geirhos. Do generative video models learn physical principles from w atching videos? arXiv preprint arXiv:2501.09038,

    arXiv

  32. [51]

    Great power, great responsibility: Recom mendations for reducing energy for training language models

    [MLF+22] Joseph McDonald, Baolin Li, Nathan Frey, Devesh Tiwari, Vijay Gadepally, and Siddharth Samsi. Great power, great responsibility: Recom mendations for reducing energy for training language models. In Findings of the Association for Computational Linguistics: NAACL 2022...

    2022

  33. [52]

    Miti- detr: Object detection based on transformers with mitigatory self-attention conv ergence

    [MZW21] Wenchi Ma, Tianxiao Zhang, and Guanghui Wang. Miti- detr: Object detection based on transformers with mitigatory self-attention conv ergence. arXiv preprint arXiv:2112.13310,

    arXiv

  34. [53]

    Osnap: Faster numerical linear algebra algorithms via sparser subspace embeddings

    [NN13] Jelani Nelson and Huy L Nguyˆ en. Osnap: Faster numerical linear algebra algorithms via sparser subspace embeddings. In 2013 ieee 54th annual symposium on foundations of computer science , pages 117–126. IEEE,

    2013

  35. [55]

    Gpt-4 technical report

    [Ope23] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  36. [56]

    Retentive network: A successor to tra nsformer for large language models

    [SDH+23] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia , Jilong Xue, Jiany- ong Wang, and Furu Wei. Retentive network: A successor to tra nsformer for large language models. arXiv preprint arXiv:2307.08621 ,

    arXiv

  37. [57]

    Numerical pruning for effic ient autoregressive models

    [SSZ+24] Xuan Shen, Zhao Song, Yufa Zhou, Bo Chen, Jing Liu, Ruiyi Z hang, Ryan A Rossi, Hao Tan, Tong Yu, Xiang Chen, et al. Numerical pruning for effic ient autoregressive models. arXiv preprint arXiv:2412.12441 ,

    arXiv

  38. [58]

    Revisiting q uantum algorithms for linear regressions: Quadratic speedups without data-dependent p arameters

    [SYZ23] Zhao Song, Junze Yin, and Ruizhe Zhang. Revisiting q uantum algorithms for linear regressions: Quadratic speedups without data-dependent p arameters. arXiv preprint arXiv:2311.14823,

    arXiv

  39. [59]

    Transformer dissection: a unified un derstanding of trans- former’s attention via the lens of kernel

    [TBY+19] Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis -Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: a unified un derstanding of trans- former’s attention via the lens of kernel. arXiv preprint arXiv:1908.11775 ,

    1908 arXiv

  40. [60]

    Alignab: Pareto- optimal energy alignment for designing nature-like antibo dies

    [WXHL24] Yibo Wen, Chenwei Xu, Jerry Yao-Chieh Hu, and Han Li u. Alignab: Pareto- optimal energy alignment for designing nature-like antibo dies. arXiv preprint arXiv:2412.20984,

  41. [61]

    Ev idence-aware fake news detection with graph neural networks

    [XWL+22] Weizhi Xu, Junfei Wu, Qiang Liu, Shu Wu, and Liang Wang. Ev idence-aware fake news detection with graph neural networks. InProceedings of the ACM web conference 2022, pages 2501–2510,

    2022

  42. [62]

    Towards better multi-head attention via channel-wise sample permutation

    [YX24] Shen Yuan and Hongteng Xu. Towards better multi-head attention via channel-wise sample permutation. arXiv preprint arXiv:2410.10914 ,

    arXiv

  43. [63]

    Tra ined transformers learn linear models in-context

    [ZFB23] Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Tra ined transformers learn linear models in-context. arXiv preprint arXiv:2306.09927 ,

    arXiv

  44. [64]

    Graph unlearning with efficient partia l retraining

    [Zha24] Jiahao Zhang. Graph unlearning with efficient partia l retraining. In Companion Proceedings of the ACM on Web Conference 2024 , pages 1218–1221,

    2024

  45. [65]

    Kdeformer: Ac- celerating transformers via kernel density estimation

    [ZHDK23] Amir Zandieh, Insu Han, Majid Daliri, and Amin Karb asi. Kdeformer: Ac- celerating transformers via kernel density estimation. In ICML. arXiv preprint arXiv:2302.02451,

    arXiv

  46. [66]

    Llama-adapter: Efficient fine- tuning of language models with zero-init attention

    [ZHZ+23] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shil in Yan, Pan Lu, Hong- sheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine- tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 ,

    arXiv

  47. [2013]

    Mind the gap: a spectral analysis of rank collapse and signal propagation in transfo rmers

    [NST24] Alireza Naderi, Thiziri Nait Saada, and Jared Tanne r. Mind the gap: a spectral analysis of rank collapse and signal propagation in transfo rmers. arXiv preprint arXiv:2410.07799,

    arXiv

  48. [2014]

    Text-to-image diffusion models canno t count, and prompt refinement cannot help

    22 [CGH+25] Yuefan Cao, Xuyang Guo, Jiayan Huo, Yingyu Liang, Zhenmei Shi, Zhao Song, Jiahao Zhang, and Zhen Zhuang. Text-to-image diffusion models canno t count, and prompt refinement cannot help. arXiv preprint arXiv:2503.06884 ,

  49. [2016]

    Conv-basis: A new paradigm for efficient attention inference and gradient computation in trans- formers

    [LLS+24] Yingyu Liang, Heshan Liu, Zhenmei Shi, Zhao Song, and Jun ze Yin. Conv-basis: A new paradigm for efficient attention inference and gradient computation in trans- formers. arXiv preprint arXiv:2405.05219 ,

    arXiv

  50. [2017]

    Always skip attention

    [JSML25] Yiping Ji, Hemanth Saratchandran, Peyman Moghadd am, and Simon Lucey. Always skip attention. arXiv preprint arXiv:2505.01996 ,

    arXiv

  51. [2018]

    Streaming kernel pca algo- rithm with small space

    [DSWZ23] Yichuan Deng, Zhao Song, Zifan Wang, and Han Zhang. Streaming kernel pca algo- rithm with small space. arXiv preprint arXiv:2303.04555 ,

    arXiv

  52. [2019]

    Scaling instruction-finetuned language models

    [CHL+22] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi T ay, William Fe- dus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Bra hma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 ,

    arXiv

  53. [2020]

    Circuit complexity bounds for rope-based transformer arch itecture

    [CLL+24] Bo Chen, Xiaoyu Li, Yingyu Liang, Jiangxuan Long, Zhenme i Shi, and Zhao Song. Circuit complexity bounds for rope-based transformer arch itecture. arXiv preprint arXiv:2411.07602,

    arXiv

  54. [2021]

    Videophy: Eval- uating physical commonsense for video generation

    [BLX+25] Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Mich al Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grov er. Videophy: Eval- uating physical commonsense for video generation. In Workshop on Video-Language Models @ NeurIPS 2024 ,

    2024

  55. [2022]

    Multi-layer transformers gradient can be approximated in almost linear time

    [LSS+24] Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song, and Yu fa Zhou. Multi-layer transformers gradient can be approximated in almost linear time. arXiv preprint arXiv:2408.13233,

    arXiv

  56. [2023]

    The geometry of bert

    [BGC25] Matteo Bonino, Giorgia Ghione, and Giansalvo Cirri ncione. The geometry of bert. arXiv preprint arXiv:2502.12033 ,

    arXiv

  57. [2024]

    Fast rope attention: Combin ing the polynomial method and fast fourier transform

    [AS25] Josh Alman and Zhao Song. Fast rope attention: Combin ing the polynomial method and fast fourier transform. In arXiv preprint arXiv:2505.11892 ,

    arXiv

  58. [2025]

    Why do llms attend to the fir st token? arXiv preprint arXiv:2504.02732,

    [BAG+25] Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christo s Perivolaropoulos, Michael Bronstein, Razvan Pascanu, et al. Why do llms attend to the fir st token? arXiv preprint arXiv:2504.02732,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.