Pith. sign in

REVIEW 3 major objections 7 minor 17 references

DeepCrossAttention: Supercharging Transformer Residual Connections

T0 review · 3 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DeepCrossAttention replaces the fixed sum in transformer residual connections with learnable, input-dependent weighting of all prior layer outputs, achieving the same language-modeling perplexity up to three times faster.

desk verdict Strong empirical paper on transformer residual weighting; the DCA speedups look real, but the theoretical 'confirmation' in the abstract is not supported by the theorems, and Lemma B.1 has an arithmetic error. read the letter →

arxiv 2502.06785 v2 pith:2NJO63RX submitted 2025-02-10 cs.LG

Mike Heddes , Adel Javanmard , Kyriakos Axiotis , Gang Fu , MohammadHossein Bateni , Vahab Mirrokni This is my paper · ORCID
classification cs.LG
keywords DeepCrossAttentionresidualconnectionstransformerarchitecturescross-attentionlanguagemodelingtrainingefficiencylow-rankmodelgeneralizednetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the plain residual connection inside transformers, which adds every previous layer's output with equal weight, dilutes useful information and is itself a trainable design axis. Its proposal, DeepCrossAttention (DCA), replaces that fixed sum with three learnable, input-dependent combiners of all prior layer outputs, one feeding queries, one keys, one values, so each block can pick the earlier representations it needs. DCA claims this reaches the same language-modeling perplexity in about one-third of the training time while adding almost no parameters, and that a linear low-rank analysis predicts when the gain should appear. A sympathetic reader would care because the change is architectural rather than a larger model: it retrofits onto pretrained transformers and behaves identically at initialization.

What carries the argument

The load-bearing object is the generalized residual stack $G_t$ and the GRN-v3 update $g_t(x) = (G_t \odot (b_t + \bar w_t))\mathbf{1}$, with $\bar w_t = \mathbf{1}\sigma(w_t^\top G_t)$; $b_t$ is per-dimension, $\bar w_t$ is input-dependent, and $\sigma = \mathrm{ReLU}$. This single identity carries both the architecture and the theory: setting the weights fixed to one recovers ResNet, a shared scalar weight vector gives GRN-v1 (DenseFormer), a per-dimension matrix gives GRN-v2, and the nonlinear input-dependent term gives GRN-v3. DCA's transformer-specific step is to place three independent GRN-v3 instances at each decoder block so that queries, keys, and values each get their own depth-wise combination of prior layers. The proofs characterize the expressible classes---identity-scaled, diagonal-plus-low-rank, plus a rank-one nonlinear term---and then bound their excess risk with the Eckart--Young--Mirsky theorem.

What would settle it

Take a trained transformer, form its residual-function Jacobians, compute the collective rank $r^* = \sum_t \mathrm{rank}(V_t)$, and compare $r^*/d$ with the thresholds in (4.1)--(4.3) using the empirical spectrum of $A-I$; the theory predicts DCA's advantage should vanish once $r^*/d$ is above threshold, so observing a 0.33x time-to-perplexity speedup in that regime would falsify the theoretical explanation.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is a generalization of residual learning, not of attention. Every decoder block maintains a stack $G_t$ of all previous layer outputs, and the block input is formed as $(G_t \odot (b_t + \bar w_t))\mathbf{1}$, where $b_t$ is a learned per-dimension weight and $\bar w_t = \mathbf{1}\sigma(w_t^\top G_t)$ makes the weighting input-dependent; running this same generalized residual network three times to produce queries, keys, and values yields DCA's depth-wise cross-attention. The theoretical result is that in a linear low-rank model the GRN classes strictly enlarge the representable maps, from $(I+M)x$ for ResNet to $(\alpha I + M)x$, $(D+M)x$, and beyond, and that their excess risk is smaller than ResNet's whenever the collective rank $r^*$ satisfies conditions (4.1)--(4.3). Empirically, DCA reaches the final perplexity of a 24-layer transformer in 0.33x the training steps and gives lower perplexity at matched parameter counts.

Load-bearing premise

The theoretical guarantees are proven only for a simplified linear network whose layers all have low rank, with the total rank smaller than the representation dimension, and for a target transformation whose deviation from the identity is positive semidefinite; the paper never checks whether actual transformer layers live in that regime.

Editorial extensions

If this is right

  • A 24-layer 1-DCA or 2-DCA model reaches the transformer's final LM1B perplexity in 0.33x of the training time while ending at 14.48/14.41 instead of 15.14 perplexity.
  • DCA is a parameter-efficient way to deepen a model: a 30-layer DCA model out-perplexities a 42-layer transformer at the same parameter budget.
  • The advantage shrinks with width, from a perplexity delta of -2.82 at width 64 to -0.39 at width 1024, matching the prediction that low collective rank relative to dimension is where GRNs help.
  • DCA improves perplexity on C4 across scales from 75M to 449M parameters and adds 0.7% ImageNet accuracy to ViT-S/16.
  • DCA mitigates the loss spikes that appear during large-model training runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because DCA behaves identically to a residual network at initialization, the reported retrofit result suggests that any pretrained residual-style model, not only the 6-layer transformer tested, could be upgraded without changing its function at insertion time.
  • The learned bias distributions, which assign negative weights to many intermediate layers, hint that DCA is partly performing per-token layer filtering; an ablation that prunes the lowest-weight layers at a fixed budget could separate that effect from richer mixing.
  • The theory's rank-ratio threshold suggests a practical placement rule: estimate $r^*/d$ per block and insert GRNs only where the rank ratio is low, since both the width-scaling experiments and Theorem 4.3 predict diminishing returns in wide models.
  • At very deep scales the per-dimension weights $b_t$ dominate the added parameter count, so low-rank factorizations of $b_t$ could make the overhead smaller still; the paper does not explore that direction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces DeepCrossAttention (DCA), a transformer residual-connection modification in which each decoder block uses three independent Generalized Residual Network (GRN-v3) instances to generate the queries, keys, and values of the attention module from a stack of previous layer outputs. The method is motivated by three generalizations of residual networks (GRN-v1, GRN-v2, GRN-v3), and the theoretical Section 4 characterizes function classes and excess-risk/model-size trade-offs for these GRNs in a low-rank linear model, claiming to confirm an improved trade-off. Experiments on LM1B, C4, and ImageNet show that DCA reaches lower perplexity for a given training time or parameter budget compared to transformer baselines and prior residual-learning methods.

Significance. If the empirical results hold, DCA is a practically useful, low-cost architectural improvement: Table 1 reports reaching the transformer's final perplexity in 0.33x the training time, Tables 3, 5, and 6 show consistent perplexity gains over strong baselines, and Table 8 reports a 0.7% ImageNet accuracy gain. The theoretical framework for GRN-v1/v2/v3 in a low-rank linear model is an original contribution and yields interpretable conditions (4.1)-(4.3). The paper also includes an ablation (Table 7), a retrofitting experiment (Table 4), and training-stability observations. However, the connection between the theory and the DCA architecture itself is not established, and a lemma used in the main proof contains an arithmetic error; these issues are fixable but mean the abstract's 'Theoretical analysis confirms' sentence overreaches as written.

major comments (3)
  1. [Section 4 and Abstract] All formal results in Section 4 — Theorems 4.1, 4.2, 4.3, and Propositions 4.4 and 4.5 — analyze a single GRN-v1/v2/v3 residual combination, and Proposition 4.5 is restricted to MLP residual functions f_t(z) = V_t φ(U_t z). DCA, as defined in Section 3.2, uses three independent GRN-v3 instances per decoder block to produce the queries, keys, and values of the attention module, and no theorem or corollary analyzes this composed model class. Therefore the abstract sentence 'Theoretical analysis confirms that DCA provides an improved trade-off between accuracy and model size...' is not supported by the paper's own analysis. Please either add a theoretical argument covering DCA (for example, showing that the DCA model class is contained in one of the classes analyzed in Theorem 4.1) or substantially qualify the abstract and conclusion to state that the theory applies to the GRN component, with DCA's benefits supported empirically.
  2. [Appendix B, Lemma B.1] Lemma B.1 contains an arithmetic error. For x ~ N(0, I) and w = ||w|| e_1, the paper computes E[σ(w^T x)^2 ||x||^2] as ||w||^2 (d+1)/2. However, E[σ(x_1)^2 x_1^2] = 3/2 and E[σ(x_1)^2] E[Σ_{i=2}^d x_i^2] = (1/2)(d-1), so the sum is (d+2)/2, not (d+1)/2. Consequently, equation (B.6), the GRN-v3 bounds in Theorem 4.2 with denominators π(d+1), and the GRN-v3 threshold in Theorem 4.3(i) (including the definitions of η and ξ0) are numerically incorrect. The qualitative form of the results may be unchanged, but the stated constants and thresholds must be corrected.
  3. [Section 4.1 and experiments] The theoretical results assume the collective rank condition r* = Σ_{t=1}^T rank(V_t) < d and, for Theorem 4.3, that A - I is positive semidefinite. The paper never measures r*/d or the spectral condition for the transformer models used in the experiments, and for a transformer with more than one layer and full-rank residual functions, r* can exceed d. The width-scaling experiment in Table 2 is suggestive but does not verify the theoretical conditions. The claim that the analysis 'confirms' an improved trade-off should therefore be conditioned on these assumptions, or the experimental section should provide evidence that the conditions hold (or state why the conditions are not necessary in practice).
minor comments (7)
  1. [Section 4.1, Theorem 4.1] The notation 'min(rt)^T_{t=1}' is ambiguous; it should be defined explicitly, for example as min_{1≤t≤T} r_t, since the proof treats it as a minimum over the layer ranks.
  2. [Theorem 4.2] The symbol 'Err*' appears to be a typo for 'ER*'; please use the same notation throughout.
  3. [Proposition 4.4] The statement introduces '˜r_*' and then defines '˜r'_*'; the notation should be made consistent.
  4. [Section 3.1] The memory/computation reduction for k-DCA is described only in words; a precise definition of which layer outputs are retained in G_t for a given k would improve reproducibility.
  5. [Table 1] The 'TIME' column is described as normalized time to reach the transformer's perplexity, but the interpolation or measurement procedure is not stated; please specify how the time is computed from the training curves.
  6. [Figure 5] The x-axis label 'min' in the right panel is unclear; it presumably refers to κ = λ_min/λ_max, and this should be stated explicitly.
  7. [Appendix H] The caption of Figure 10 describes the learned GRN-v3 weights as 'bias values,' but in the GRN-v3 definition the learned term is σ(w_t^T G_t), which is an input-dependent weight rather than a bias; please reconcile the terminology.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: the theoretical trade-off bounds are derived from first principles for a stylized linear model and are not fitted to or defined by the empirical claims; the caveats are scope and correctness issues, not circularity.

full rationale

The central theoretical results (Theorems 4.1-4.3, Propositions 4.4-4.5) are derived from the definitions of the GRN classes in Section 3.1. Theorem 4.1 algebraically characterizes the representable function classes; Theorem 4.2 computes excess-risk upper bounds via Eckart-Young-Mirsky and Gaussian moment estimates; Theorem 4.3 and Proposition 4.4 manipulate those bounds to obtain conditions (4.1)-(4.3). No parameter is fitted to the LM1B, C4, or ImageNet experiments, no step assumes the empirical conclusion, and the bottleneck-rank extension cites independent prior work (Jacot, 2023). The empirical speed and quality claims in Table 1 and Figures 6-7 are measured training outcomes, not predictions generated from the theory by construction. The legitimate concerns raised about this paper are not circularity: the abstract's sentence 'Theoretical analysis confirms that DCA provides an improved trade-off...' overreaches because the theorems analyze a single GRN-v1/v2/v3 residual combination in a low-rank linear model, whereas DCA composes three independent GRN-v3 instances to produce attention queries, keys, and values; and Lemma B.1's stated value (d+1)/2 for E[sigma(w^T x)^2 ||x||^2] is an arithmetic error, with the correct value (d+2)/2. These are correctness and scope issues, not derivation-by-construction, fitted-input-as-prediction, or load-bearing self-citation, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the low-rank linear model and on standard matrix-analysis assumptions rather than on fitted parameters. The most consequential axioms are the regime restriction r* < d and the positive-semidefinite condition on A - I, neither of which is verified for the transformer experiments. The Gaussian and isotropy assumptions are used to obtain closed-form expectations in the GRN-v3 bound. No novel entities are postulated; GRN and DCA are architectural modules, not new physical or mathematical objects.

assumptions (5)
  • domain assumption Features are isotropic: E[xx^T] = I, and Lemma B.1 takes x ~ N(0,I).
    Used to compute the excess risk in Section 4.2 and the Gaussian expectations in Lemma B.1; the paper states the isotropy assumption and then assumes Gaussianity for the GRN-v3 lemma.
  • domain assumption Collective rank of all residual layers is below ambient dimension: r* = sum_t rank(V_t) < d.
    Defines the regime of Section 4.1 and underpins Theorems 4.1-4.3; never checked for transformer experiments.
  • domain assumption The target linear map satisfies A - I >= 0 (positive semidefinite) with lambda_min > 0.
    Assumed in Theorem 4.3 to derive the threshold conditions (4.1)-(4.3); no evidence is given that language-modeling targets satisfy it.
  • standard math Bottleneck-rank subadditivity properties from Jacot (2023): rank(f∘g) <= min{rank(f), rank(g)} and rank(f+g) <= rank(f)+rank(g).
    Used in Proposition 4.5 to extend the class characterizations to nonlinear MLPs; accepted from the cited prior work.
  • domain assumption Each layer has rank at least one when counting parameters in Lemma C.1.
    Used to bound T' in terms of the collective rank when equating parameter budgets; plausible but restricts the comparison to non-degenerate layers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeepCrossAttention: Supercharging Transformer Residual Connections." pith.science (2026). https://pith.science/paper/2NJO63RX

@misc{pith2026250206785,
  author       = {Pith},
  title        = {Pith review of: DeepCrossAttention: Supercharging Transformer Residual Connections},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2NJO63RX}},
  note         = {Machine review of arXiv:2502.06785}
}
read the original abstract

Transformer networks have achieved remarkable success across diverse domains, leveraging a variety of architectural innovations, including residual connections. However, traditional residual connections, which simply sum the outputs of previous layers, can dilute crucial information. This work introduces DeepCrossAttention (DCA), an approach that enhances residual learning in transformers. DCA employs learnable, input-dependent weights to dynamically combine layer outputs, enabling the model to selectively focus on the most relevant information in any of the previous layers. Furthermore, DCA incorporates depth-wise cross-attention, allowing for richer interactions between layers at different depths. Our language modeling experiments show that DCA achieves improved perplexity for a given training time. Moreover, DCA obtains the same model quality up to 3x faster while adding a negligible number of parameters. Theoretical analysis confirms that DCA provides an improved trade-off between accuracy and model size when the ratio of collective layer ranks to the ambient dimension falls below a critical threshold.

Figures

Figures reproduced from arXiv: 2502.06785 by the authors.

Figure 1
Figure 1. Training low-rank linear models to learn the identity and a random transformation. Each model consists of 10 linear layers, each of rank 3, and is trained using mini-batch SGD. to fully reconstruct the input even after seeing 103 batches (105 examples), while the model with learnable residual weights is able to reach extremely small loss values, even with 100x fewer examples. This confirms that ResNet does not addre… view at source ↗
Figure 2
Figure 2. Two alternative schematic representations of standard ResNet. The top represents the recursive form, the bottom repre￾sents the explicit sum [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Computation diagram of GRN-v3. 3.1. Generalized Residual Networks (GRN) We propose three generalizations of ResNets by consider￾ing weighted linear combinations of previous layer outputs. The parameters of the modules and the generalizations are all optimized during training using the AdamW optimizer (Loshchilov & Hutter, 2017). Dimension-independent weights (GRN-v1). We consider simple linear combinations as gt(x) … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Computation diagram of a DCA decoder block. 4. Theoretical analysis Motivated by language modeling tasks, we focus on the regime where the size of the training set (n) significantly exceeds the input dimension (n ≫ d). As we increase the number of model parameters, the…
Figure 5
Figure 5. Figure 5: Gain in the model performance achieved by GRN-v1 and GRN-v2 over ResNet. The plots represents the lower bounds for G1 and G2 given in Theorem 4.3(ii). Observe that the gain at larger dimension d is higher. Left panel shows that the gain decreases as the collective rank…
Figure 6
Figure 6. Figure 6: Perplexity on LM1B with 6, 12, 18, 24, 30, 36, and 42 layer transformer and DCA models. First and last-k. DCA can be made more efficient by including only the first and last-k layer outputs explicitly in the input stack to the GRNs (see Section 3.1). In this experiment…
Figure 7
Figure 7. Figure 7: Perplexity on LM1B versus the training time with trans￾former and 2-DCA models of various depths. Model width scaling. Our theoretical results indicate that the benefit of GRN is inversely related to the rank of the model. With this experiment, we validate whether the …
Figure 8
Figure 8. Figure 8: Perplexity on LM1B pre-training versus the training time with transformer and DCA models of various depths. G. Steps versus perplexity on the C4 dataset This appendix provides additional results on training steps versus perplexity for 8-DCA models [PITH_FULL_IMAGE:fig…
Figure 9
Figure 9. Figure 9: Perplexity on C4 pre-training versus the number of steps with transformer and 8-DCA models of various sizes [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: shows the distribution of the learned bias values for each GRN-v3 instance of a 30-layer model. The layers tend to weight the inputs and the last few layers the most and frequently assign a negative bias for the intermediate layers, indicating that the layers are filt…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 6 canonical work pages

  1. [1]

    Better plain vit baselines for imagenet-1k

    Beyer, L., Zhai, X., and Kolesnikov, A. Better plain vit baselines for imagenet-1k. arXiv preprint arXiv:2205.01580, 2022

    arXiv 2022

  2. [2]

    One billion word benchmark for measuring progress in statistical language modeling

    Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., and Robinson, T. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013

    arXiv 2013

  3. [3]

    W., Sutton, C., Gehrmann, S., et al

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  4. [4]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., and Zhai, X. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021

    work page 2021

  5. [5]

    Ast: Audio spectrogram transformer

    Gong, Y., Chung, Y.-A., and Glass, J. Ast: Audio spectrogram transformer. In Interspeech 2021, pp.\ 571--575, 2021

    work page 2021

  6. [6]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  7. [7]

    Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 4700--4708, 2017

    2017

  8. [8]

    Implicit bias of large depth networks: a notion of rank for nonlinear functions

    Jacot, A. Implicit bias of large depth networks: a notion of rank for nonlinear functions. The Eleventh International Conference on Learning Representations, 2023

    work page 2023

Show all 17 references
  1. [9]

    L., Bahl, L

    Jelinek, F., Mercer, R. L., Bahl, L. R., and Baker, J. K. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62 0 (S1): 0 S63--S63, 1977

    1977

  2. [10]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  3. [11]

    Laurel: Learned augmented residual layer

    Menghani, G., Kumar, R., and Kumar, S. Laurel: Learned augmented residual layer. In Workshop on Efficient Systems for Foundation Models II, 2024

    2024

  4. [12]

    Denseformer: Enhancing information flow in transformers via depth weighted averaging

    Pagliardini, M., Mohtashami, A., Fleuret, F., and Jaggi, M. Denseformer: Enhancing information flow in transformers via depth weighted averaging. arXiv preprint arXiv:2402.02622, 2024

    2024 arXiv

  5. [13]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 0 (140): 0 1--67, 2020 a

    2020

  6. [14]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020 b

    2020

  7. [15]

    K., Greff, K., and Schmidhuber, J

    Srivastava, R. K., Greff, K., and Schmidhuber, J. Training very deep networks. Advances in neural information processing systems, 28, 2015

    2015

  8. [16]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  9. [17]

    Hyper-connections

    Zhu, D., Huang, H., Huang, Z., Zeng, Y., Mao, Y., Wu, B., Min, Q., and Zhou, X. Hyper-connections. arXiv preprint arXiv:2409.19606, 2024

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.