REVIEW 4 major objections 8 minor 1 cited by
EcoTransformer: Attention without Multiplication
T0 review · 4 major / 8 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read EcoTransformer replaces dot-product attention scores with L1 distances between queries and keys, removing multiplication from scoring while matching or beating the standard mechanism on NLP, bioinformatics, and vision benchmarks.
desk verdict The L1-distance kernel is a real variant, but the paper's two headline claims—accuracy parity and significantly lower energy—are not both supported by the evidence it presents. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the L1 distance score matrix $L$ with entries $L_{ij}=-|Q_i-K_j|_1$, fed through $\operatorname{softmax}(\lambda L/\sqrt{D_k})$ with a tunable bandwidth $\lambda$. This makes attention an exponential distance kernel: for L1 it is a Laplacian kernel $k(d)=\exp(-\lambda|d|/\sqrt{D_k})$, and the context vector $O_i=\sum_j \alpha_{ij}V_j$ is a convolution of the values with that kernel. The squared-L2 special case is the Gaussian kernel, equivalent to dot-product attention on normalized queries and keys; the energy argument rests on per-operation cost figures, FP32 multiplication at roughly 3.7 pJ versus addition at 0.9 pJ, together with the claim that attention scoring accounts for a large share of Transformer inference energy.
What would settle it
Measure end-to-end energy per inference (or per training step) for a tuned L1 EcoTransformer and the dot-product baseline with identical architecture, sequence length, and batch size on a modern accelerator; if the L1 model does not consume significantly less energy than the dot-product model, or takes longer, the 'significantly less energy' claim is falsified even if test accuracy matches.
Extended reading notes
Core claim
The central claim is that attention weights can be generated from a distance kernel rather than an inner-product kernel. Define $L_{ij}=-|Q_i-K_j|_1$, set $\alpha = \operatorname{softmax}(\lambda L/\sqrt{D_k})$, and output $O=\alpha V$. The paper observes that squared L2 distance attention with $\lambda=1/2$ reproduces scaled dot-product attention when queries and keys are unit-normalized, so distance attention subsumes dot-product attention; the L1 variant trades the quadratic decay of the Gaussian kernel for the linear decay of the Laplacian kernel. In experiments with identical training settings, tuned L1 attention matches dot-product attention on SciQ, StoryCloze, HellaSwag, and BoolQ and beats it on TCGA, METABRIC, VDJdb, and CIFAR-10. The energy claim is that replacing $N^2D_k$ multiplications with $N^2D_k$ absolute differences and additions reduces score-module energy by up to roughly 61 percent, although the paper concedes that hardware optimized for matrix multiplication may not yet realize this saving in practice.
Load-bearing premise
The lower-energy conclusion rests on the premise that replacing FP32 multiplications with additions and absolute differences in the score matrix materially reduces total attention-layer energy; if QKV projections, the final value multiplication, softmax, or memory movement dominate, or if the L1 score cannot run efficiently on matrix-multiply-optimized hardware, accuracy parity could still hold while the energy savings evaporate.
Editorial extensions
If this is right
- A Transformer can be trained with L1-distance attention in place of scaled dot-product attention and, after tuning $\lambda$, stay within about a point of the dot-product baseline on the tested NLP tasks; with a task-chosen $\lambda$ it surpasses the baseline on three of four benchmarks.
- Because squared-L2 distance attention is equivalent to dot-product attention on normalized queries and keys, the distance formulation contains standard attention as a special case, so existing sparse, low-rank, and kernel approximations can be carried over to L1 attention; the paper demonstrates this with Longformer-style and Linformer-style constructions at linear complexity.
- The score matrix for one head is computed from $N^2D_k$ absolute differences and additions rather than $N^2D_k$ multiplications, so under the paper's energy arithmetic the scoring module's energy could fall by up to roughly 61 percent.
- The L1 kernel decays linearly in distance rather than quadratically, spreading attention more broadly; increasing $\lambda$ sharpens the kernel toward the L2 profile, giving a task-dependent knob that explains why a tuned $\lambda$ can outperform the fixed dot-product baseline.
- The method is demonstrated across NLP, genomics, immunology, and vision; reported gains on TCGA, METABRIC, VDJdb, and CIFAR-10 range from about 0.01 to 0.25 in precision, 0.01 to 0.05 in accuracy, and up to 0.05 in AUROC.
Reading between the lines
- The energy claim should be settled by direct end-to-end measurement: QKV projections, the final value multiplication, softmax, and memory traffic remain, so actual savings depend on whether the score matrix dominates on a given accelerator; a wall-clock and energy benchmark against the dot-product baseline would decide it.
- If future hardware adds efficient absolute-difference and addition units, L1 attention could become faster in practice; until then, chips optimized for dense matrix multiply may neutralize the theoretical advantage, making the hardware rather than the algorithm the bottleneck.
- Because the context vector is already written as a convolution of values with a distance kernel, the L1 kernel could in principle be made data-dependent or learned directly, connecting the mechanism to implicit-convolution sequence models.
- L1 distances and additions are friendly to low-precision and integer arithmetic, so the scoring module is a natural target for quantization, which could extend the energy savings well beyond the FP32 comparison the paper uses.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes EcoTransformer, which replaces the scaled dot-product attention score with softmax of a scaled negative L1-distance matrix between queries and keys, followed by multiplication with values. Section 2 gives a derivation showing that a squared-L2 distance kernel with λ=1/2 reproduces dot-product attention when queries and keys are L2-normalized. Section 4 argues on the basis of per-operation energy costs that the L1 score matrix is cheaper than QK^T. Section 5 sketches how L1 attention could be combined with Longformer and Linformer, and Section 6 reports classification results on NLP, genomics, TCR-epitope, and CIFAR-10 benchmarks. The core claim is that the method matches or exceeds dot-product attention while consuming significantly less energy.
Significance. The clean algebraic observation connecting dot-product attention to squared-L2 distance is correct, and the L1 score matrix is indeed free of multiplications. If validated with measured energy and rigorous accuracy comparisons, the method could be practically valuable for low-power inference. However, the paper currently provides no energy or runtime measurements, and the accuracy results are reported without uncertainty quantification; the central significance claim therefore rests on unverified extrapolation rather than evidence. The authors are to be credited for the explicit derivation in Section 2 and for identifying the per-operation count in Section 4, but these strengths do not compensate for the missing empirical support.
major comments (4)
- [Section 4; Abstract] The abstract's claim of 'significantly less energy' is not supported by the evidence in the manuscript. Section 4 computes only operation counts using Horowitz (2014) pJ figures for FP32 addition and multiplication and applies them to the score matrix. This arithmetic omits the QKV projections, the final αV multiplication, softmax, and memory movement; it also miscounts the L1 distance entry, which requires one subtraction, one absolute-value operation, and one addition per dimension rather than a single addition. The section itself concedes that tensor cores are heavily optimized for dense matmul and that the baseline therefore has a practical performance edge. Since no measured energy or wall-clock time is reported anywhere in Section 6, the paper's central energy claim is not demonstrated.
- [Section 6.1, Table 1] The claim that L1 attention 'outperforms' dot-product attention is based on grid-searching λ per dataset and reporting the best test accuracy, which is a form of test-set selection. No error bars, seed counts, or confidence intervals are provided. The text states that λ=3 is robust and gives differences of +0.0190, -0.0102, -0.0026, and +0.0034, so at that configuration the L1 model is worse on two of the four datasets. The large gains reported for λ=15 on StoryCloze and BoolQ are unexplained and need verification. A valid comparison would fix λ on a validation set and report variability across seeds.
- [Section 6.2, Table 2] The biological and vision experiments also lack uncertainty quantification. The perfect TCGA numbers (precision, recall, F1, and accuracy all 1.0000) are suspicious and require checking for label leakage or task triviality. The METABRIC results show a decrease in AUROC of 0.0208 relative to the baseline, which is inconsistent with the text's statement that the method consistently outperforms across all metrics. Without repeated runs and significance testing, the 'consistently outperforms' claim is not supported.
- [Section 5, Eqs. (3)–(5)] The linear-complexity extension is not correctly specified. In Eq. (3), softmax is applied to the product of the distance matrix and a value slice, and in Eq. (5) two such quantities are added, so the equations mix attention weights with context vectors and do not define a valid attention mechanism. The unbalanced parentheses and undefined indexing in the equations make the proposal impossible to implement as written. This section should be rewritten with a standard attention formulation or removed.
minor comments (8)
- [Title/Abstract] The title and abstract say 'Attention without Multiplication,' but only the score matrix is multiplication-free; the QKV projections, the αV product, and the output projection remain matrix multiplications. Please qualify the claim.
- [Section 4] The text repeatedly says the L1 method 'replaces multiplications with additions'; the actual operation is a subtraction, an absolute value, and an addition per dimension, so the operation-count argument should be revised accordingly.
- [Section 6.1] For λ=1, the text reports the accuracy difference range as -0.0018 to -0.0110, but Table 1 gives -0.0004 on HellaSwag; please correct the inconsistency.
- [Section 6] Reproducibility details are missing: model sizes, optimizer, learning rate schedule, number of seeds, hardware, and any code are not provided, so the experiments cannot be repeated from the manuscript.
- [Related Work] The closely related 'Addition is all you need' (Luo and Sun, 2024) is cited but not compared or discussed; the authors should position EcoTransformer with respect to that prior work.
- [Section 3] There is a typo ('ad L1 attention' should be 'and L1 attention'), and Figure 1 is not self-contained because its axes are not labeled.
- [Section 5] The equations for the Longformer and Linformer variants contain unbalanced parentheses and undefined notation; please use standard attention notation, e.g., A = softmax(...), O = A V.
- [Section 2] The subsumption claim ('the new architecture subsumes scaled dot-product attention as a special case') holds only under L2-normalized queries and keys; this condition is not used in the experiments and should be stated whenever the claim appears.
Circularity Check
No significant circularity: the L1-distance attention formula is an explicit reparameterization of dot-product attention, and the reported gains rest on empirical tuning rather than on a self-referential derivation.
full rationale
The paper's derivation chain is self-contained. Section 2 begins with the algebraic identity <Qi,Kj> = 1/2(||Qi||^2 + ||Kj||^2 - ||Qi-Kj||^2), which correctly shows that, under unit L2 normalization, softmax over squared L2 distances with prefactor 1/(2*sqrt(Dk)) coincides with scaled dot-product attention; this identity is used only as motivation for generalizing to L1 distances via alpha_new = softmax(lambda*L/sqrt(Dk)), not as evidence of superiority. The subsequent performance claims are empirical: lambda is explicitly a 'tuning parameter' and the text repeatedly says results hold 'with proper tuning' or when 'the optimal lambda is selected via grid search.' Selecting lambda per benchmark is a model-selection protocol that can overstate generalization if done on the test set, but it is not a definitional circularity because the definition of alpha_new does not presuppose any accuracy outcome, and no equation reduces to its own input. The energy claim in Section 4 is explicitly labeled 'theoretical' and 'currently limited by hardware constraints,' so the absence of measured end-to-end energy is an evidence gap rather than a circular step. There are no self-citations carrying the argument, no imported uniqueness theorems, and no fitted parameter renamed as a prediction; hence the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (1)
- attention kernel bandwidth lambda =
per dataset: SciQ 3, StoryCloze 15, HellaSwag 1, BoolQ 3; TCGA 1, METABRIC 10, VDJdb 25, CIFAR-10 1
assumptions (3)
- standard math The identity ||Qi-Kj||_2^2 = ||Qi||^2 + ||Kj||^2 - 2<Qi,Kj> is used in Section 2 to express dot-product attention as squared-L2-distance attention.
- domain assumption Softmax over negative L1 distances is a valid and sufficiently expressive replacement for dot-product attention.
- domain assumption Operation-level energy ratios from Horowitz (2014) transfer to end-to-end attention energy.
Cite this review
Pith. "Pith review of EcoTransformer: Attention without Multiplication." pith.science (2026). https://pith.science/paper/S7WKVTIS
@misc{pith2026250720096,
author = {Pith},
title = {Pith review of: EcoTransformer: Attention without Multiplication},
year = {2026},
howpublished = {\url{https://pith.science/paper/S7WKVTIS}},
note = {Machine review of arXiv:2507.20096}
}
read the original abstract
The Transformer, with its scaled dot-product attention mechanism, has become a foundational architecture in modern AI. However, this mechanism is computationally intensive and incurs substantial energy costs. We propose a new Transformer architecture EcoTransformer, in which the output context vector is constructed as the convolution of the values using a Laplacian kernel, where the distances are measured by the L1 metric between the queries and keys. Compared to dot-product based attention, the new attention score calculation is free of matrix multiplication. It performs on par with, or even surpasses, scaled dot-product attention in NLP, bioinformatics, and vision tasks, while consuming significantly less energy. (This version (v2) supersedes v1 and reflects the intended release and licensing.)
Figures
Forward citations
Cited by 1 Pith paper
-
Lapis: Laplacian Spiking Attention via First-Spike Timing and Membrane Leakage
Lapis replaces dot-product query-key scoring in spiking vision transformers with L1-distance-based Laplacian kernels on first-spike latencies, reaching near-dot-product accuracy at lower estimated arithmetic cost.
Reference graph
Works this paper leans on
-
[1]
Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473
arXiv 2014
-
[2]
Beltagy, I., Peters, M. E., and Cohan, A. (2020). Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150
arXiv 2020
-
[3]
K., Trivedi, H., Balasubramanian, A., and Balasubramanian, N
Cao, Q., Lal, Y. K., Trivedi, H., Balasubramanian, A., and Balasubramanian, N. (2021). Irene: Interpretable energy prediction for transformers. arXiv preprint arXiv:2106.01199
arXiv 2021
-
[4]
Child, R., Gray, S., Radford, A., and Sutskever, I. (2019). Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509
arXiv 2019
-
[5]
Q., Mohiuddin, A., Kaiser, L., Belanger, D., Colwell, L., and Weller, A
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q., Mohiuddin, A., Kaiser, L., Belanger, D., Colwell, L., and Weller, A. (2021). Rethinking attention with performers. In International Conference on Learning Representations (ICLR)
work page 2021
-
[6]
Dao, T. (2023). Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691
arXiv 2023
-
[7]
Y., Ermon, S., Rudra, A., and R \'e , C
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and R \'e , C. (2022). Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems
work page 2022
-
[8]
De Vries, A. (2023). The growing energy footprint of artificial intelligence. Joule , 7(10):2191--2194
work page 2023
Show all 19 references
-
[9]
and Dao, T
Gu, A. and Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752
2023 arXiv
-
[10]
Horowitz, M. (2014). 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC) , pages 10--14. IEEE
2014
-
[11]
and Sun, W
Luo, H. and Sun, W. (2024). Addition is all you need for energy-efficient language models. arXiv preprint arXiv:2410.00907
2024 arXiv
-
[12]
Luong, M.-T., Pham, H., and Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025
2015 arXiv
-
[13]
Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and R \'e , C
Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and R \'e , C. (2023). Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning , pages 28043--28078. PMLR
2023
-
[14]
Qin, Z., Hu, Z., Shen, Y., Chao, W.-L., and Hu, X. B. (2022). Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 11544--11552
2022
-
[15]
Research, F. A. (2022). xformers: A modular and hackable transformer library. https://github.com/facebookresearch/xformers. Accessed: 2025-07-24
2022
-
[16]
Tay, Y., Bahri, D., Metzler, D., Juan, D.-C., Zhao, Z., and Zheng, C. (2021). Synthesizer: Rethinking self-attention for transformer models. In International conference on machine learning , pages 10183--10192. PMLR
2021
-
[17]
N., Kaiser, ., and Polosukhin, I
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems , 30
2017
-
[18]
Z., Khabsa, M., Fang, H., and Ma, H
Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. (2020). Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768
2020 arXiv
-
[19]
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., and Ahmed, A. (2020). Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems , volume 33, pages 17283--17297
2020
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.