{"id":"cc0bca5e-7f71-4f17-91a7-096c5e2f1a8e","arxiv_id":"2507.20096","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"EcoTransformer measures attention by L1 distance between queries and keys, removing the QK^T multiplication and claiming comparable accuracy at lower energy cost.","lead":"This paper replaces dot-product attention in Transformers with an L1-distance-based score that uses additions and absolute differences instead of matrix multiplication. A generalist might read it to see whether cheaper arithmetic can cut the energy cost of modern AI models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Energy claim is unsupported: §4 operation-count arithmetic ignores QKV/αV matmuls, softmax, memory movement, and tensor-core specialization; no measured energy is reported.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the energy claim extrapolates per-operation pJ numbers to an end-to-end conclusion without measurement or a hardware model. My read confirms this and adds two specifics: the attention module's remaining matmuls are ignored, and abs-diff is not equivalent to a single FP32 addition. The L1-attention construction itself is mathematically coherent, and the squared-L2 equivalence in Section 2 is standard when queries/keys are normalized. The accuracy experiments are single-run with λ tuned per dataset and no error bars, so the parity claim is also weakly supported, but the energy claim is the more decisive weakness because it appears in the title, abstract, and conclusion. Section 4's own concession that hardware constraints limit the algorithm is an explicit limitation that the manuscript never resolves. A revision that replaces the energy discussion with measured runtime/energy data, or at least with a realistic roofline/tensor-core-aware model, could make the paper publishable as a negative or conditional result; as written, the advertised claim is unsupported.","tokens_in":7954,"tokens_out":5345,"duration_ms":61615,"concrete_test":"Implement both attention score kernels in CUDA for the CIFAR-10 configuration in Table 2, run on an A100, and profile wall-clock time and GPU energy per training step using nvidia-smi or a hardware energy model. If the L1-based model is not faster or lower-energy than the dot-product baseline with FP16 tensor-core matmul, the 'significantly less energy' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central promise of 'significantly less energy' rests entirely on the Section 4 operation-count comparison: replacing N^2D_k multiplications in QK^T with N^2D_k absolute-difference operations, priced at 0.9 pJ versus 3.7 pJ from Horowitz 2014. This is not an end-to-end energy model and does not support the abstract's claim. First, the attention module still contains QKV projections and the final αV multiplication, which are matrix multiplications; for typical Transformer dimensions these matmuls often contribute as much or more FLOPs than the score matrix, so a 61% reduction in the score module alone cannot translate into 'significantly less energy' for the model. Second, an L1 distance entry requires one subtraction, one absolute-value/select operation, and one addition per dimension—roughly three cheap operations, not one addition—and abs-diff is not the FP32 add in Horowitz's table. Third, modern GPUs execute dense GEMM on tensor cores in FP16/BF16 with far higher throughput and lower energy per operation than a custom absolute-difference kernel, as Section 4 itself concedes: 'the algorithm ... is currently limited by hardware constraints.' Finally, softmax and memory movement are unchanged and typically dominate attention energy at many scales. The paper reports no energy or runtime measurements, so the energy arm of the central claim is not demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EcoTransformer, which replaces the scaled dot-product attention score with softmax of a scaled negative L1-distance matrix between queries and keys, followed by multiplication with values. Section 2 gives a derivation showing that a squared-L2 distance kernel with λ=1/2 reproduces dot-product attention when queries and keys are L2-normalized. Section 4 argues on the basis of per-operation energy costs that the L1 score matrix is cheaper than QK^T. Section 5 sketches how L1 attention could be combined with Longformer and Linformer, and Section 6 reports classification results on NLP, genomics, TCR-epitope, and CIFAR-10 benchmarks. The core claim is that the method matches or exceeds dot-product attention while consuming significantly less energy.","tokens_in":8215,"tokens_out":12585,"duration_ms":118403,"significance":"The clean algebraic observation connecting dot-product attention to squared-L2 distance is correct, and the L1 score matrix is indeed free of multiplications. If validated with measured energy and rigorous accuracy comparisons, the method could be practically valuable for low-power inference. However, the paper currently provides no energy or runtime measurements, and the accuracy results are reported without uncertainty quantification; the central significance claim therefore rests on unverified extrapolation rather than evidence. The authors are to be credited for the explicit derivation in Section 2 and for identifying the per-operation count in Section 4, but these strengths do not compensate for the missing empirical support.","major_comments":[{"comment":"The abstract's claim of 'significantly less energy' is not supported by the evidence in the manuscript. Section 4 computes only operation counts using Horowitz (2014) pJ figures for FP32 addition and multiplication and applies them to the score matrix. This arithmetic omits the QKV projections, the final αV multiplication, softmax, and memory movement; it also miscounts the L1 distance entry, which requires one subtraction, one absolute-value operation, and one addition per dimension rather than a single addition. The section itself concedes that tensor cores are heavily optimized for dense matmul and that the baseline therefore has a practical performance edge. Since no measured energy or wall-clock time is reported anywhere in Section 6, the paper's central energy claim is not demonstrated.","section":"Section 4; Abstract"},{"comment":"The claim that L1 attention 'outperforms' dot-product attention is based on grid-searching λ per dataset and reporting the best test accuracy, which is a form of test-set selection. No error bars, seed counts, or confidence intervals are provided. The text states that λ=3 is robust and gives differences of +0.0190, -0.0102, -0.0026, and +0.0034, so at that configuration the L1 model is worse on two of the four datasets. The large gains reported for λ=15 on StoryCloze and BoolQ are unexplained and need verification. A valid comparison would fix λ on a validation set and report variability across seeds.","section":"Section 6.1, Table 1"},{"comment":"The biological and vision experiments also lack uncertainty quantification. The perfect TCGA numbers (precision, recall, F1, and accuracy all 1.0000) are suspicious and require checking for label leakage or task triviality. The METABRIC results show a decrease in AUROC of 0.0208 relative to the baseline, which is inconsistent with the text's statement that the method consistently outperforms across all metrics. Without repeated runs and significance testing, the 'consistently outperforms' claim is not supported.","section":"Section 6.2, Table 2"},{"comment":"The linear-complexity extension is not correctly specified. In Eq. (3), softmax is applied to the product of the distance matrix and a value slice, and in Eq. (5) two such quantities are added, so the equations mix attention weights with context vectors and do not define a valid attention mechanism. The unbalanced parentheses and undefined indexing in the equations make the proposal impossible to implement as written. This section should be rewritten with a standard attention formulation or removed.","section":"Section 5, Eqs. (3)–(5)"}],"minor_comments":[{"comment":"The title and abstract say 'Attention without Multiplication,' but only the score matrix is multiplication-free; the QKV projections, the αV product, and the output projection remain matrix multiplications. Please qualify the claim.","section":"Title/Abstract"},{"comment":"The text repeatedly says the L1 method 'replaces multiplications with additions'; the actual operation is a subtraction, an absolute value, and an addition per dimension, so the operation-count argument should be revised accordingly.","section":"Section 4"},{"comment":"For λ=1, the text reports the accuracy difference range as -0.0018 to -0.0110, but Table 1 gives -0.0004 on HellaSwag; please correct the inconsistency.","section":"Section 6.1"},{"comment":"Reproducibility details are missing: model sizes, optimizer, learning rate schedule, number of seeds, hardware, and any code are not provided, so the experiments cannot be repeated from the manuscript.","section":"Section 6"},{"comment":"The closely related 'Addition is all you need' (Luo and Sun, 2024) is cited but not compared or discussed; the authors should position EcoTransformer with respect to that prior work.","section":"Related Work"},{"comment":"There is a typo ('ad L1 attention' should be 'and L1 attention'), and Figure 1 is not self-contained because its axes are not labeled.","section":"Section 3"},{"comment":"The equations for the Longformer and Linformer variants contain unbalanced parentheses and undefined notation; please use standard attention notation, e.g., A = softmax(...), O = A V.","section":"Section 5"},{"comment":"The subsumption claim ('the new architecture subsumes scaled dot-product attention as a special case') holds only under L2-normalized queries and keys; this condition is not used in the experiments and should be stated whenever the claim appears.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a clear core idea and a correct algebraic observation, but the main performance and energy claims need substantial new evidence. I would support revision only if the authors add measured energy/runtime benchmarks, use validation-set selection of λ with multiple seeds and confidence intervals, fix the Section 5 equations, and temper the abstract. Without those changes, I would not recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The L1-distance kernel for attention is a legitimate variant, but the paper's two headline claims—accuracy parity and significantly lower energy—are not both backed by what it actually shows.\n\nWhat's new: replacing the QK^T dot-product scores with a negative L1 distance times a bandwidth λ, i.e., a Laplacian kernel on query-key differences. That specific construction isn't in the cited papers, and the link to distance-kernel attention is correct: with normalized queries/keys and squared L2 distance, dot-product attention is a special case. The Section 4 arithmetic (multiplications vs. absolute-difference operations) is accurate as far as it goes. That part is worth taking seriously.\n\nThe soft spots are the usual ones for an efficiency paper. First, the energy claim is an operation-count calculation, not a measurement. The paper itself concedes that tensor cores are optimized for matmul, so on current hardware the L1 kernel may well be slower. Even in theory, the Q,K,V projections and the final αV matmul remain intact; only the score matrix becomes multiplication-free. So \"significantly less energy\" for the whole attention module is not established. No runtime or energy numbers are reported. Second, the accuracy comparisons are single runs with no error bars, and λ is grid-searched per dataset with the best value reported. That makes 'outperforms dot-product' a fitted result. Table 1 even shows λ=3 giving +1.9% on SciQ but −1.0% on StoryCloze; the gains are not consistent. Third, the paper cites Luo and Sun (2024) 'Addition is all you need' but never compares against it, which is a glaring omission for this specific thesis.\n\nNone of this kills the core idea. The L1 kernel is a reasonable thing to try, and the empirical results suggest it can be competitive when λ is tuned. But the version in front of you does not demonstrate the advertised energy advantage and oversells the accuracy results.\n\nWho is this for? Researchers working on efficient attention, especially edge deployment, would get a useful kernel variant and a cautionary example of how to (not) support an efficiency claim.\n\nRecommendation for peer review: send it out. A serious referee, given the right experimental requirements—measured energy or at least wall-clock time, error bars, a fixed λ selection rule, and a comparison to Luo and Sun—could turn this into a solid workshop or conference paper. As written, it's not acceptable, but it deserves referee time rather than a desk reject.","headline":"The L1-distance kernel is a real variant, but the paper's two headline claims—accuracy parity and significantly lower energy—are not both supported by the evidence it presents.","tokens_in":8728,"tokens_out":2471,"would_cite":false,"duration_ms":23960,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EcoTransformer replaces dot-product attention scores with L1 distances between queries and keys, removing multiplication from scoring while matching or beating the standard mechanism on NLP, bioinformatics, and vision benchmarks.","keywords":["EcoTransformer","L1 distance attention","Laplacian kernel","multiplication-free attention","energy-efficient Transformer","distance-based attention","convolution attention"],"falsifier":"Measure end-to-end energy per inference (or per training step) for a tuned L1 EcoTransformer and the dot-product baseline with identical architecture, sequence length, and batch size on a modern accelerator; if the L1 model does not consume significantly less energy than the dot-product model, or takes longer, the 'significantly less energy' claim is falsified even if test accuracy matches.","tokens_in":7756,"feed_emoji":"⚡","tokens_out":11791,"duration_ms":95292,"temperature":0.7,"pith_summary":"The paper sets out to show that the scoring step of Transformer attention need not multiply anything. It replaces the scaled dot product between query and key with their negative L1 distance, passes softmax of those distances through a bandwidth parameter $\\lambda$, and forms the context vector as a weighted sum of values under a Laplacian kernel. With the bandwidth tuned, the authors claim, this L1 attention matches or exceeds scaled dot-product attention on four NLP benchmarks, three biological datasets, and CIFAR-10, while removing the $N^2 D_k$ multiplications of the $QK^T$ score matrix. Because each score entry is built from absolute differences and additions, the paper argues the attention module can consume markedly less energy, citing per-operation energy figures and attention's roughly 38 percent share of GPT-2 inference energy. If these results hold, multiplication-free scoring could lower Transformer energy cost without changing the rest of the architecture.","feed_headline":"Attention without multiplication: L1 scores match dot products","feed_subtitle":"Absolute-distance scoring needs only additions; tuned models match or beat dot-product attention across tasks.","key_machinery":"The load-bearing object is the L1 distance score matrix $L$ with entries $L_{ij}=-|Q_i-K_j|_1$, fed through $\\operatorname{softmax}(\\lambda L/\\sqrt{D_k})$ with a tunable bandwidth $\\lambda$. This makes attention an exponential distance kernel: for L1 it is a Laplacian kernel $k(d)=\\exp(-\\lambda|d|/\\sqrt{D_k})$, and the context vector $O_i=\\sum_j \\alpha_{ij}V_j$ is a convolution of the values with that kernel. The squared-L2 special case is the Gaussian kernel, equivalent to dot-product attention on normalized queries and keys; the energy argument rests on per-operation cost figures, FP32 multiplication at roughly 3.7 pJ versus addition at 0.9 pJ, together with the claim that attention scoring accounts for a large share of Transformer inference energy.","core_discovery":"The central claim is that attention weights can be generated from a distance kernel rather than an inner-product kernel. Define $L_{ij}=-|Q_i-K_j|_1$, set $\\alpha = \\operatorname{softmax}(\\lambda L/\\sqrt{D_k})$, and output $O=\\alpha V$. The paper observes that squared L2 distance attention with $\\lambda=1/2$ reproduces scaled dot-product attention when queries and keys are unit-normalized, so distance attention subsumes dot-product attention; the L1 variant trades the quadratic decay of the Gaussian kernel for the linear decay of the Laplacian kernel. In experiments with identical training settings, tuned L1 attention matches dot-product attention on SciQ, StoryCloze, HellaSwag, and BoolQ and beats it on TCGA, METABRIC, VDJdb, and CIFAR-10. The energy claim is that replacing $N^2D_k$ multiplications with $N^2D_k$ absolute differences and additions reduces score-module energy by up to roughly 61 percent, although the paper concedes that hardware optimized for matrix multiplication may not yet realize this saving in practice.","pith_inferences":["The energy claim should be settled by direct end-to-end measurement: QKV projections, the final value multiplication, softmax, and memory traffic remain, so actual savings depend on whether the score matrix dominates on a given accelerator; a wall-clock and energy benchmark against the dot-product baseline would decide it.","If future hardware adds efficient absolute-difference and addition units, L1 attention could become faster in practice; until then, chips optimized for dense matrix multiply may neutralize the theoretical advantage, making the hardware rather than the algorithm the bottleneck.","Because the context vector is already written as a convolution of values with a distance kernel, the L1 kernel could in principle be made data-dependent or learned directly, connecting the mechanism to implicit-convolution sequence models.","L1 distances and additions are friendly to low-precision and integer arithmetic, so the scoring module is a natural target for quantization, which could extend the energy savings well beyond the FP32 comparison the paper uses."],"forward_implications":["A Transformer can be trained with L1-distance attention in place of scaled dot-product attention and, after tuning $\\lambda$, stay within about a point of the dot-product baseline on the tested NLP tasks; with a task-chosen $\\lambda$ it surpasses the baseline on three of four benchmarks.","Because squared-L2 distance attention is equivalent to dot-product attention on normalized queries and keys, the distance formulation contains standard attention as a special case, so existing sparse, low-rank, and kernel approximations can be carried over to L1 attention; the paper demonstrates this with Longformer-style and Linformer-style constructions at linear complexity.","The score matrix for one head is computed from $N^2D_k$ absolute differences and additions rather than $N^2D_k$ multiplications, so under the paper's energy arithmetic the scoring module's energy could fall by up to roughly 61 percent.","The L1 kernel decays linearly in distance rather than quadratically, spreading attention more broadly; increasing $\\lambda$ sharpens the kernel toward the L2 profile, giving a task-dependent knob that explains why a tuned $\\lambda$ can outperform the fixed dot-product baseline.","The method is demonstrated across NLP, genomics, immunology, and vision; reported gains on TCGA, METABRIC, VDJdb, and CIFAR-10 range from about 0.01 to 0.25 in precision, 0.01 to 0.05 in accuracy, and up to 0.05 in AUROC."],"supporting_citations":[{"why":"Defines the scaled dot-product attention and Transformer architecture that EcoTransformer replaces and compares against.","marker":"Vaswani et al. (2017)"},{"why":"Supplies the per-operation energy numbers, FP32 multiply about 3.7 pJ versus addition about 0.9 pJ, that the energy-savings claim is built on.","marker":"Horowitz (2014)"},{"why":"Provides the estimate that attention accounts for roughly 37.94 percent of GPT-2 inference energy, converting per-operation savings into a module-level energy claim.","marker":"Cao et al. (2021)"},{"why":"Longformer is the sparse-attention architecture that Section 5 extends to an L1-distance variant, supporting the claim that existing complexity reductions carry over.","marker":"Beltagy et al. (2020)"},{"why":"Linformer is the low-rank projection architecture that Section 5 extends to an L1-distance variant, supporting the linear-complexity carry-over claim.","marker":"Wang et al. (2020)"}],"fun_headline_variants":["L1 distance attention: no multiplication, matches or beats dot products","No-multiply attention: L1 scores outperform dot products on some tasks","EcoTransformer: attention via absolute differences, not dot products","Attention without multiplication: Laplacian kernel rivals dot-product","Multiply-free attention: L1 scores beat dot products on several tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The lower-energy conclusion rests on the premise that replacing FP32 multiplications with additions and absolute differences in the score matrix materially reduces total attention-layer energy; if QKV projections, the final value multiplication, softmax, or memory movement dominate, or if the L1 score cannot run efficiently on matrix-multiply-optimized hardware, accuracy parity could still hold while the energy savings evaporate.","fun_headline_variants_meta":{"raw":{"variants":["L1 distance attention: no multiplication, matches or beats dot products","No-multiply attention: L1 scores outperform dot products on some tasks","EcoTransformer: attention via absolute differences, not dot products","Attention without multiplication: Laplacian kernel rivals dot-product","Multiply-free attention: L1 scores beat dot products on several tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0012,"raw_usage":{"total_tokens":4918,"prompt_tokens":888,"completion_tokens":4030,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":3940}},"tokens_in":504,"tokens_out":4030,"duration_ms":28212,"temperature":1.0,"reasoning_tokens":3940,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:50:39.808833+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure end-to-end energy per inference (or per training step) for a tuned L1 EcoTransformer and the dot-product baseline with identical architecture, sequence length, and batch size on a modern accelerator; if the L1 model does not consume significantly less energy than the dot-product model, or takes longer, the 'significantly less energy' claim is falsified even if test accuracy matches.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the per-operation energy numbers, FP32 multiply about 3.7 pJ versus addition about 0.9 pJ, that the energy-savings claim is built on."}],"review_version":2}