Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operators

Giorgos Dimitrakopoulos; Kosmas Alexandridis; Vasileios Titopoulos

REVIEW 3 major objections 5 minor 31 references

Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operators

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fused exponential-multiplication hardware unit using logarithmic quantization and exponent adjustment reduces FlashAttention accelerator area by about 29% and power by about 18% without visible accuracy loss on GLUE.

desk verdict A clean fused-exp-multiply idea for FlashAttention-2 with plausible area/power savings, but the accuracy case rests on a single model/benchmark and an approximation with up to ~49% per-call error. read the letter →

arxiv 2505.14314 v2 pith:SBTEUJCR submitted 2025-05-20 cs.AR cs.LG

Kosmas Alexandridis , Vasileios Titopoulos , Giorgos Dimitrakopoulos This is my paper

classification cs.ARcs.LG

keywords hardwareoperatorsarchitecturesattentionflashattentionproposedacceleratorsarea

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Transformer models use attention to decide how much each token should influence every other token. The attention score depends on softmax, which needs exponentials, and then the result is multiplied by value vectors. FlashAttention computes these values in an online, tiled way so memory use stays low, but every step still needs expensive exponential calculators and floating-point multipliers in hardware.

This paper introduces ExpMul, a fused hardware operator that computes e^x times a vector V in one cheap step instead of evaluating e^x and then multiplying. The trick is logarithmic quantization. Since x is always zero or negative inside FlashAttention, e^x lies between 0 and 1, so it can be written as 2^{-L}. The hardware clips x to a small range, converts it to fixed point, computes L with integer shifts and additions, and then subtracts L from the exponent field of every element of V. No full floating-point multiplication is needed. The authors synthesize two designs, one with separate exp units and multipliers and one with ExpMul, in a 28nm flow. They report a 28.8% smaller area and 17.6% lower power on average.

The accuracy check uses Google's FLAN-T5 on ten GLUE tasks. Numbers stay close to the FP32 and BF16 baselines, with some small wins and losses. However, the approximation is coarse. Rounding L to an integer can make a single e^x factor wrong by up to roughly 40%, and the paper validates safety on one model and one benchmark without error bars. The hardware savings are plausible and clearly explained, but the claim that accuracy is broadly unaffected needs wider testing.

Extended reading notes

Core claim

In Eq. (4) the paper defines ExpMul(x,V)=e^x V, and in Section V it claims that 'utilizing ExpMul operators reduces the hardware area by more than 28.8% on average' and 'reducing power consumption by over 17.8% on average' compared with separate exponential-evaluation and floating-point-multiplication hardware, while Table I shows accuracy on FLAN-T5/GLUE that is comparable to FP32 and BF16 baselines. If true, FlashAttention-2 accelerators can compute the output update using only clipping, integer shift-add, and exponent-field decrements, eliminating FP multipliers and dedicated exp units.

Load-bearing premise

The approximation e^x is roughly 2^{-L}, where L is an integer obtained by rounding -x times the shift-based constant (1 + 1/2 - 1/16) on a clipped fixed-point x, and where the mantissa of V is left completely unchanged, is accurate enough for LLM attention. In the worst case the factor is off by roughly sqrt(2), about 41%, so preservation of accuracy is not mathematically guaranteed. It rests on empirical results from one model, FLAN-T5, on one benchmark set, GLUE, reported in Table I without error bars. Equations (6) to (9) and Section V-A are where this assumption enters.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The fused ExpMul operator is a genuinely neat idea: replace the separate exp-and-multiply path in FlashAttention-2's online update with an exponent-field decrement driven by a shift-add log2 approximation. The math in Section IV is internally consistent, and the hardware story is clear. The 28nm synthesis results — 28.8% area and ~17.8% power savings across hidden sizes — are plausible given that you are deleting FP multipliers and dedicated exp units. Credit is also due for the public HLS code and for the clear derivation of the operator from logarithmic quantization.

The soft spot is the accuracy evidence. Table I is one model (FLAN-T5) on GLUE, with no error bars, and the ExpMul variants occasionally drop a few points (e.g., MNLI-mm BF16 82.1 vs 83.3; CoLA BF16 69 vs 72). More importantly, the approximation is not a small perturbation. For x = -y, the operator produces 2^{-ceil(23y/16)}, which can be off by a factor anywhere from about 0.55 to 1.49 relative to e^{-y}. Because Algorithm 4 applies the quantized rescale inside the online maximum update, the error is path-dependent: ExpMul(a+b, V) is not the same as applying ExpMul(a, ·) then rescaling by ExpMul(b, ·), so the final attention weight of an early token depends on the sequence of maximum updates. The single FLAN-T5 run does not bound this kind of accumulation. The paper either needs an analytic error bound or much broader validation across model families and tasks, ideally with confidence intervals.

Minor issues: the abstract says 17.6% power savings while the body says 17.8%, and the GitHub repo has no commit hash, so exact replication is harder than it should be.

Overall, this is a solid hardware idea with a plausible efficiency story, but the accuracy-preservation claim is under-supported. It deserves a serious referee, and the authors should be asked to strengthen the validation before publication. I would send it to peer review, but I would not cite it as a reliable accuracy-preserving technique until that validation exists.

Referee Report

3 major / 5 minor

Summary. The paper proposes a fused exponential-vector multiplication (ExpMul) operator for FlashAttention-2 hardware accelerators. At the algorithm level, the output update in Alg. 2 is rewritten as two ExpMul calls (Eq. 5). At the hardware level, ExpMul approximates e^x V as 2^{-L} V, where L is obtained from a clipped fixed-point version of x using integer shift-and-add approximations of log2 e, and the multiplication is realized by subtracting L from the floating-point exponent of V. The authors implement ExpMul-based and baseline FlashAttention-2 kernels in HLS, synthesize them in a 28nm technology, and report average area and power reductions of 28.8% and 17.6-17.8%, respectively, while reporting comparable FLAN-T5 accuracy on GLUE.

Significance. If the accuracy claim holds, the proposal is attractive: it removes dedicated exponential units and floating-point multipliers from the attention output-update datapath and replaces them with clipping, integer shift-add arithmetic, and exponent-field adjustment. The derivation of the fused operation from the online softmax recurrence is clean, and the public HLS implementation supports reproducibility. The hardware savings are concrete and the paper honestly identifies the numerical approximation as the main risk. The central weakness is that the accuracy evidence is narrow: the approximation has per-factor errors up to roughly 49% and the online maximum-rescaling is not exactly multiplicative, so the one-model, one-benchmark validation is not yet sufficient to establish the paper's central claim that accuracy is not degraded.

major comments (3)

[Section V-A, Table I] The accuracy evidence consists of a single FLAN-T5 model evaluated once on GLUE, with no error bars, multiple seeds, or repetitions. Given that individual ExpMul factors can be off by as much as a factor of about 1.49 (about 49% upward) for values of x near the clip boundary, the small differences in Table I are not by themselves sufficient to establish that the approximation 'does not hinder' LLM capabilities. Please add multiple runs or seeds, report standard deviations or confidence intervals, and ideally evaluate on additional models or sequence lengths, or provide a formal bound on the resulting attention-output error.
[Alg. 4 and Eq. (5)] The online rescaling in the ExpMul version is not equivalent to exact FlashAttention rescaling. If q(x) = 2^{-L(x)} is the quantized factor, then q(a+b) is not equal to q(a) q(b); the recurrence o_i^* = q(m_{i-1}-m_i) o_{i-1}^* + q(s_i-m_i) v_i^* therefore produces attention weights that depend on the history of maximum updates, rather than only on s_i - m_N. This path-dependent error accumulation is not analyzed anywhere in the paper. Please quantify this effect, for example by comparing the approximate and exact attention matrices directly, or by giving a worst-case bound on the final output as a function of the per-call error.
[Section V-B, Figs. 3 and 4] The area and power comparison would be more convincing with a per-component breakdown. The baseline uses piecewise-linear exponent evaluation and separate floating-point multipliers, while the ExpMul design replaces them with clip/shift-add/exponent-decrement logic. Please report the area and power of the ExpMul unit versus the baseline exponent-plus-multiplier unit alone, and clarify whether the reported totals include the dot-product and final-division units. This will confirm that the 28.8% area and 17.6-17.8% power savings are due to the proposed fusion and not to particular baseline implementation choices.

minor comments (5)

[Eq. (9)] The exponent variable is written as E_A in the result of Eq. (9), while the operand is E_V; please make the notation consistent.
[Text after Eq. (9)] 'In case of an overflow' should be 'in case of an underflow', since subtracting a positive L from the exponent field can only push the value below the minimum representable exponent.
[Table I and contribution list] 'STS-2' in Table I appears to be a typo for 'SST-2', and 'GLEU' in the contribution list should be 'GLUE'.
[Abstract and Section V-B] The abstract reports power savings of 17.6%, while Section V-B reports 'over 17.8% on average'; please align these numbers.
[Section V-A] The sentence referring to 'reduced-precision models ('FP32, BF16-ExpMul')' is garbled; it should read 'FP32-ExpMul and BF16-ExpMul'.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central design depends on a small set of hand-chosen approximation parameters and on the empirical assumption that coarse power-of-two exponent approximation preserves LLM accuracy. The math itself is standard floating-point and shift-add arithmetic.

free parameters (3)

log2e_shift_approx = 1.4375 (1 + 1/2 - 1/16)
Hand-chosen shift-and-add approximation to log2(e), taken from prior work [26]; not fit to this paper's data, but it sets the per-element approximation error.
clip_bound = -15
Hand-chosen upper clipping bound for the exponential argument, because e^{-15} is about 3.1e-7 and assumed negligible; affects where the approximation saturates.
fixed_point_format = 16-bit, 6 integer bits and 10 fraction bits
Hand-chosen to cover the clipped range [-15,0] scaled by log2(e), giving [-21.64,0]; controls quantization loss in the ExpMul path.

assumptions (4)

domain assumption In FlashAttention-2, the ExpMul arguments m_{i-1}-m_i and s_i-m_i are always less than or equal to zero.
Used in Algorithm 4 and Section IV-B to restrict the exp input range for clipping; guaranteed by the definition of the running maximum, but it relies on the lazy softmax formulation.
standard math Floating-point multiplication by 2^{-L} can be replaced by an exponent-field decrement with no mantissa arithmetic and underflow set to zero.
Equation (9) and Algorithm 3 line 5; this is true in normal IEEE-754 ranges, with subnormal and overflow behavior approximated by zeroing.
domain assumption The log-quantized approximation of e^x is accurate enough for LLM inference.
Empirical assumption validated only on FLAN-T5 over GLUE in Section V-A; not proven across model families or tasks, and worst-case per-factor error can be large.
domain assumption The baseline design with separate exp and multiplication operators is a fair state-of-the-art comparison.
The area and power savings are measured against the authors' in-house implementation using piece-wise linear exp and FP multipliers; other state-of-the-art baselines might yield different savings.

invented entities (1)

ExpMul fused exponential-vector multiplication operator
purpose: Computes e^x times each element of a floating-point vector V using clipping, integer shift-add for L, and exponent-field decrement instead of a separate exp unit and FP multipliers.
Proposed hardware operator implemented in HLS and evaluated by synthesis, but not validated by independent silicon measurements or a third-party implementation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operators." pith.science (2026). https://pith.science/paper/SBTEUJCR

@misc{pith2026250514314,
  author       = {Pith},
  title        = {Pith review of: Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SBTEUJCR}},
  note         = {Machine review of arXiv:2505.14314}
}

read the original abstract

Attention mechanisms, particularly within Transformer architectures and large language models (LLMs), have revolutionized sequence modeling in machine learning and artificial intelligence applications. To compute attention for increasingly long sequences, specialized accelerators have been proposed to execute key attention steps directly in hardware. Among the various recently proposed architectures, those based on variants of the FlashAttention algorithm, originally designed for GPUs, stand out due to their optimized computation, tiling capabilities, and reduced memory traffic. In this work, we focus on optimizing the kernel of floating-point-based FlashAttention using new hardware operators that fuse the computation of exponentials and vector multiplications, e.g., e^x, V. The proposed ExpMul hardware operators significantly reduce the area and power costs of FlashAttention-based hardware accelerators. When implemented in a 28nm ASIC technology, they achieve improvements of 28.8% in area and 17.6% in power, on average, compared to state-of-the-art hardware architectures with separate exponentials and vector multiplications hardware operators.

Figures

Figures reproduced from arXiv: 2505.14314 by the authors.

**Figure 2.** The organization of the new hardware unit is the same [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 2.** The FlashAttention-2 kernel optimized with the proposed ExpMul [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Figures from the paper (2 more)

**Figure 3.** Figure 3: The hardware area at 28 nm for the FlashAttention-2 kernel, evaluated [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]

**Figure 4.** Figure 4: The average power consumption of the FlashAttention-2 kernel, [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 21 canonical work pages

[1]

Exploring the limits of transfer learning with a unified text-to-text transformer,

C. Raffelet al., “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020

work page 2020
[2]

Transformers in time series: A survey,

Q. Wenet al., “Transformers in time series: A survey,”arXiv preprint arXiv:2202.07125, 2022

arXiv 2022
[3]

Attention is all you need,

A. Vaswaniet al., “Attention is all you need,” inIntern. Conf. on Neural Information Processing Systems (NIPS), 2017, p. 6000–6010

work page 2017
[4]

Longformer: The long- document transformer,

I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long- document transformer,”arXiv preprint arXiv:2004.05150, 2020

arXiv 2004
[5]

Generating long sequences with sparse transformers,

R. Childet al., “Generating long sequences with sparse transformers,” arXiv preprint arXiv:1904.10509, 2019

arXiv 1904
[6]

Transformers are rnns: Fast autoregressive transformers with linear attention,

A. Katharopouloset al., “Transformers are rnns: Fast autoregressive transformers with linear attention,” inIntern. conference on machine learning, 2020, pp. 5156–5165

work page 2020
[7]

Linformer: Self-attention with linear complexity,

S. Wanget al., “Linformer: Self-attention with linear complexity,”arXiv preprint arXiv:2006.04768, 2020

arXiv 2006
[8]

A3: Accelerating attention mechanisms in neural net- works with approximation,

T. J. Hamet al., “A3: Accelerating attention mechanisms in neural net- works with approximation,” inIEEE Intern. Symp. on High-Performance Computer Architecture (HPCA), 2020, p. 328–341

work page 2020

Show all 31 references

[9]

Mnnfast: a fast and scalable system architecture for memory-augmented neural networks,

H. Janget al., “Mnnfast: a fast and scalable system architecture for memory-augmented neural networks,” inIntern. Symp. on Computer Architecture (ISCA), 2019, p. 250–263

2019
[10]

COSA plus: Enhanced co-operative systolic arrays for attention mechanism in transformers,

Z. Wang, G. Wang, and G. He, “COSA plus: Enhanced co-operative systolic arrays for attention mechanism in transformers,”IEEE Trans. on Computer-Aided Design of Integrated Circuits and Systems (TCAD), vol. 44, no. 2, p. 723–736, 2025

2025
[11]

Softermax: Hardware/software co-design of an efficient softmax for transformers,

J. R. Stevenset al., “Softermax: Hardware/software co-design of an efficient softmax for transformers,” inACM/IEEE Design Automation Conference (DAC), 2021, pp. 469–474

2021
[12]

Flashattention: Fast and memory-efficient exact attention with IO-awareness,

T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with IO-awareness,”Advances in neural information processing systems, vol. 35, pp. 16 344–16 359, 2022

2022
[13]

Flashattention-2: Faster attention with better parallelism and work partitioning,

T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,”arXiv preprint arXiv:2307.08691, 2023

2023 arXiv
[14]

Self-attention does not needO(n 2) memory,

M. N. Rabe and C. Staats, “Self-attention does not needO(n 2) memory,”arXiv preprint arXiv:2112.05682, 2021

2021 arXiv
[15]

Alexandridis, https://github.com/ic-lab-duth/Fused-ExpMul.git, 2025

K. Alexandridis, https://github.com/ic-lab-duth/Fused-ExpMul.git, 2025

2025
[16]

Glue: A multi-task benchmark and analysis platform for natural language understanding,

A. Wanget al., “Glue: A multi-task benchmark and analysis platform for natural language understanding,”arXiv preprint arXiv:1804.07461, 2018

2018 arXiv
[17]

A 95.6-TOPS/W deep learning inference accelerator with per-vector scaled 4-bit quantization in 5 nm,

B. Kelleret al., “A 95.6-TOPS/W deep learning inference accelerator with per-vector scaled 4-bit quantization in 5 nm,”IEEE Journal of Solid-State Circuits, vol. 58, no. 4, p. 1129–1141, 2023

2023
[18]

Hardware accelerator for multi-head attention and position- wise feed-forward in the transformer,

S. Luet al., “Hardware accelerator for multi-head attention and position- wise feed-forward in the transformer,” inIEEE Intern. System-on-Chip Conference (SOCC), 2020, pp. 84–89

2020
[19]

X-former: In-memory acceleration of transformers,

S. Sridharanet al., “X-former: In-memory acceleration of transformers,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 31, no. 8, pp. 1223–1233, 2023

2023
[20]

ELSA: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks,

T. J. Hamet al., “ELSA: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks,” inIntern. Symp. on Computer Architecture (ISCA), 2021, p. 692–705

2021
[21]

TSAcc: An efficient tempo-spatial similarity aware accelerator for attention acceleration,

Z. Songet al., “TSAcc: An efficient tempo-spatial similarity aware accelerator for attention acceleration,” inACM/IEEE Design Automation Conference, 2024

2024
[22]

Swifttron: An efficient hardware accelerator for quantized transformers,

A. Marchisioet al., “Swifttron: An efficient hardware accelerator for quantized transformers,” inIntern. Joint Conference on Neural Networks (IJCNN), 2023, pp. 1–9

2023
[23]

Online normalizer calculation for softmax,

M. Milakov and N. Gimelshein, “Online normalizer calculation for softmax,”arXiv preprint arXiv:1805.02867, 2018

2018 arXiv
[24]

Online alignment and addi- tion in multiterm floating-point adders,

K. Alexandridis and G. Dimitrakopoulos, “Online alignment and addi- tion in multiterm floating-point adders,”IEEE Trans. on VLSI Systems, vol. 33, no. 4, pp. 1182–1186, 2025

2025
[25]

Hardware-efficient softmax approximation for self- attention networks,

N. A. Kocaet al., “Hardware-efficient softmax approximation for self- attention networks,” inIntern. Symp. on Circuits and Systems (ISCAS), 2023, p. 1–5

2023
[26]

SOLE: hardware-software co-design of softmax and layernorm for efficient transformer inference,

W. Wanget al., “SOLE: hardware-software co-design of softmax and layernorm for efficient transformer inference,” inIEEE/ACM Intern. Conference on Computer Aided Design (ICCAD), 2023, p. 1–9

2023
[27]

Peano-vit: Power-efficient approximations of non- linearities in vision transformers,

M. E. Sadeghiet al., “Peano-vit: Power-efficient approximations of non- linearities in vision transformers,” inACM/IEEE Intern. Symposium on Low Power Electronics and Design (ISLPED), 2024, pp. 1–6

2024
[28]

A hardware accelerator for computing an exact dot product,

J. Koeniget al., “A hardware accelerator for computing an exact dot product,” inIEEE Symp. on Comp. Arith. (ARITH), 2017, pp. 114–121

2017
[29]

A study of bfloat16 for deep learning training,

D. Kalamkaret al., “A study of bfloat16 for deep learning training,” arXiv preprint arXiv:1905.12322, 2019

1905 arXiv
[30]

Promptbench: A unified library for evaluation of large language models,

K. Zhuet al., “Promptbench: A unified library for evaluation of large language models,”arXiv preprint arXiv:2312.07910, 2023

2023 arXiv
[31]

I-bert: Integer-only bert quantization,

S. Kimet al., “I-bert: Integer-only bert quantization,” inInternational conference on machine learning. PMLR, 2021, pp. 5506–5518

2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Exploring the limits of transfer learning with a unified text-to-text transformer,

C. Raffelet al., “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020

work page 2020

[2] [2]

Transformers in time series: A survey,

Q. Wenet al., “Transformers in time series: A survey,”arXiv preprint arXiv:2202.07125, 2022

arXiv 2022

[3] [3]

Attention is all you need,

A. Vaswaniet al., “Attention is all you need,” inIntern. Conf. on Neural Information Processing Systems (NIPS), 2017, p. 6000–6010

work page 2017

[4] [4]

Longformer: The long- document transformer,

I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long- document transformer,”arXiv preprint arXiv:2004.05150, 2020

arXiv 2004

[5] [5]

Generating long sequences with sparse transformers,

R. Childet al., “Generating long sequences with sparse transformers,” arXiv preprint arXiv:1904.10509, 2019

arXiv 1904

[6] [6]

Transformers are rnns: Fast autoregressive transformers with linear attention,

A. Katharopouloset al., “Transformers are rnns: Fast autoregressive transformers with linear attention,” inIntern. conference on machine learning, 2020, pp. 5156–5165

work page 2020

[7] [7]

Linformer: Self-attention with linear complexity,

S. Wanget al., “Linformer: Self-attention with linear complexity,”arXiv preprint arXiv:2006.04768, 2020

arXiv 2006

[8] [8]

A3: Accelerating attention mechanisms in neural net- works with approximation,

T. J. Hamet al., “A3: Accelerating attention mechanisms in neural net- works with approximation,” inIEEE Intern. Symp. on High-Performance Computer Architecture (HPCA), 2020, p. 328–341

work page 2020