Fused3S: Fast Sparse Attention on Tensor Cores

Aparna Chandramowlishwaran; Zitong Li

REVIEW 1 major objections 5 minor 50 references

Fused3S: Fast Sparse Attention on Tensor Cores

T0 review · 1 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fused3S claims the first tensor-core kernel that fuses all three sparse-attention steps into one on-chip pass, cutting kernel time up to 16x on H100 and A30 GPUs.

desk verdict The first fused tensor-core 3S kernel with a clever bitmap block format, but the missing accuracy validation for the fp16 path is the gap to close before acceptance. read the letter →

arxiv 2505.08098 v1 pith:P6B6MKMI submitted 2025-05-12 cs.DC cs.LG

Zitong Li , Aparna Chandramowlishwaran This is my paper · ORCID

classification cs.DCcs.LG

keywords sparseattentiontensorcoreskernelfusionSDDMMSpMMgraphtransformeronlinesoftmaxGPUacceleration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that the three sparse operations behind sparse attention—computing attention scores, row-normalizing them with softmax, and aggregating values—can be fused into a single GPU kernel that keeps data on chip and still uses tensor cores. The obstacle is that tensor cores expect dense, regularly shaped operand tiles, while graph sparsity is unstructured. The paper's answer is a block-sparse storage format whose 16x8 tiles mirror MMA operand shapes, combined with node-parallel tiling, online softmax, register remapping, and fp16/fp32 mixed precision. If the claim holds, sparse attention on GPUs no longer has to choose between tensor-core throughput and fused execution; the paper reports kernel-level speedups of 1.5x to 16.3x on H100 and A30 GPUs and end-to-end Graph Transformer inference speedups of 1.05x to 5.36x. The practical payoff would be faster and more memory-efficient graph and long-sequence models without changing the attention math.

What carries the argument

The central object is the Binary Sparse Block (BSB) format, a block-sparse layout that slices each row window of the attention mask into 16x8 tensor-core tiles and encodes each tile's nonzero pattern as a fixed-size bitmap. It is paired with TBGemm, a warp-level matrix-multiply primitive that issues MMA instructions with the left operand staged in shared memory and the right operand gathered directly from HBM into registers. Together they let SDDMM and SpMM share normalized scores in shared memory while online softmax keeps the computation numerically stable and fp32-accumulated, and row-window reordering plus register remapping handle load balance and memory coalescing on irregular graphs.

What would settle it

Run Fused3S and an fp32 reference implementation of the same 3S attention on one of the paper's graph datasets and compare normalized attention outputs or final model predictions; a relative error above roughly 1e-2, or a measurable drop in downstream accuracy, would show the mixed-precision pipeline is not accuracy-preserving.

Watch

Extended reading notes

Core claim

Fused3S is claimed to be the first fused 3S algorithm that simultaneously engages tensor cores and minimizes data movement. It treats the sparse attention pattern $O = \operatorname{softmax}(QK^T \odot A)V$ as one kernel: SDDMM, online softmax, and SpMM run in a single thread block per row window, with intermediate attention scores kept in registers and shared memory rather than written to global memory. The enabling object is the Binary Sparse Block format, which compacts columns within row windows and encodes each tensor-core tile's sparsity pattern as one bitmap, so the sparse mask contributes no per-nonzero index overhead. The kernel's speedups are attributed to this fusion plus row-window reordering, split-column warp partitioning, and register remapping for coalesced gathers. The intended conclusion is that unstructured sparsity is not inherently incompatible with tensor cores if the sparse format and scheduling are built around MMA operand shapes.

Load-bearing premise

The speedups only translate to usable acceleration if the fp16 mixed-precision pipeline, with fp32 accumulation and online softmax, keeps attention outputs accurate enough for the downstream model; the paper assumes this and reports no accuracy or numerical-error measurements.

Editorial extensions

If this is right

Fused3S removes the need to materialize the full attention-score matrix in global memory, so peak memory for sparse attention scales with the number of nonzeros rather than with $N^2$.
Any model whose attention is expressed as the 3S pattern, including graph attention networks, attention-based GNNs, and sparse transformers with dynamic masks, can use the kernel without changing the math.
On memory-constrained GPUs, the fused design completes graphs that separate-kernel baselines cannot fit, because the intermediate score matrix is never stored.
End-to-end Graph Transformer inference speeds up by 1.05x to 5.36x, with the largest gains when attention dominates the runtime.
The speedups are largest on graphs whose row windows are large enough to fill tensor-core tiles, while extremely skewed degree distributions leave smaller gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper asserts but does not measure that its fp16/fp32 pipeline preserves model accuracy; if fp16 normalized scores change predictions, the reported speedups would not transfer to usable inference, and an accuracy study on the same datasets is the natural next check.
Because the BSB format assumes binary sparsity, Fused3S exploits the fact that adjacency matrices and masks are 0/1; extending the bitmap idea to weighted or quantized sparse attention would change the storage and density trade-offs the paper analyzes.
The same fusion logic should carry over to the backward pass, where SDDMM and SpMM reappear in reverse order; the paper notes this but does not implement it, so a training-oriented extension is a testable prediction.
Row-window reordering helps only when imbalance comes from heavy row windows; on extremely skewed graphs the paper's own data show limited gains, suggesting finer-grained tiling or thread-block clusters would be needed, a direction the paper flags.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: Fused3S is a real contribution—the first fused tensor-core kernel for the 3S sparse attention pattern—but it ships without any accuracy validation for its mixed-precision path, and that is the one gap a referee should push on.

What is actually new: the paper fuses SDDMM, online softmax, and SpMM into a single kernel using tensor-core MMA instructions, with BSB, a bitmap-based block format that replaces integer indices with a 128-bit mask per 16x8 tile. That is a clean idea and it removes real overhead relative to ME-TCF. The node-parallel fusion with row-window reordering is sensible, the register-remapping trick for coalesced loads is well explained, and the evaluation is broad: 15 single-graph and 10 batched datasets on H100 and A30, with DF-GNN, FlashSparse, PyG, and DGL baselines. Kernel-level speedups are large and consistent across figures. The code is open source. Credit is due.

Soft spots, in order of importance.

First, the paper claims a mixed-precision pipeline 'balances performance and accuracy' without a single numerical experiment. Algorithm 1 casts normalized scores E to fp16 before the SpMM, and softmax is online rather than global. Both decisions can move results. The stress-test's subnormal argument is overstated—values around 1e-4 are still normal in fp16—but the missing measurement is not. A max-relative-error comparison against an fp32 reference on a few graphs, or one end-task accuracy number on, say, Cora or Reddit, would settle it. This is a required addition, not a nice-to-have.

Second, Section 4.2 claims Fused3S is the only kernel to run on IGB-large and Ogbn-papers100M but explicitly says 'results not shown.' Either show the runs or soften the claim. An unshown result is not a result.

Third, no repeated runs or error bars anywhere. Minor for a systems paper, but it would raise confidence.

The central argument holds up: the design is sound, the speedups are measured against external baselines, and the omissions are addressable. The audience is systems-for-ML researchers, GPU kernel developers, and anyone deploying sparse attention in GNNs or transformers. This is exactly the kind of paper that belongs in the peer-review process. Send it out; require the accuracy experiments and the large-graph data before sign-off. I'd bring it to reading group and would cite it for the BSB format and the fusion strategy.

Referee Report

1 major / 5 minor

Summary. The paper introduces Fused3S, a fused kernel for the 3S sparse-attention pattern (SDDMM, softmax, SpMM) on GPU tensor cores. It proposes a block-structured sparse format (BSB) aligned with MMA operand shapes, fuses the three operations in a single kernel using node-parallel execution with split-column warp partitioning, and uses a mixed-precision pipeline with fp16 operands, fp32 accumulation, and online softmax. The evaluation reports kernel-level speedups of 1.6–16.3x on H100 and 1.5–14x on A30 against DF-GNN, FlashSparse, and PyG across 15 single-graph and 10 batched-graph datasets, plus end-to-end Graph Transformer inference speedups of 1.05–5.36x over DGL, DF-GNN, and FlashSparse baselines.

Significance. If the reported speedups and the unvalidated accuracy assumption both hold, this is a useful systems contribution: it is, to my knowledge, the first fused 3S implementation that targets tensor cores, and the BSB format with a binary bitmap is a clean way to reduce index overhead for binary sparsity. The evaluation is broad and includes two GPU architectures, many datasets, and several baselines. The paper also ships open-source code, which is a strength. The main gap is the complete absence of numerical-accuracy or task-accuracy validation for the mixed-precision pipeline, which is central because the paper explicitly claims that its precision design 'balances performance and accuracy' without providing any measurement.

major comments (1)

[Section 3.5, Algorithm 1 line 19, Section 4] The claim that the mixed-precision design 'balances performance and accuracy' is unsupported by any measurement in the paper. The normalized attention scores E are cast to fp16 before the SpMM (Algorithm 1, line 19), and Table 5 lists E as 'fp32 → fp16'. This is a lossy step, and the paper itself cites Golden et al. [10] that online softmax can be less stable than the global variant. On high-degree graphs such as Reddit (Table 6, Table 7), many rows contain hundreds or thousands of attention entries, and individual normalized weights can be very small; in fp16 these fall into the subnormal range where relative error grows substantially. Section 4 reports only runtimes; there is no numerical comparison against an fp32 reference, no error metric for the kernel output, and no end-task accuracy (e.g., node classification) for the Graph Transformer experiments in Section 4.4. Since the paper's central claim is about practical acceleration of sparse attention, this omission is load-bearing: if the fp16 E cast or online softmax changes the output enough to affect model quality, the reported speedups would not translate to usable inference acceleration. I request at least: (i) maximum and mean absolute/relative error of the Fused3S output vs. a fp32 3S reference on a representative subset of the single-graph and batched datasets; (ii) end-task accuracy for the Graph Transformer with each 3S kernel; and (iii) a short discussion of the error behavior on the largest-degree graphs.

minor comments (5)

[Section 4.1] All performance figures report single-measurement runtimes without any repeated-run statistics. Please report at least 3 runs per configuration, or explicitly state that the kernel is deterministic and a single run is representative, with supporting evidence.
[Section 4.2] The statement 'On H100, Fused3S remains the only kernel to run on the largest graphs tested (IGB-large and Ogbn-papers100M, results not shown)' is unverifiable as written. Either add a small table with runtime and memory usage for those two graphs, or soften the claim to avoid relying on unreported data.
[Figures 5, 6, 8] The baseline names are inconsistently hyphenated: 'DF-GNN_tiling' in text, 'DF-GNN-tiling' in figure labels, and 'DF-GNN_Tiling' in one caption. Please unify the notation.
[Table 3] The symbol b_c is used for 'stored columns after compaction' but its scope (per row window vs. total over the whole matrix) is not defined precisely. A one-sentence clarification would prevent confusion.
[Algorithm 1, line 9] When t < W (fewer TCBs than warps), some warps will be idle; the paper does not state this behavior. Please add a sentence describing the sub-warp case and whether it affects the claimed performance on low-degree batched graphs.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Fused3S's speedups are empirical comparisons against external baselines; design choices are evaluated, not assumed.

full rationale

Fused3S makes no 'prediction' that is fitted from its own inputs. The central results—kernel-level speedups of 1.6–16.3x and 1.5–14x on H100 and A30, and end-to-end Graph Transformer speedups of 1.05–5.36x—are measured timings against external baselines (DF-GNN, FlashSparse, PyG, DGL), not derived from assumed constants or from a fitted parameter renamed as a prediction. Algorithm 1 computes softmax(QK^T ⊙ A)V using an online softmax adapted from FlashAttention-2 (an external, well-established result) and a BSB format built on ME-TCF/TCF (also external prior work). No target quantity is used to define an input parameter that is then 'predicted'; no author self-citation is load-bearing; no uniqueness theorem from the authors' prior work is invoked to forbid alternatives; and no known empirical pattern is merely renamed as an organization. The only potentially weak claim—that fp16 casting of normalized attention scores preserves accuracy—is a numerical-stability or correctness risk, not a circularity, because the paper does not use that assumption to derive its measured speedups. The validation is empirical and compares against external kernels, so the derivation is self-contained with respect to circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 1 invented entities

The central claim is an empirical performance claim. The only hand-chosen numeric design parameter is the TCB size; no constants are fitted to data. The paper relies on standard online softmax properties and hardware capabilities as assumptions. The BSB format is a newly introduced data structure without independent external evidence beyond this paper's experiments.

free parameters (1)

Tensor core block (TCB) dimensions (r, c) = 16x8
Chosen by hand as a compromise between tensor core utilization and sparsity density, aligned to supported mma tile shapes. Not fitted to data, but a design parameter that affects all reported performance.

assumptions (3)

standard math Online softmax computes the same row-wise softmax as the global max-stabilized form.
Invoked in Algorithm 1 lines 16-18, citing FlashAttention-2. Required for correctness of the fused blocking strategy across row-window sub-blocks.
domain assumption Tensor core mma instructions can load fp16 operands directly from HBM into registers without staging in shared memory.
Required by Algorithm 2 (TBGemm) lines 8-9, where right-hand-side operands in SDDMM and SpMM are gathered from global memory and consumed once. This is a hardware capability assumption for PTX mma on the tested GPUs.
domain assumption The measured GPU performance characteristics (FP16 tensor core throughput, memory bandwidth, SM counts) follow NVIDIA datasheet values.
Used in Section 2.2 and Section 4.1 to motivate the design and interpret speedups. The authors do not independently verify these hardware numbers.

invented entities (1)

Binary Sparse Block (BSB) format
purpose: Compact, bitmap-encoded sparse matrix representation aligned to tensor core operand shapes (16x8 blocks) to reduce indexing overhead and enable fusion.
The format's utility is demonstrated only within this paper's benchmarks; no external artifact or independent implementation is provided in the preprint. The claimed GitHub link is not verifiable from the arXiv version.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Fused3S: Fast Sparse Attention on Tensor Cores." pith.science (2026). https://pith.science/paper/P6B6MKMI

@misc{pith2026250508098,
  author       = {Pith},
  title        = {Pith review of: Fused3S: Fast Sparse Attention on Tensor Cores},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P6B6MKMI}},
  note         = {Machine review of arXiv:2505.08098}
}

abstract

Sparse attention is a core building block in many leading neural network models, from graph-structured learning to sparse sequence modeling. It can be decomposed into a sequence of three sparse matrix operations (3S): sampled dense-dense matrix multiplication (SDDMM), softmax normalization, and sparse matrix multiplication (SpMM). Efficiently executing the 3S computational pattern on modern GPUs remains challenging due to (a) the mismatch between unstructured sparsity and tensor cores optimized for dense operations, and (b) the high cost of data movement. Previous works have optimized these sparse operations individually or addressed one of these challenges. This paper introduces Fused3S, the first fused 3S algorithm that jointly maximizes tensor core utilization and minimizes data movement. Across real-world graph datasets, Fused3S achieves $1.6- 16.3\times$ and $1.5-14\times$ speedup over state-of-the-art on H100 and A30 GPUs. Furthermore, integrating Fused3S into Graph Transformer inference accelerates end-to-end performance by $1.05-5.36\times$, consistently outperforming all 3S baselines across diverse datasets (single and batched graphs) and GPU architectures.

Figures

Figures reproduced from arXiv: 2505.08098 by the authors.

**Figure 1.** Binary Sparse Block (BSB) format. The TCB size in this example is 4 × 2 while in practice the size is larger (i.e., 16×8). Red boxes highlight how the first row window in compacted, tiled, and stored in BSB format. ME-TCF uses two arrays to store non-zero elements: one for the number of non-zero elements in each TCB and another to store the local index of each nonzero element in all TCBs. We make the observation th… view at source ↗

**Figure 2.** Comparison of node-parallel (top) and edge [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Work partitioning among warps within a thread block. Top: split-column (column blocks of K 𝑇 and V are divided among warps). Each warp independently computes a 𝑟 ×𝑐 tile of S and O. Bottom: split-row (row blocks of K 𝑇 and V are divided among warps). All warps collaborate to compute each 𝑟 ×𝑐 tile of S and O. to the SIMT execution model of GPUs, enabling efficient parallel computation on independent tiles. 3.4 Data… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Register remapping in SDDMM (left) and SpMM (right). Top: original layouts. Bottom: permuted layouts. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: 3S kernel performance on single graph datasets in Table [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 6.** Figure 6: 3S kernel performance on batched graphs from LRGB [ [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

**Figure 7.** Figure 7: Comparison of SM active time on A30 with [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8: Graph Transformer inference performance with different 3S kernels. Missing bars indicate OOM. Labels [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 20 canonical work pages

[10]

Alicia Golden, Samuel Hsia, Fei Sun, Bilge Acun, Basil Hosmer, Yejin Lee, Zachary DeVito, Jeff Johnson, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. 2024. Is Flash Attention Stable? arXiv:2405.02803 [cs.LG] https://arxiv.org/abs/2405.02803

arXiv 2024
[1]

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020)

arXiv 2020
[2]

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Gener- ating Long Sequences with Sparse Transformers. CoRR abs/1904.10509 (2019). arXiv:1904.10509 http://arxiv.org/abs/1904.10509

arXiv 2019
[3]

Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Par- allelism and Work Partitioning. In The Twelfth International Confer- ence on Learning Representations . https://openreview.net/forum?id= mZn2Xyh9Ec

work page 2024
[4]

Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré
[5]

Vijay Prakash Dwivedi and Xavier Bresson. 2021. A Generalization of Transformer Networks to Graphs. AAAI Workshop on Deep Learning on Graphs: Methods and Applications (2021)

work page 2021
[6]

Vijay Prakash Dwivedi, Ladislav Rampášek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. 2022. Long Range Graph Benchmark. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview. net/forum?id=in7XC5RcjEn

work page 2022
[7]

Ruibo Fan, Wei Wang, and Xiaowen Chu. 2024. DTC-SpMM: Bridging the Gap in Accelerating General Sparse Matrix Multiplication with Tensor Cores. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 (ASPLOS ’24). Association for Computing Machinery, New York, NY, USA, 253–2...

arXiv 2024

Show all 50 references

[8]

Matthias Fey and Jan E. Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. InICLR Workshop on Representation Learning on Graphs and Manifolds

2019
[9]

Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. 2020. Sparse GPU kernels for deep learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (Atlanta, Georgia) (SC ’20). IEEE Press, Article 17, 14 pages

2020
[11]

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org

2016
[12]

Hamilton, Rex Ying, and Jure Leskovec

William L. Hamilton, Rex Ying, and Jure Leskovec. 2018. Inductive Representation Learning on Large Graphs. arXiv:1706.02216 [cs.SI] https://arxiv.org/abs/1706.02216

2018 arXiv
[13]

Yoonsang Han, Inseo Kim, Jinsung Kim, and Gordon Euhyun Moon
[14]

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020. Open Graph Benchmark: Datasets for Machine Learning on Graphs. arXiv preprint arXiv:2005.00687 (2020)

2020 arXiv
[15]

Eun-Jin Im, Katherine Yelick, and Richard Vuduc. 2004. Sparsity: Optimization framework for sparse matrix kernels. The International Journal of High Performance Computing Applications 18, 1 (2004), 135– 158

2004
[16]

Arpandeep Khatua, Vikram Sharma Mailthody, Bhagyashree Taleka, Tengfei Ma, Xiang Song, and Wen-mei Hwu. 2023. IGB: Addressing The Gaps In Labeling, Features, Heterogeneity, and Size of Public Graph Datasets for Deep Learning Research. In In Proceedings of the 29th ACM SIGKDD C...

2023 doi arXiv
[17]

Eunji Lee, Yoonsang Han, and Gordon Euhyun Moon. 2024. Acceler- ated Block-Sparsity-Aware Matrix Reordering for Leveraging Tensor Cores in Sparse Matrix-Multivector Multiplication. In Euro-Par 2024: Parallel Processing, Jesus Carretero, Sameer Shende, Javier Garcia-Blas, Ivona...

2024
[18]

Heejun Lee, Jina Kim, Jeffrey Willette, and Sung Ju Hwang. 2024. SEA: Sparse Linear Attention with Estimated Attention Mask. In The Twelfth International Conference on Learning Representations . https: //openreview.net/forum?id=JbcwfmYrob

2024
[19]

Jure Leskovec and Andrej Krevl. 2014. SNAP Datasets: Stanford Large Network Dataset Collection. http://snap.stanford.edu/data

2014
[20]

Shigang Li, Kazuki Osawa, and Torsten Hoefler. 2022. Efficient quan- tized sparse matrix operations on tensor cores. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (Dallas, Texas) (SC ’22). IEEE Press, Article 37,...

2022
[21]

Jiahui Liu, Zhenkun Cai, Zhiyong Chen, and Minjie Wang. 2024. DF-GNN: Dynamic Fusion Framework for Attention Graph Neural Networks on GPUs. In The Third Learning on Graphs Conference . https://openreview.net/forum?id=8GNDnBbUfF

2024
[22]

Liu Liu, Zheng Qu, Zhaodong Chen, Fengbin Tu, Yufei Ding, and Yuan Xie. 2022. Dynamic Sparse Attention for Scalable Transformer Acceleration. IEEE Trans. Comput. 71, 12 (2022), 3165–3178. doi:10. 1109/TC.2022.3208206

2022
[23]

Luis Müller, Mikhail Galkin, Christopher Morris, and Ladislav Ram- pášek. 2024. Attending to Graph Transformers. Transactions on Ma- chine Learning Research (2024). https://openreview.net/forum?id= HhbqHBBrfZ

2024
[24]

NVIDIA TESLA V100

NVIDIA Corporation 2018. NVIDIA TESLA V100 . NVIDIA Corpora- tion. https://images.nvidia.com/content/technologies/volta/pdf/tesla- volta-v100-datasheet-letter-fnl-web.pdf

2018
[25]

NVIDIA A30 TENSOR CORE GPU

NVIDIA Corporation 2022. NVIDIA A30 TENSOR CORE GPU . NVIDIA Corporation. https://www.nvidia.com/content/dam/en-zz/Solutions/ data-center/products/a30-gpu/pdf/a30-datasheet.pdf

2022
[26]

NVIDIA GH200 Grace Hopper Superchip

NVIDIA Corporation 2025. NVIDIA GH200 Grace Hopper Superchip . NVIDIA Corporation. https://resources.nvidia.com/en-us-grace-cpu/ grace-hopper-superchip?ncid=no-ncid

2025
[27]

Patrik Okanovic, Grzegorz Kwasniewski, Paolo Sylos Labini, Ma- ciej Besta, Flavio Vella, and Torsten Hoefler. 2024. High Per- formance Unstructured SpMM Computation Using Tensor Cores. arXiv:2408.11551 [cs.DC]

2024 arXiv
[28]

Meng Pang, Xiang Fei, Peng Qu, Youhui Zhang, and Zhaolin Li. 2024. A Row Decomposition-based Approach for Sparse Matrix Multiplication on GPUs (PPoPP ’24) . Association for Computing Machinery, New York, NY, USA, 377–389. doi:10.1145/3627535.3638470

2024
[29]

Khaledur Rahman, Majedul Haque Sujon, and Ariful Azad

Md. Khaledur Rahman, Majedul Haque Sujon, and Ariful Azad. 2021. FusedMM: A Unified SDDMM-SpMM Kernel for Graph Embedding and Graph Neural Networks . In 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE Computer Society, Los Alamitos, CA, USA,...

2021
[30]

Rossi and Nesreen K

Ryan A. Rossi and Nesreen K. Ahmed. 2015. The Network Data Repos- itory with Interactive Graph Analytics and Visualization. In AAAI. https://networkrepository.com

2015
[31]

Ahsan Shehzad, Feng Xia, Shagufta Abid, Ciyuan Peng, Shuo Yu, Dongyu Zhang, and Karin Verspoor. 2024. Graph Transformers: A Survey. arXiv:2407.09777 [cs.LG] https://arxiv.org/abs/2407.09777 Fused3S: Fast Sparse Attention on Tensor Cores ICS ’25, June 8–11, 2025, Salt Lake City...

2024
[32]

Jinliang Shi, Shigang Li, Youxuan Xu, Rongtian Fu, Xueying Wang, and Tong Wu. 2024. FlashSparse: Minimizing Computation Re- dundancy for Fast Sparse Matrix Multiplications on Tensor Cores. arXiv:2412.11007 [cs.DC] https://arxiv.org/abs/2412.11007

2024 arXiv
[33]

Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J Sutherland, and Ali Kemal Sinop. 2023. Exphormer: Sparse trans- formers for graphs. In International Conference on Machine Learning . arXiv:2303.06147

2023 arXiv
[34]

Kiran K Thekumparampil, Chong Wang, Sewoong Oh, and Li-Jia Li
[35]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. At- tention is all you need. Advances in neural information processing systems 30 (2017)

2017
[36]

Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. In International Conference on Learning Representations . https://openreview.net/forum?id=rJXMpikCZ

2018
[37]

Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, et al. 2019. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.01315 (2019)

2019 arXiv
[38]

Yuke Wang, Boyuan Feng, Zheng Wang, Guyue Huang, and Yufei Ding. 2023. TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUs. In 2023 USENIX Annual Technical Conference (USENIX ATC 23). USENIX Association, Boston, MA, 149–164. https: //www.usenix.org/conferenc...

2023
[39]

Weidele, Claudio Bellei, Tom Robinson, and Charles E

Mark Weber, Giacomo Domeniconi, Jie Chen, Daniel Karl I. Weidele, Claudio Bellei, Tom Robinson, and Charles E. Leiserson. 2019. Anti- Money Laundering in Bitcoin: Experimenting with Graph Convo- lutional Networks for Financial Forensics. arXiv:1908.02591 [cs.SI] https://arxiv....

2019 arXiv
[40]

Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. 2023. Flash- LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity. doi:10.48550/arXiv.2309. 10285 ar...

2023 doi arXiv
[41]

Cohen, and Ruslan Salakhutdinov

Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. 2016. Revis- iting semi-supervised learning with graph embeddings. In Proceedings of the 33rd International Conference on International Conference on Ma- chine Learning - Volume 48 (New York, NY, USA) (ICML’16). JMLR.org, 40–48

2016
[42]

Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: transformers for longer sequences. In Proceedings of the 34th International Conference on Neural Inf...

2020
[43]

Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. 2020. GraphSAINT: Graph Sampling Based Inductive Learning Method. arXiv:1907.04931 [cs.LG] https://arxiv. org/abs/1907.04931

2020 arXiv
[44]

Kaige Zhang, Xiaoyan Liu, Hailong Yang, Tianyu Feng, Xinyu Yang, Yi Liu, Zhongzhi Luan, and Depei Qian. 2024. Jigsaw: Accelerating SpMM with Vector Sparsity on Sparse Tensor Core. In Proceedings of the 53rd International Conference on Parallel Processing (ICPP ’24) . Associati...

2024
[45]

Meng Zhang, Jie Sun, Qinghao Hu, Peng Sun, Zeke Wang, Yonggang Wen, and Tianwei Zhang. 2024. TorchGT: A Holistic System for Large- Scale Graph Transformer Training. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis...

2024 arXiv
[46]

Haisha Zhao, San Li, Jiaheng Wang, Chunbao Zhou, Jue Wang, Zhikuang Xin, Shunde Li, Zhiqiang Liang, Zhijie Pan, Fang Liu, Yan Zeng, Yangang Wang, and Xuebin Chi. 2024. Acc-SpMM: Accelerating General-purpose Sparse Matrix-Matrix Multiplication with GPU Ten- sor Cores. arXiv:250...

2024 arXiv
[2018]

arXiv preprint arXiv:1803.03735 (2018)

Attention-based graph neural network for semi-supervised learning. arXiv preprint arXiv:1803.03735 (2018)

2018 arXiv
[2022]

arXiv:2205.14135 [cs.LG] https://arxiv.org/abs/2205

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135 [cs.LG] https://arxiv.org/abs/2205. 14135

arXiv
[2024]

Electronics 13, 20 (Jan

Tensor Core-Adapted Sparse Matrix Multiplication for Accel- erating Sparse Deep Neural Networks. Electronics 13, 20 (Jan. 2024),

2024
[3981]

doi:10.3390/electronics13203981 Number: 20 Publisher: Multi- disciplinary Digital Publishing Institute

doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [10]

Alicia Golden, Samuel Hsia, Fei Sun, Bilge Acun, Basil Hosmer, Yejin Lee, Zachary DeVito, Jeff Johnson, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. 2024. Is Flash Attention Stable? arXiv:2405.02803 [cs.LG] https://arxiv.org/abs/2405.02803

arXiv 2024

[2] [1]

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020)

arXiv 2020

[3] [2]

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Gener- ating Long Sequences with Sparse Transformers. CoRR abs/1904.10509 (2019). arXiv:1904.10509 http://arxiv.org/abs/1904.10509

arXiv 2019

[4] [3]

Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Par- allelism and Work Partitioning. In The Twelfth International Confer- ence on Learning Representations . https://openreview.net/forum?id= mZn2Xyh9Ec

work page 2024

[5] [4]

Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

[6] [5]

Vijay Prakash Dwivedi and Xavier Bresson. 2021. A Generalization of Transformer Networks to Graphs. AAAI Workshop on Deep Learning on Graphs: Methods and Applications (2021)

work page 2021

[7] [6]

Vijay Prakash Dwivedi, Ladislav Rampášek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. 2022. Long Range Graph Benchmark. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview. net/forum?id=in7XC5RcjEn

work page 2022

[8] [7]

Ruibo Fan, Wei Wang, and Xiaowen Chu. 2024. DTC-SpMM: Bridging the Gap in Accelerating General Sparse Matrix Multiplication with Tensor Cores. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 (ASPLOS ’24). Association for Computing Machinery, New York, NY, USA, 253–2...

arXiv 2024