Pith. sign in

REVIEW 3 major objections 5 minor 32 references

BSA: Ball Sparse Attention for Large-scale Geometries

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sparse attention for text is re-targeted to point clouds via ball-tree neighborhoods, matching Full Attention at sub-quadratic cost and 5x faster at length 65,536.

desk verdict The paper overstates its main efficiency claim: with fixed block size, the compressed and selection branches are quadratic, not sub-quadratic, so the headline novelty needs correction; the architecture itself is a reasonable integration worth reviewing. read the letter →

arxiv 2506.12541 v1 pith:WQK7ZX4N submitted 2025-06-14 cs.LG cs.AIcs.CV

Catalin E. Brita , Hieu Nguyen , Lohithsai Yadala Chanchu , Domonkos Nagy , Maksim Zhdanov This is my paper
classification cs.LGcs.AIcs.CV
keywords ballsparseattentionpointcloudsnativetreeirregulargeometriesairflowpressurepredictionsub-quadratic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a sparse-attention framework built for text, Native Sparse Attention (NSA), can be re-targeted to unordered 3D point clouds by replacing its sliding-window branch with Ball Tree Attention. The result, Ball Sparse Attention (BSA), claims accuracy close to Full Attention on an airflow pressure prediction task (MSE 14.31 vs 13.29) while using roughly a third of the FLOPs and running five times faster at a sequence length of 65,536. The authors argue this closes a gap for large-scale physical systems on irregular geometries, where quadratic self-attention is impractical but sparse methods assume regular sequence structure. A second experiment on stress-field prediction shows similar accuracy to Full Attention and Erwin, with the efficiency advantage smaller on that shorter-sequence benchmark.

What carries the argument

The central object is the Ball Tree, a spatial hierarchy whose leaf balls of fixed size hold the point tokens; Ball Tree Attention applies scaled dot-product attention within each ball, giving local geometric attention. BSA wraps this branch together with NSA-style compression and selection branches, combined by learned sigmoid gates. The selection branch uses coarse query and key vectors to build a similarity matrix, groups query positions contiguously, averages scores per group, and fetches the top-k coarse key-value blocks at full resolution for the whole group. This yields a global receptive field: local balls capture near interactions, selected blocks capture distant regions, and compressed coarse tokens supply a global summary.

What would settle it

Randomly permute the point indices of the serialized input while keeping the 3D coordinates fixed, then retrain BSA from scratch; if accuracy and runtime change materially relative to the original ordering, the contiguity assumption is doing unexplained work.

Watch

Extended reading notes

Core claim

BSA adapts NSA's three-branch design, compression, selection, and a local branch, to point clouds by replacing the sliding window with Ball Tree Attention from the Erwin Transformer, so local attention operates inside geometrically defined balls rather than sequence chunks. Selection is grouped: query indices are partitioned into contiguous groups, each group's scores are averaged, and one top-k set of key-value blocks is fetched for the whole group, cutting the number of top-k calls and improving memory access. An optional coarsening step pools queries before selection and compression, further reducing compute at a small accuracy cost. On ShapeNet-Car, BSA attains MSE 14.31 versus 13.29 for Full Attention and 15.85 for Erwin, with 27.91 GFLOPS versus 87.08 for Full Attention; on the Elasticity benchmark its RMSE is 0.38 versus 0.30 for Full Attention. A scaling study shows BSA overtakes Full Attention around sequence length 4096 and is five times faster at 65,536, although the absence of a specialized kernel leaves room for further speedups.

Load-bearing premise

The locality claim rests on group selection treating contiguous indices in the serialized point order as nearby points; for unordered point clouds that index order is not shown to reflect geometry, and if the ordering is arbitrary the selection branch may not actually exploit spatial locality.

Editorial extensions

If this is right

  • Sparse attention no longer requires a canonical sequence order: ball-tree spatial neighborhoods supply the locality structure that lets NSA-style branches operate on unordered point sets.
  • On large physical systems, users can get near-full-attention accuracy with sub-quadratic FLOPs and, at sequence lengths in the tens of thousands, a five-times runtime advantage over Full Attention.
  • Grouped selection, which cuts the number of top-k calls and fetches key-value blocks contiguously, offers a reusable template for lowering memory-access overhead in sparse attention beyond point clouds.
  • The accuracy-efficiency trade-off is tunable: enabling query compression (BSA with group compression) reduces runtime from 36.53 ms to 23.42 ms at a cost of 0.49 MSE on ShapeNet.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If query groups were formed by ball-tree proximity instead of serialized index contiguity, the selection branch should become invariant to the arbitrary ordering of points; this is a testable variant the paper does not explore.
  • The reported five-times speedup is achieved without a specialized GPU kernel, which the authors list as future work; a kernel would likely move the runtime crossover below sequence length 4096 and sharpen the efficiency claim.
  • Because the Elasticity result shows a much smaller gap to Full Attention at sequence length 972, the method's practical benefit appears to concentrate in long-sequence regimes; extending BSA to larger operator-learning benchmarks would test that reading.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Ball Sparse Attention (BSA), an attention mechanism for unordered point clouds that combines the ball-tree local attention of Erwin with the compression and selection branches of Native Sparse Attention (NSA). The method groups query positions into contiguous blocks, compresses keys and values into coarse tokens, and performs top-k selection based on coarse similarity scores, while also using local ball-tree attention. Experiments on the ShapeNet-Car airflow pressure task and the Elasticity stress-field benchmark show BSA achieving MSE 14.31 on ShapeNet (vs. 13.29 for Full Attention and 15.85 for Erwin in Table 1) and RMSE 0.38 on Elasticity (vs. 0.34 for Erwin). The paper claims a global receptive field at sub-quadratic cost and reports up to 5x runtime speedup over Full Attention at sequence length 65,536.

Significance. The central contribution is a sparse attention layer that retains a global receptive field via compression and selection while using local ball attention for fine-grained interactions. If the complexity claim were rigorously supported, BSA would be a useful building block for large-scale physical simulation on irregular geometries. The paper provides an open-source implementation, evaluates on two standard benchmarks, and compares against several published baselines, which are strengths. However, the sub-quadratic claim is not established by the presented equations, because the compression and selection branches have quadratic complexity for fixed block sizes. In addition, the locality-based grouping relies on a serialized point order that is never justified. These issues affect the paper's main efficiency claim and its methodological grounding, so the contribution is currently less robust than claimed.

major comments (3)
  1. [§2.2, Eqs. (13)-(15) and Appendix A, Table 4] The compression branch in Eq. (15) computes attention over Q_cmp, K_cmp, V_cmp, each of size ⌈N/ℓ⌉, yielding Θ((N/ℓ)^2) operations, and the selection branch in Eq. (14) forms an (N/ℓ)×(N/ℓ) similarity matrix. With the compression block size fixed at ℓ=8 (Appendix A, Table 4), both branches scale as Θ(N^2) as N grows. The abstract's claim of 'sub-quadratic cost' is therefore not supported by the provided complexity analysis. Please either state a scaling rule for ℓ, g, and k* that makes the method asymptotically sub-quadratic, or revise the claim to 'reduced-constant quadratic'. Because this is the paper's central efficiency claim, it must be corrected before acceptance.
  2. [§2.2, Eqs. (10)-(12)] The group selection defines groups G_p as contiguous index ranges in the serialized point order. For unordered point clouds, index contiguity does not imply spatial proximity, yet the paper never explains how the input points are ordered or why contiguous groups are spatially localized. If the point order is arbitrary, the 'locality-based sparsification' of the selection branch is not grounded, and the pooled query in Eq. (11) may average features from spatially distant points. Please clarify the ordering mechanism (e.g., ball-tree order, space-filling curve, or dataset-specific ordering) and provide an experiment or argument showing that the results are robust to the choice of ordering.
  3. [§3.2, Tables 1 and 3] Erwin's reported MSE differs between Table 1 (15.85) and Table 3 (16.12) for the same ShapeNet task, and no error bars, number of seeds, or significance tests are provided. The claim that BSA achieves accuracy 'comparable to Full Attention' rests on a single run with a 1.02 MSE gap (14.31 vs. 13.29), which may or may not be significant given the missing variance information. Please report repeated-run statistics (e.g., mean and standard deviation over several seeds) and reconcile the inconsistent Erwin baseline values across tables.
minor comments (5)
  1. [§3.2, Tables 1 and 3] The naming of the proposed method is inconsistent: Table 1 lists 'BSA (Ours)' while Table 3 uses 'BSA'; please unify the notation for clarity.
  2. [Figure 1] The label 'Groping' in Figure 1 is a typo and should read 'Grouping'.
  3. [§4, Conclusion] The concluding sentence says BSA 'drastically reducing computational efficiency'; this appears to be a typo for 'drastically reducing computational cost' and should be corrected.
  4. [Appendix B, Table 5] The configuration with compression block size 32 and group selection size 32 yields Val. MSE 132.14, which is dramatically worse than all other configurations; no explanation is provided, so please discuss whether this is an optimization failure, a numerical issue, or a genuine property of the method.
  5. [§3.3, Figure 2] The description of the receptive-field visualization is qualitative; please state whether the receptive field is computed analytically or measured empirically, and consider adding a quantitative metric to support the claim that the receptive field 'increases with more components'.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the accuracy results are empirical and externally benchmarked; the one co-authored citation (Erwin) is a building block, not a self-supporting proof.

full rationale

BSA is constructed by combining two published mechanisms: NSA's compression/selection branches and Erwin's ball-tree attention. The Erwin citation (Zhdanov et al., 2025) is co-authored by a present author, and it is load-bearing in the sense that BSA inherits the ball-tree structure from it; however, the paper's central claims are empirical and are tested against external data (ShapeNet-Car, Elasticity) and external baselines, including Erwin itself, so the self-citation does not substitute for evidence. No parameter is fitted to a subset and then reported as a prediction: the reported MSE/RMSE values come from training on the standard splits, and runtime/GFLOPS are measured, not derived from fitted constants. No uniqueness theorem or ansatz is imported from the authors' prior work; NSA is cited as prior art and its equations are reused transparently. The abstract's 'sub-quadratic cost' claim is questionable as a scaling statement because Eqs. 13-15 with fixed block size ell=8 define a compressed attention matrix of size (N/ell)x(N/ell), i.e., Theta(N^2) asymptotically; but that is a correctness/overstatement issue, not circularity, since the claim is not made true by definition or by self-citation. The paper also candidly notes missing Triton-kernel support and future ablations. Overall, the derivation chain is self-contained: one minor, non-load-bearing self-citation warrants a score of 1, not higher.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim rests on hand-chosen hyperparameters and two domain assumptions about point order and ball locality. No new physical entities are introduced.

free parameters (5)
  • Ball size m = 256
    Hand-chosen; controls local attention neighborhood in the ball tree, affecting accuracy and cost.
  • Compression block size l = 8
    Hand-chosen; Table 5 ablation shows strong sensitivity (l=32 gives MSE 132.14).
  • Selection block size = 8
    Hand-chosen; determines block granularity for top-k selection.
  • Number of selected blocks k* = 4
    Hand-chosen; controls global context amount; not ablated in the paper.
  • Group selection size g = 8
    Hand-chosen; Table 5 ablation shows 8 is best, but no justification for the value is given.
assumptions (3)
  • domain assumption Contiguous query groups in the serialized point order correspond to spatially local clusters
    Invoked in Eq. 10-12; for unordered point sets the serialization order is arbitrary, so this assumption may fail.
  • domain assumption Ball tree partitioning provides a useful geometric locality prior
    Assumed throughout; balls are treated as coherent neighborhoods without empirical verification.
  • domain assumption NSA's compression and selection branches transfer to point clouds
    BSA inherits NSA's design; assumes the learned sparsity pattern remains effective on irregular geometries.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BSA: Ball Sparse Attention for Large-scale Geometries." pith.science (2026). https://pith.science/paper/WQK7ZX4N

@misc{pith2026250612541,
  author       = {Pith},
  title        = {Pith review of: BSA: Ball Sparse Attention for Large-scale Geometries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQK7ZX4N}},
  note         = {Machine review of arXiv:2506.12541}
}
read the original abstract

Self-attention scales quadratically with input size, limiting its use for large-scale physical systems. Although sparse attention mechanisms provide a viable alternative, they are primarily designed for regular structures such as text or images, making them inapplicable for irregular geometries. In this work, we present Ball Sparse Attention (BSA), which adapts Native Sparse Attention (NSA) (Yuan et al., 2025) to unordered point sets by imposing regularity using the Ball Tree structure from the Erwin Transformer (Zhdanov et al., 2025). We modify NSA's components to work with ball-based neighborhoods, yielding a global receptive field at sub-quadratic cost. On an airflow pressure prediction task, we achieve accuracy comparable to Full Attention while significantly reducing the theoretical computational complexity. Our implementation is available at https://github.com/britacatalin/bsa.

Figures

Figures reproduced from arXiv: 2506.12541 by the authors.

Figure 1
Figure 1. Ball Sparse Attention (BSA) pipeline. A ball tree im￾poses spatial locality, then three sparse-attention branches: group￾ing (clustering points for shared selection), compression (MLP￾based token pooling), and selection (top-k block retrieval) operate alongside fine-grained Ball Tree Attention. A learnable gate fuses their outputs into the final attention. patterns (e.g., BigBird (Zaheer et al., 2020), where selecti… view at source ↗
Figure 2
Figure 2. Receptive field visualization of a car in the Shapenet dataset with different components: ball attention; ball attention and selection; ball attention, selection and compression. The receptive field increases with more components. values at discrete points on three-dimensional geometric structures. We use the ShapeNet-Car dataset (Umetani & Bickel, 2018) with 889 car models, each modeled with 3586 surface points in … view at source ↗
Figure 3
Figure 3. Runtime of BSA and Full attention with increasing se￾quence length (from 256 to 65536). away from the current point. Additionally, to prevent the model from over-fitting to local patterns already captured by the BTA, we mask all blocks within each ball, encouraging it to attend to more distant regions. Finally, with compression, it achieves a global receptive field by attending to all coarse key–value representation… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Runtime of BSA and its variants with increasing sequence length (from 256 to 32768). 7 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 19 canonical work pages

  1. [1]

    J., Bambrick, J., Bodenstein, S

    Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., Bodenstein, S. W., Evans, D. A., Hung, C.-C., O’Neill, M., Reiman, D., Tunyasuvunakool, K., Wu, Z., Žemgulytė, A., Arvaniti, E., Beattie, C., Bertolli, O., Bridgland, A., Cherepanov, A., Congreve, M., Cowen-Rivers, A. I., Co...

    2024

  2. [2]

    Universal physics transformers

    Alkin, B., Fürst, A., Schmid, S., Gruber, L., Holzleitner, M., and Brandstetter, J. Universal physics transformers. arXiv preprint arXiv:2402.12365, 2024

    arXiv 2024

  3. [3]

    Neuralcfd: Deep learning on high-fidelity automotive aerodynamics simulations

    Bleeker, M., Dorfer, M., Kronlachner, T., Sonnleitner, R., Alkin, B., and Brandstetter, J. Neuralcfd: Deep learning on high-fidelity automotive aerodynamics simulations. CoRR, abs/2502.09692, 2025. URL https://doi.org/10.48550/arXiv.2502.09692

    doi:10.48550/arxiv.2502.09692 2025

  4. [4]

    Geometrically aware transformer for point cloud analysis

    Chen, S., Fang, Z., Wan, S., Zhou, T., Chen, C., Wang, M., and Li, Q. Geometrically aware transformer for point cloud analysis. Scientific Reports, 15 0 (1), May 2025. ISSN 2045-2322. doi:10.1038/s41598-025-00789-7. URL http://dx.doi.org/10.1038/s41598-025-00789-7

    work page doi:10.1038/s41598-025-00789-7 2025

  5. [5]

    Octformer: Efficient octree-based transformer for point cloud compression with local enhancement

    Cui, M., Long, J., Feng, M., Li, B., and Kai, H. Octformer: Efficient octree-based transformer for point cloud compression with local enhancement. Proceedings of the AAAI Conference on Artificial Intelligence, 37: 0 470--478, 06 2023. doi:10.1609/aaai.v37i1.25121

    work page doi:10.1609/aaai.v37i1.25121 2023

  6. [6]

    Resolution-agnostic transformer-based climate downscaling

    Curran, D., Saleem, H., Hobeichi, S., and Salim, F. Resolution-agnostic transformer-based climate downscaling. In NeurIPS 2024 Workshop on Tackling Climate Change with Machine Learning, 2024. URL https://www.climatechange.ai/papers/neurips2024/74

    work page 2024

  7. [7]

    Flashattention: Fast and memory-efficient exact attention with i/o-awareness

    Dao, T., Fu, D., Ermon, S., Rudra, A., and R \'e , C. Flashattention: Fast and memory-efficient exact attention with i/o-awareness. In Advances in Neural Information Processing Systems, volume 35, pp.\ 16344--16359, 2022

    work page 2022

  8. [8]

    Gnot: A general neural operator transformer for operator learning

    Hao, Z., Ying, C., Wang, Z., Su, H., Dong, Y., Liu, S., Cheng, Z., Zhu, J., and Song, J. Gnot: A general neural operator transformer for operator learning. arXiv preprint arXiv:2302.14376, 2023

    arXiv 2023

Show all 32 references
  1. [9]

    Fourier neural operator for parametric partial differential equations, 2021

    Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. Fourier neural operator for parametric partial differential equations, 2021. URL https://arxiv.org/abs/2010.08895

    2021 arXiv

  2. [10]

    B., Choy, C

    Li, Z., Kovachki, N. B., Choy, C. B., Li, B., Kossaifi, J., Otta, S. P., Nabian, M. A., Stadler, M., Hundt, C., Azizzadenesheli, K., and Anandkumar, A. Geometry-informed neural operator for large-scale 3d pdes. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and ...

    2023

  3. [11]

    Li, Z., Meidani, K., and Farimani, A. B. Transformer for partial differential equations operator learning. Transactions on Machine Learning Research, 2023 b . ISSN 2835-8856. URL https://openreview.net/forum?id=EPPqt3uERT

    2023

  4. [12]

    Flatformer: Flattened window attention for efficient point cloud transformer

    Liu, Z., Yang, X., Tang, H., Yang, S., and Han, S. Flatformer: Flattened window attention for efficient point cloud transformer. pp.\ 1200--1211, 06 2023. doi:10.1109/CVPR52729.2023.00122

    2023

  5. [13]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    2019 arXiv

  6. [14]

    D2t-net: A dual-domain transformer network exploiting spatial and channel dimensions for semantic segmentation of urban mobile laser scanning point clouds

    Luo, Z., Zeng, Z., Wan, J., Tang, W., Jin, Z., Xie, Z., and Xu, Y. D2t-net: A dual-domain transformer network exploiting spatial and channel dimensions for semantic segmentation of urban mobile laser scanning point clouds. International Journal of Applied Earth Observation and...

    2024

  7. [15]

    Locality-sensitive hashing-based efficient point transformer with applications in high-energy physics

    Miao, S., Lu, Z., Liu, M., Duarte, J., and Li, P. Locality-sensitive hashing-based efficient point transformer with applications in high-energy physics. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  8. [16]

    Linear attention coupled fourier neural operator for simulation of three-dimensional turbulence

    Peng, W., Yuan, Z., Li, Z., and Wang, J. Linear attention coupled fourier neural operator for simulation of three-dimensional turbulence. Physics of Fluids, 35 0 (1): 0 015106, 01 2023

    2023

  9. [17]

    Pengmei, Z., Li, Z., chan Tien, C., Kondor, R., and Dinner, A. R. Transformers are efficient hierarchical chemical graph learners, 2023. URL https://arxiv.org/abs/2310.01704

    2023 arXiv

  10. [18]

    R., Su, H., Mo, K., and Guibas, L

    Qi, C. R., Su, H., Mo, K., and Guibas, L. J. Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv preprint arXiv:1612.00593, 2016

    2016 arXiv

  11. [19]

    GLU variants improve transformer

    Shazeer, N. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL https://arxiv.org/abs/2002.05202

    2002 arXiv

  12. [20]

    SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds, pp.\ 426--442

    Sun, P., Tan, M., Wang, W., Liu, C., Xia, F., and Leng, Z. SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds, pp.\ 426--442. 11 2022. ISBN 978-3-031-20079-3. doi:10.1007/978-3-031-20080-9_25

    2022 doi

  13. [21]

    and Bickel, B

    Umetani, N. and Bickel, B. Learning three-dimensional flow for interactive aerodynamic design. ACM Trans. Graph., 37 0 (4), July 2018

    2018

  14. [22]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS), pp.\ 6000--6010, 2017. URL https://pro...

    2017

  15. [23]

    and Wang, C

    Wang, T. and Wang, C. Latent neural operator for solving forward and inverse pde problems. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  16. [24]

    Solving high-dimensional pdes with latent spectral models

    Wu, H., Hu, T., Luo, H., Wang, J., and Long, M. Solving high-dimensional pdes with latent spectral models. In International Conference on Machine Learning, 2023

    2023

  17. [25]

    Transolver: A fast transformer solver for pdes on general geometries

    Wu, H., Luo, H., Wang, H., Wang, J., and Long, M. Transolver: A fast transformer solver for pdes on general geometries. In International Conference on Machine Learning, 2024 a

    2024

  18. [26]

    Point transformer v3: Simpler, faster, stronger

    Wu, X., Jiang, L., Wang, P.-S., Liu, Z., Liu, X., Qiao, Y., Ouyang, W., He, T., and Zhao, H. Point transformer v3: Simpler, faster, stronger. In CVPR, 2024 b

    2024

  19. [27]

    Point transformer v3: Simpler, faster, stronger

    Wu, X., Jiang, L., Wang, P.-S., Liu, Z., Liu, X., Qiao, Y., Ouyang, W., He, T., and Zhao, H. Point transformer v3: Simpler, faster, stronger. In CVPR, 2024 c

    2024

  20. [28]

    Improved operator learning by orthogonal attention

    Xiao, Z., Hao, Z., Lin, B., Deng, Z., and Su, H. Improved operator learning by orthogonal attention. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 , 2024

    2024

  21. [29]

    X., Wang, L., Xiao, Z., Wang, Y., Ruan, C., Zhang, M., Liang, W., and Zeng, W

    Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y. X., Wang, L., Xiao, Z., Wang, Y., Ruan, C., Zhang, M., Liang, W., and Zeng, W. Native sparse attention: Hardware-aligned and natively trainable sparse attention, 2025. URL https://arxiv.org/abs/2502.11089

    2025 arXiv

  22. [30]

    A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al

    Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33, 2020

    2020

  23. [31]

    and Sennrich, R

    Zhang, B. and Sennrich, R. Root Mean Square Layer Normalization . In Advances in Neural Information Processing Systems 32, Vancouver, Canada, 2019. URL https://openreview.net/references/pdf?id=S1qBAf6rr

    2019

  24. [32]

    Erwin: A tree-based hierarchical transformer for large-scale physical systems, 2025

    Zhdanov, M., Welling, M., and van de Meent, J.-W. Erwin: A tree-based hierarchical transformer for large-scale physical systems, 2025. URL https://arxiv.org/abs/2502.17019

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.