Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

ESPFormer: Doubly-Stochastic Attention with Expected Sliced Transport Plans

T0 review · 3 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Transformer attention mechanism built from sliced optimal transport enforces balanced, doubly-stochastic attention without Sinkhorn's iterative normalization.

desk verdict Genuinely new attention mechanism via expected sliced transport, but the headline efficiency claim is wrong as written and the 'doubly stochastic' label is off by a factor of N. read the letter →

arxiv 2502.07962 v2 pith:2JXJIV2Z submitted 2025-02-11 cs.LG

Ashkan Shahbazi , Elaheh Akbari , Darian Salehi , Xinran Liu , Navid Naderializadeh , Soheil Kolouri This is my paper · ORCID
classification cs.LG MSC 68T0749Q22
keywords doubly-stochasticattentionslicedoptimaltransportExpectedPlanssoftsortingregularizationTransformertransport-basedESPFormer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ESPFormer tries to show that doubly-stochastic attention—an attention matrix whose rows and columns each sum to one—can be obtained from sliced optimal transport instead of the iterative Sinkhorn normalization used by Sinkformer. It builds the attention matrix as an Expected Sliced Transport Plan (ESP): for each one-dimensional slice of the queries and keys, tokens are soft-sorted and matched, and the lifted matching matrices are averaged. Because the operation is fully parallel across slices and uses a temperature-based soft sorting for differentiability, it avoids the sequential row and column normalization that makes Sinkformer expensive. The paper reports that this mechanism improves accuracy over vanilla Transformer, DiffTransformer, and Sinkformer baselines on image classification, point cloud classification, sentiment analysis, and machine translation, while running faster than Sinkformer. If correct, this gives a cheaper, parallelizable way to impose balanced attention as an inductive bias.

What carries the argument

The central object is the Expected Sliced Transport Plan (ESP), defined as an aggregate of lifted one-dimensional optimal transport plans. For a slice direction, queries and keys are projected onto a line, soft-sorted to approximate the optimal matching, and the matching is encoded as a permutation-based transport matrix $U_\theta$; these matrices are combined as $G = \sum_l \sigma_l^\tau U_{\theta_l}$, where $\sigma^\tau$ is a softmax over per-slice transport costs controlled by an inverse temperature $\tau$. SoftSort supplies the differentiable relaxation of sorting, and the paper sets the slicer to the identity matrix $\Theta = I_m$, so each coordinate dimension acts as a slice. This construction turns attention into a doubly-stochastic transport plan in $O(mN^2)$ time with parallelism across slices, bypassing Sinkhorn's sequential normalization.

What would settle it

Run a controlled experiment where the true transport plan between query and key distributions is known to align with off-axis directions—for example, synthetic tokens whose relevant matching coordinate is a fixed rotation of the input axes—and compare ESPFormer's attention matrix against the exact optimal transport plan; if the matrix or downstream accuracy degrades measurably relative to Sinkhorn as the slice count grows, the axis-aligned assumption is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that queries and keys can be treated as two uniform empirical distributions, and that the doubly-stochastic attention matrix is a transportation plan between them. Instead of solving the entropy-regularized optimal transport problem with Sinkhorn iterations, ESPFormer computes, for each axis-aligned slice, the one-dimensional optimal matching via soft sorting, lifts each matching back to a permutation-based plan over the original tokens, and aggregates these plans with weights determined by the sliced transport cost and an inverse-temperature parameter. The aggregated matrix is doubly stochastic, differentiable through SoftSort, and parallelizable over slice dimensions. The authors argue that because the query and key projections are learned, the fixed axis-aligned slicer needs no learned slice parameters, and that the resulting attention mechanism produces balanced attention and better or comparable accuracy than Sinkformer and standard softmax attention across the tasks tested.

Load-bearing premise

The load-bearing premise is that fixed axis-aligned slicing directions are sufficient because the learned query and key projections can encode any useful slice orientation; if the optimal matching between queries and keys genuinely requires off-axis structure that the projections cannot capture, the ESP attention matrix stops being a good transport plan.

Editorial extensions

If this is right

  • ESPFormer enforces doubly-stochastic attention without iterative Sinkhorn normalization, with runtime $O(mN(N+d))$ and full parallelization across slices.
  • With temperature annealing during fine-tuning and a switch to hard sorting at inference, the attention matrix becomes exactly doubly stochastic and inference cost drops to $O(mN\log N)$.
  • Across image classification, point cloud classification, sentiment analysis, and neural machine translation, ESP attention improves or matches Transformer, DiffTransformer, and Sinkformer baselines, with the largest gains in low-data image classification settings.
  • ESP attention can be plugged into pre-trained Transformers and into differential attention architectures, improving accuracy after only a few epochs of fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If fixed axis-aligned slices truly suffice because learned projections adapt, the same ESP construction should transfer to cross-attention with unequal token counts via the paper's interpolation matrix, without adding learned slicing parameters.
  • The hard-sorting inference trick suggests a general recipe for other differentiable-sorting layers: train with annealed soft sorting, then switch to exact permutations at test time to gain speed and exactness.
  • The stated incompatibility with causal attention is structural, not merely practical; overcoming it would require relaxing either the lower-triangular constraint or the doubly-stochastic constraint, which the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes ESPFormer, a Transformer attention mechanism that replaces softmax or Sinkhorn normalization with an Expected Sliced Transport Plan (ESP) between the empirical distributions of queries and keys. For each of m axis-aligned slices, the projected queries and keys are soft-sorted, a per-slice transport plan is formed as (1/N) A_l^T B_l from the soft permutation matrices, and the plans are aggregated with inverse-temperature weights into an attention matrix G. The authors claim this yields a fully parallelizable doubly-stochastic attention mechanism with training complexity O(mN(N+d)) and, after annealing to hard sorting at inference, O(LN log N) complexity. Experiments on ModelNet40, IMDb, TweetEval, IWSLT14, Cats vs. Dogs, and MNIST report small but consistent accuracy improvements over the vanilla Transformer, Sinkformer, and DiffTransformer baselines, together with runtime comparisons against Sinkformer.

Significance. If the efficiency claim were correct, ESPFormer would be a valuable alternative to Sinkhorn-based doubly-stochastic attention, since it replaces sequential iterative normalization with parallel soft sorting over slices. The empirical study is broad, includes multiple architectures and domains, and the appendices provide implementation details, hyperparameters, and ablations; the axis-aligned slicing choice is also tested against learnable and frozen slicers in Table 6, which is a genuine strength. However, the paper's central efficiency claim is not supported by Algorithm 1 as written: forming the dense N x N transport plan A_l^T B_l for each slice costs O(N^3) per slice under standard matrix multiplication, so the training complexity is O(mN^3), not O(mN^2). In addition, the constructed attention matrix G has row and column sums 1/N rather than 1, so it is not doubly stochastic in the standard sense used by Sinkformer. These are load-bearing issues: the headline motivation is that ESPFormer is a fast, exact doubly-stochastic attention mechanism.

major comments (3)
  1. [Section 3.3 and Algorithm 1] The claimed complexity O(mN(N+d)) is not supported by the pseudocode. In Algorithm 1, Step 4 computes U_l = (1/N) A_l^T B_l, where A_l and B_l are dense N x N matrices output by SoftSort in Eq. (7); standard matrix multiplication for this product costs O(N^3) per slice, and no structure is specified that would reduce this cost. Steps 5 and 8 also operate on dense N x N matrices, so the training cost is O(mN^3), not O(mN^2). Because the paper's central motivation is that ESPFormer is computationally more efficient than Sinkformer (whose cost is O(S N^2)), this error directly undermines the main claim. Please either provide an implementation that avoids dense N x N transport-plan products per slice and analyze its cost, or correct the complexity statement, the runtime discussion, and Figure 3.
  2. [Section 3.2 and 'Transition to hard sorting'] The matrix G is not doubly stochastic in the standard sense. For hard sorting, each U_l = (1/N) A_l^T B_l is 1/N times a permutation matrix, so every row and column of U_l sums to 1/N, and the same holds for the convex combination G in Eq. (8). Thus G is a valid transport plan between uniform empirical measures, but its row and column sums are 1/N, in contrast to the doubly-stochastic attention matrices produced by Sinkformer, whose rows and columns sum to 1. The paper should either define what it means by 'doubly stochastic up to the 1/N scaling', use unnormalized permutation matrices for the attention map, or stop calling G an exact doubly-stochastic matrix.
  3. ['Transition to hard sorting' paragraph] The claimed inference-time reduction from O(N^2) to O(LN log N) is incomplete. Even with hard sorting, computing the attention output V G requires either constructing the dense N x N matrix G (which costs O(LN^2) to aggregate) or applying each slice's permutation to V separately and summing the results (O(LN d)); the latter procedure is not described in the paper. Please specify the exact operations and memory footprint for the hard-sorting inference mode, including how the 1/N scaling is handled.
minor comments (7)
  1. [Section 2.3] The text states that Sinkhorn's algorithm has 'an improved complexity of O(N^2 log N)', which is inconsistent with the O(SN^2) complexity stated in Section 1 and Section 3.3; each Sinkhorn iteration is O(N^2), so the total is O(S N^2).
  2. [Figure 2 caption] There is a typo: 'Sinkrhorn' should be 'Sinkhorn'.
  3. [Contributions bullet list] The bullet claiming a reduction 'from O(N^2) to O(LN log N)' is vague: sorting each slice is O(N log N), but the cost of forming or applying the attention matrix is not specified; please clarify the assumptions about whether the dense matrix G is materialized.
  4. [Table 6] The 'Frozen' slicer configuration is not defined in the main text or appendix; please specify how the frozen slicing directions are generated and whether they are random or fixed.
  5. [Section 4.2] The dataset is referred to as 'IMDB' in the text but 'IMDb' elsewhere; please use consistent spelling.
  6. [Impact Statement] The phrase 'improves model robustness and interoperability' appears to mean 'interpretability'; please correct the wording.
  7. [Algorithm 1] Algorithm 1 uses L = m slices because the slicer is set to the identity, but Section 3.2 introduces a general slice count L; please clarify that L = m in the implemented axis-aligned setting.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ESPFormer applies the authors' prior ESP framework to attention, and its central claims are empirical and benchmarked externally.

full rationale

ESPFormer builds on the Expected Sliced Transport Plan (ESP) of Liu et al. (2025), a paper co-authored by several of the present authors, and cites it for the definition of the lifted sliced transport plan and for the associated distance property. This is a genuine self-citation, but it is not circular: the cited result is a parameter-free mathematical construction from a prior, peer-reviewed paper, and the present paper's contribution is the application of that construction to Transformer attention. The attention matrix G is defined directly in Eqs. (7)-(12) as an explicit convex combination of soft-sorting transport plans; no fitted parameter is later re-labeled as a prediction, and no quantity in the derivation is defined in terms of the experimental outcomes it is claimed to explain. Hyperparameters such as the SoftSort temperature t and inverse temperature tau are tuned per dataset and reported in Appendix C, which is standard practice rather than a fitted-input-called-prediction loop. The ablations on learnable, frozen, and axis-aligned slicers and on L and tau are additional experiments, not post-hoc redefinitions of the method. The claimed O(mN(N+d)) training complexity has a potential internal inconsistency with Algorithm 1, where U_l = (1/N) A_l^T B_l multiplies two dense N x N soft-permutation matrices, which costs O(N^3) per slice under standard multiplication, and the 'doubly stochastic' terminology is non-standard since each U_l has row and column sums 1/N; however, these are correctness and consistency concerns, not circularity. The Limitations section's admission that the method is incompatible with causal attention likewise does not create a logical loop. In sum, no derivation step reduces to its own input, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method depends on the ESP framework from the authors' prior work (Liu et al., 2025) for the validity of the transport plan and the distance equivalence. It introduces hyperparameters tau and t, which are tuned per dataset. The assumption that uniform empirical measures with equal mass 1/N represent queries and keys, and that axis-aligned slices suffice, are unproven modeling choices. No new entities are introduced.

free parameters (3)
  • inverse temperature tau = 0.1 (ModelNet, Cats-Dogs, MNIST), 10.0 (IMDB, TweetEval, IWSLT)
    Controls the softmax concentration over slices in Eq. (9). Tuned per dataset via validation (Appendix C).
  • soft-sort temperature t = 1e-3 across all experiments
    Temperature in SoftSort (Eq. 7). Set to 1e-3 in Tables 8-14; annealed only in the hard-sorting fine-tuning experiments.
  • slice count L = L = m for axis-aligned; 1, 8, 32, 64, 128 in ablation
    Number of slicing directions. Fixed to the feature dimension for the main results, but varied in the slicer ablation (Table 6).
assumptions (5)
  • domain assumption The Expected Sliced Transport Plan (ESP) is a valid transport plan and its cost D_p is equivalent to the Wasserstein distance (Liu et al., 2025).
    The method's correctness rests on the ESP construction from the authors' earlier paper; this paper does not re-derive it.
  • domain assumption Queries and keys are represented as uniform empirical measures with equal mass 1/N (Section 3.2).
    The transport plan G is a coupling between these uniform measures, so row sums equal 1/N rather than 1; the paper nonetheless treats it as a standard attention matrix.
  • ad hoc to paper Axis-aligned slices (Theta = I) suffice because learned projections W_Q and W_K can adapt to a fixed slicing scheme (Section 3.2).
    The paper argues learning slices is unnecessary, but provides no proof that the projections can compensate for poor slice orientation.
  • standard math SoftSort provides a valid differentiable relaxation of the argsort operator (Prillo & Eisenschlos, 2020).
    Used to make the sorting operation differentiable; this is an existing result.
  • domain assumption Doubly stochastic attention improves information flow relative to softmax attention (Sander et al., 2022).
    The motivation for the entire line of work comes from Sinkformer's empirical and theoretical results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ESPFormer: Doubly-Stochastic Attention with Expected Sliced Transport Plans." pith.science (2026). https://pith.science/paper/2JXJIV2Z

@misc{pith2026250207962,
  author       = {Pith},
  title        = {Pith review of: ESPFormer: Doubly-Stochastic Attention with Expected Sliced Transport Plans},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2JXJIV2Z}},
  note         = {Machine review of arXiv:2502.07962}
}
read the original abstract

While self-attention has been instrumental in the success of Transformers, it can lead to over-concentration on a few tokens during training, resulting in suboptimal information flow. Enforcing doubly-stochastic constraints in attention matrices has been shown to improve structure and balance in attention distributions. However, existing methods rely on iterative Sinkhorn normalization, which is computationally costly. In this paper, we introduce a novel, fully parallelizable doubly-stochastic attention mechanism based on sliced optimal transport, leveraging Expected Sliced Transport Plans (ESP). Unlike prior approaches, our method enforces doubly stochasticity without iterative Sinkhorn normalization, significantly enhancing efficiency. To ensure differentiability, we incorporate a temperature-based soft sorting technique, enabling seamless integration into deep learning models. Experiments across multiple benchmark datasets, including image classification, point cloud classification, sentiment analysis, and neural machine translation, demonstrate that our enhanced attention regularization consistently improves performance across diverse applications. Our implementation code can be found at https://github.com/dariansal/ESPFormer.

Figures

Figures reproduced from arXiv: 2502.07962 by the authors.

Figure 1
Figure 1. An overview of the proposed ESP attention mechanism. By integrating the slicing operator into the key and query matrices, each dimension is treated as a learnable slice. For each slice, tokens are (soft) sorted, and a doubly-stochastic correspondence matrix is computed between the keys and queries. Finally, these correspondence matrices across all dimensions are aggregated to form a single doubly-stochastic attentio… view at source ↗
Figure 2
Figure 2. The attention weights between an example pair of keys (red) and queries (green) obtained by Sinkhorn’s algorithm (top row) with different numbers of iterations and by Expected Sliced Transport Plans (bottom row) with different inverse temperature values. Note that for Sinkrhorn and at zero iterations, i.e., S 0 , the computed attention reduces to classic self-attention. The weights are represented by the width of th… view at source ↗
Figure 3
Figure 3. Runtime analysis of ESPFormer versus Sinkformer with varying iteration counts (S) for sequence lengths N ∈ {50, 100, 500, 1000} with d = 1024, averaged over 10 runs.ESPFormer achieves superior computational efficiency across all sequence lengths with hard sorting, and outperforms Sinkformer for S > 3 under soft sorting, while maintaining model expressivity. . exhibits a runtime complexity of O((S + m)N2 ), where S d… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Runtime analysis of Transformer, DiffTransformer, ESPFormer, and Sinkformer with varying iteration counts (S) for sequence lengths N ∈ {50, 100, 500, 1000} with d = 1024, averaged over 10 runs. ESPFormer consistently demonstrates superior computational efficiency compa…
Figure 5
Figure 5. Figure 5: Comparison of MNIST test accuracy across different patch sizes for Transformer, DiffTransformer, Sinkformer, and ESPFormer architectures. Results are averaged over 3 runs. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Constrained Sliced Wasserstein Embedding

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Adding SWGG dissimilarity constraints to sliced Wasserstein embedding, trained via primal-dual optimization with a softsort relaxation, improves pooling accuracy on image, point cloud, and protein-sequence benchmarks.

Reference graph

Works this paper leans on

65 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    J., Bambrick, J., et al

    Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp.\ 1--3, 2024

    2024

  3. [3]

    TweetEval:Unified Benchmark and Comparative Evaluation for Tweet Classification

    Barbieri, F., Camacho-Collados, J., Espinosa-Anke, L., and Neves, L. TweetEval:Unified Benchmark and Comparative Evaluation for Tweet Classification . In Proceedings of Findings of EMNLP, 2020

    work page 2020

  4. [4]

    The iwslt 2014 evaluation campaign

    Cettolo, C., Niehues, M., and Federico, M. The iwslt 2014 evaluation campaign. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), 2014

    work page 2014

  5. [5]

    Rethinking attention with performers

    Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020

    arXiv 2009

  6. [6]

    Sinkhorn distances: Lightspeed computation of optimal transport

    Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013

    2013

  7. [7]

    Differentiable ranking and sorting using optimal transport

    Cuturi, M., Teboul, O., and Vert, J.-P. Differentiable ranking and sorting using optimal transport. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    work page 2019

  8. [8]

    Deshpande, I., Hu, Y.-T., Sun, R., Pyrros, A., Siddiqui, N., Koyejo, S., Zhao, Z., Forsyth, D., and Schwing, A. G. Max-sliced Wasserstein distance and its use for GANs . In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10648--10656, 2019

    work page 2019

Show all 65 references
  1. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021

    2021

  2. [10]

    P., Luu, A

    Dwivedi, V. P., Luu, A. T., Laurent, T., Bengio, Y., and Bresson, X. Graph neural networks with learnable structural and positional representations. arXiv preprint arXiv:2110.07875, 2021

    2021 arXiv

  3. [11]

    Stochastic optimization of sorting networks via continuous relaxations

    Grover, A., Wang, E., Zweig, A., and Ermon, S. Stochastic optimization of sorting networks via continuous relaxations. arXiv preprint arXiv:1903.08850, 2019

    1903 arXiv

  4. [12]

    R., and Hu, S.-M

    Guo, M.-H., Cai, J.-X., Liu, Z.-N., Mu, T.-J., Martin, R. R., and Hu, S.-M. Pct: Point cloud transformer. Computers & Graphics, 102: 0 1--13, 2021

    2021

  5. [13]

    Flatten transformer: Vision transformer using focused linear attention

    Han, D., Pan, X., Han, Y., Song, S., and Huang, G. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 5961--5971, 2023

    2023

  6. [14]

    J., Oktay, D., Lin, Z., Verkuil, R., Tran, V

    Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., et al. Simulating 500 million years of evolution with a language model. Science, pp.\ eads0018, 2025

    2025

  7. [15]

    Axial attention in multidimensional transformers

    Ho, J., Kalchbrenner, N., Weissenborn, D., and Salimans, T. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180, 2019

    1912 arXiv

  8. [16]

    Cross attention network for few-shot classification

    Hou, R., Chang, H., Ma, B., Shan, S., and Chen, X. Cross attention network for few-shot classification. Advances in neural information processing systems, 32, 2019

    2019

  9. [17]

    Highly accurate protein structure prediction with alphafold

    Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Z \' dek, A., Potapenko, A., et al. Highly accurate protein structure prediction with alphafold. nature, 596 0 (7873): 0 583--589, 2021

    2021

  10. [18]

    Kaggle. Dogs vs. cats. https://www.kaggle.com/c/dogs-vs-cats, 2013. Accessed: 2025-01-29

    2013

  11. [19]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp.\ 5156--5165. PMLR, 2020

    2020

  12. [20]

    W., Khan, F

    Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., and Shah, M. Transformers in vision: A survey. ACM computing surveys (CSUR), 54 0 (10s): 0 1--41, 2022

    2022

  13. [21]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6980

    2015 arXiv

  14. [22]

    Kolouri, S., Zou, Y., and Rohde, G. K. Sliced wasserstein kernels for probability distributions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 5258--5267, 2016

    2016

  15. [23]

    K., and Hoffmann, H

    Kolouri, S., Rohde, G. K., and Hoffmann, H. Sliced Wasserstein distance for learning Gaussian mixture models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 3427--3436, 2018

    2018

  16. [24]

    Generalized sliced Wasserstein distances

    Kolouri, S., Nadjahi, K., Simsekli, U., Badeau, R., and Rohde, G. Generalized sliced Wasserstein distances. Advances in neural information processing systems, 32, 2019

    2019

  17. [25]

    Gradient-based learning applied to document recognition

    LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 0 (11): 0 2278--2324, 1998

    1998

  18. [26]

    R., Choi, S., and Teh, Y

    Lee, J., Lee, Y., Kim, J., Kosiorek, A. R., Choi, S., and Teh, Y. W. Set transformer: A framework for attention-based permutation-invariant neural networks. In International Conference on Machine Learning, pp.\ 3744--3753. PMLR, 2019

    2019

  19. [27]

    Linear attention mechanism: An efficient attention for semantic segmentation

    Li, R., Su, J., Duan, C., and Zheng, S. Linear attention mechanism: An efficient attention for semantic segmentation. arXiv preprint arXiv:2007.14902, 2020

    2007 arXiv

  20. [28]

    Evolutionary-scale prediction of atomic-level protein structure with a language model

    Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379 0 (6637): 0 1123--1130, 2023

    2023

  21. [29]

    D., Bai, Y., Shahbazi, A., Thorpe, M., Aldroubi, A., and Kolouri, S

    Liu, X., Mart \' n, R. D., Bai, Y., Shahbazi, A., Thorpe, M., Aldroubi, A., and Kolouri, S. Expected sliced transport plans. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=P7O1Vt1BdU

    2025

  22. [30]

    Sparser is faster and less is more: Efficient sparse attention for long-range transformers

    Lou, C., Jia, Z., Zheng, Z., and Tu, K. Sparser is faster and less is more: Efficient sparse attention for long-range transformers. arXiv preprint arXiv:2406.16747, 2024

    2024 arXiv

  23. [31]

    L., Hunley, R., Chen, D., Y, N

    Maas, A. L., Hunley, R., Chen, D., Y, N. B. N., and Ng, A. Y. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL), pp.\ 142--150, 2011

    2011

  24. [32]

    Fast optimal transport through sliced generalized wasserstein geodesics

    Mahey, G., Chapel, L., Gasso, G., Bonet, C., and Courty, N. Fast optimal transport through sliced generalized wasserstein geodesics. In Thirty-seventh Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=n3XuYdvhNW

    2024

  25. [33]

    Martins, A. F. T. and Astudillo, R. F. From softmax to sparsemax: A sparse model of attention and multi-label classification. ArXiv, abs/1602.02068, 2016. URL https://api.semanticscholar.org/CorpusID:16432551

    2016 arXiv

  26. [34]

    Learning latent permutations with gumbel-sinkhorn networks

    Mena, G., Belanger, D., Linderman, S., and Snoek, J. Learning latent permutations with gumbel-sinkhorn networks. arXiv preprint arXiv:1802.08665, 2018

    2018 arXiv

  27. [35]

    F., Andrews, R., Hoffmann, H., and Kolouri, S

    Naderializadeh, N., Comer, J. F., Andrews, R., Hoffmann, H., and Kolouri, S. Pooling by sliced- Wasserstein embedding. Advances in Neural Information Processing Systems, 34: 0 3389--3400, 2021

    2021

  28. [36]

    Constrained sliced Wasserstein embedding

    NaderiAlizadeh, N., Salehi, D., Liu, X., and Kolouri, S. Constrained sliced Wasserstein embedding. arXiv preprint arXiv:2506.02203, 2025

    2025 arXiv

  29. [37]

    and Ho, N

    Nguyen, K. and Ho, N. Energy-based sliced wasserstein distance. Advances in Neural Information Processing Systems, 36, 2024

    2024

  30. [38]

    Hierarchical sliced Wasserstein distance

    Nguyen, K., Ren, T., Nguyen, H., Rout, L., Nguyen, T., and Ho, N. Hierarchical sliced Wasserstein distance. arXiv preprint arXiv:2209.13570, 2022

    2022 arXiv

  31. [39]

    Ott, M., Edunov, S., Grangier, D., and Le, Q. V. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019

    2019

  32. [40]

    Computational optimal transport: With applications to data science

    Peyr \'e , G., Cuturi, M., et al. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning , 11 0 (5-6): 0 355--607, 2019

    2019

  33. [41]

    and Eisenschlos, J

    Prillo, S. and Eisenschlos, J. Softsort: A continuous relaxation for the argsort operator. In International Conference on Machine Learning, pp.\ 7793--7802. PMLR, 2020

    2020

  34. [42]

    A general approximation framework for direct optimization of information retrieval measures

    Qin, T., Liu, T.-Y., and Li, H. A general approximation framework for direct optimization of information retrieval measures. Information retrieval, 13: 0 375--397, 2010

    2010

  35. [43]

    Geometric transformer for fast and robust point cloud registration

    Qin, Z., Yu, H., Wang, C., Guo, Y., Peng, Y., and Xu, K. Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 11143--11152, 2022

    2022

  36. [44]

    Wasserstein barycenter and its application to texture mixing

    Rabin, J., Peyr \'e , G., Delon, J., and Bernot, M. Wasserstein barycenter and its application to texture mixing. In Scale Space and Variational Methods in Computer Vision: Third International Conference, SSVM 2011, Ein-Gedi, Israel, May 29--June 2, 2011, Revised Selected Pape...

    2011

  37. [45]

    P., Luu, A

    Ramp \'a s ek, L., Galkin, M., Dwivedi, V. P., Luu, A. T., Wolf, G., and Beaini, D. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35: 0 14501--14515, 2022

    2022

  38. [46]

    L., Ma, J., et al

    Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118 0 (...

    2021

  39. [47]

    E., Ablin, P., Blondel, M., and Peyr \'e , G

    Sander, M. E., Ablin, P., Blondel, M., and Peyr \'e , G. Sinkformers: Transformers with doubly stochastic attention. In International Conference on Artificial Intelligence and Statistics, pp.\ 3515--3530. PMLR, 2022

    2022

  40. [48]

    J., and Sinop, A

    Shirzad, H., Velingker, A., Venkatachalam, B., Sutherland, D. J., and Sinop, A. K. Exphormer: Sparse transformers for graphs. In International Conference on Machine Learning, pp.\ 31613--31632. PMLR, 2023

    2023

  41. [49]

    A relationship between arbitrary positive matrices and doubly stochastic matrices

    Sinkhorn, R. A relationship between arbitrary positive matrices and doubly stochastic matrices. The annals of mathematical statistics, 35 0 (2): 0 876--879, 1964

    1964

  42. [50]

    Softrank: optimizing non-smooth rank metrics

    Taylor, M., Guiver, J., Robertson, S., and Minka, T. Softrank: optimizing non-smooth rank metrics. In Proceedings of the 2008 International Conference on Web Search and Data Mining, pp.\ 77--86, 2008

    2008

  43. [51]

    Training data-efficient image transformers & distillation through attention

    Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jegou, H. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pp.\ 10347--10357, July 2021 a

    2021

  44. [52]

    Going deeper with image transformers

    Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., and J\'egou, H. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 32--42, October 2021 b

    2021

  45. [53]

    Deit iii: Revenge of the vit

    Touvron, H., Cord, M., and Jegou, H. Deit iii: Revenge of the vit. arXiv preprint arXiv:2204.07118, 2022

    2022 arXiv

  46. [54]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Informatio...

    2017

  47. [55]

    Z., Khabsa, M., Fang, H., and Ma, H

    Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  48. [56]

    3d shapenets: A deep representation for volumetric shapes

    Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 1912--1920, 2015

    1912

  49. [57]

    and Chen, H

    Xu, Y. and Chen, H. Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 21241--21251, 2023

    2023

  50. [58]

    Differential transformer

    Ye, T., Dong, L., Xia, Y., Sun, Y., Zhu, Y., Huang, G., and Wei, F. Differential transformer. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OvoCm1gGhN

    2025

  51. [59]

    Do transformers really perform badly for graph representation? Advances in neural information processing systems, 34: 0 28877--28888, 2021

    Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., and Liu, T.-Y. Do transformers really perform badly for graph representation? Advances in neural information processing systems, 34: 0 28877--28888, 2021

    2021

  52. [60]

    Yun, S., Jeong, M., Kim, R., Kang, J., and Kim, H. J. Graph transformer networks. Advances in neural information processing systems, 32, 2019

    2019

  53. [61]

    Alignment attention by matching key and query distributions

    Zhang, S., Fan, X., Zheng, H., Tanwisuth, K., and Zhou, M. Alignment attention by matching key and query distributions. Advances in Neural Information Processing Systems, 34: 0 13444--13457, 2021

    2021

  54. [62]

    Fspool: Learning set representations with featurewise sort pooling

    Zhang, Y., Hare, J., and Prügel-Bennett, A. Fspool: Learning set representations with featurewise sort pooling. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=HJgBA2VYwH

    2020

  55. [63]

    W., Lacoste-Julien, S., Burghouts, G

    Zhang, Y., Zhang, D. W., Lacoste-Julien, S., Burghouts, G. J., and Snoek, C. G. Unlocking slot attention by changing optimal transport costs. In International Conference on Machine Learning, pp.\ 41931--41951. PMLR, 2023

    2023

  56. [64]

    H., and Koltun, V

    Zhao, H., Jiang, L., Jia, J., Torr, P. H., and Koltun, V. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 16259--16268, 2021

    2021

  57. [65]

    and Zhou, M

    Zheng, H. and Zhou, M. Comparing probability distributions with conditional transport. stat, 1050: 0 9, 2021

    2021

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.