Pith. sign in

REVIEW 4 major objections 5 minor 97 references

Parallel Sequence Modeling via Generalized Spatial Propagation Network

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GSPN claims that row-by-row 2D propagation can replace self-attention for images, cutting the effective sequence length to $\sqrt{N}$ and enabling 16K generation at up to 84x faster inference.

desk verdict Strong empirical architecture with a broken stability proof; referee it but require fixes. read the letter →

arxiv 2501.12381 v1 pith:BIN6XPLA submitted 2025-01-21 cs.CV cs.LG

Hongjun Wang , Wonmin Byeon , Jiarui Xu , Jinwei Gu , Ka Chun Cheung , Xiaolong Wang , Kai Han , Jan Kautz
show 1 more author
Sifei Liu
This is my paper · ORCID
classification cs.CVcs.LG
keywords generalizedspatialpropagationnetwork2Dlinearstability-contextconditionattentionstate-spacemodelsimageclassificationtext-to-imagegenerationhigh-resolutiondiffusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that images do not need to be flattened into 1D token strings for sequence modeling: a 2D linear propagation scheme that scans rows and columns can play the role of self-attention in vision networks. Its central claim is that the Generalized Spatial Propagation Network (GSPN), built on tridiagonal row-stochastic propagation matrices and a four-direction line scan, keeps propagation stable and context-aware while cutting the effective sequence length of an $N$-pixel square map to $\sqrt{N}$. If that is right, GSPN is a drop-in attention replacement that works without positional embeddings, scales to 16K-resolution image generation on a single GPU, and beats or matches transformer, linear-attention, and Mamba baselines on ImageNet classification and diffusion-based generation. A sympathetic reader would take the main empirical thesis as: long-range, spatially coherent context can be obtained at sub-quadratic cost by propagating along rows rather than over the whole token set.

What carries the argument

The Stability-Context Condition is the load-bearing mechanism: each propagation matrix $w_\tau$ is made row-stochastic by sigmoid activation followed by row normalization, and the condition asserts that products $W_{ij} = \prod_{\tau=j+1}^{i} w_\tau$ are then row-stochastic (context) with operator norm bounded by 1 (stability). The architecture uses a 3-way tridiagonal connection so that multiplying such matrices over many rows fills in off-diagonal entries and produces dense affinity; the four scanning directions make the pairwise graph complete. This machinery lets GSPN process rows in parallel, reducing the effective sequence length to $\sqrt{N}$, and a learnable merger aggregates the directional information.

What would settle it

Train any GSPN variant and inspect the learned tridiagonal propagation matrices: the Stability-Context Condition predicts that every row-stochastic weight matrix has largest singular value at most 1, in every direction and group configuration. Finding even one trained matrix whose largest singular value exceeds 1 would show that row-stochasticity alone does not deliver the stated stability guarantee, and long-range propagation could amplify rather than preserve signals.

Watch

Extended reading notes

Core claim

The discovery, on the paper's own terms, is that a sequence of tridiagonal row-stochastic matrices multiplied together becomes dense while staying row-stochastic, so a row-by-row scan can deliver dense pairwise interactions with only $\mathcal{O}(\sqrt{N})$ sequential steps. GSPN writes each output row as a weighted sum of the current input row and three neighbors in the previous row; because the weights are sigmoid-activated and row-normalized, the accumulated propagation matrices remain row-stochastic. With left-to-right, right-to-left, top-to-bottom, and bottom-to-top scans, each pixel receives connections from all pixels, and a learnable linear merger combines the four directional affinity maps. The paper reports that this design reaches 83.0% ImageNet top-1 accuracy at the tiny scale, 84.3% at base scale, FID 15.26 for GSPN-XL/2 class-conditional generation, and roughly 84x faster SD-XL text-to-image generation at 16K resolution.

Load-bearing premise

The load-bearing premise is that row-normalizing the three-way propagation weights keeps each step from amplifying the hidden state—that is, that the largest singular value of the weight matrix is at most 1—so long-range context accumulates stably.

Editorial extensions

If this is right

  • GSPN can replace self-attention in vision architectures without positional embeddings, so the quadratic cost of softmax attention disappears while two-dimensional spatial structure is preserved.
  • Effective sequence length drops from $N$ to $\sqrt{N}$ for square maps, and local GSPN reaches $\mathcal{O}(1)$ per-pixel work when each row is its own group, which is what makes 16K-resolution generation possible on a single A100.
  • Reported results would make GSPN the strongest sub-quadratic attention alternative on ImageNet classification (83.0% top-1 at tiny scale) and class-conditional diffusion generation (FID 15.26), using fewer parameters than DiT-based rivals at matching FID.
  • Because products of row-stochastic matrices are row-stochastic, inference adapts to unseen resolutions without extra normalization, avoiding the resolution-specific tuning that raster-scan state-space baselines need.
  • The four-direction line scan plus learnable merger yields dense pairwise connections, so long-range context is available for both classification and generation, not only local texture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the same scan logic applies to any regular grid, a natural extension is to video or volumetric data: stacking planes or frames would reduce the effective sequence length to roughly $N^{1/3}$ for a cube, amplifying the efficiency gain.
  • If a future version replaces the eigenvalue-based stability argument with a direct bound on the largest singular value of row-stochastic tridiagonal matrices, the theoretical foundation would extend cleanly to other propagation schemes and norms.
  • A testable extension of the stability story is to monitor the largest singular value of trained propagation matrices across layers and directions; if the condition is doing the work, it should remain at or below 1 throughout training.
  • The equivalence in Eq. (5) positions GSPN between causal linear attention and 2D recurrent state-space models, suggesting hybrid designs that use row scans for spatial axes and a separate temporal scan for video or long-context multimodal input.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the Generalized Spatial Propagation Network (GSPN), a vision-oriented attention mechanism based on 2D linear propagation with learnable, input-dependent, row-stochastic tridiagonal weights. The authors propose a 'Stability-Context Condition' that, they claim, guarantees stable propagation and dense long-range pairwise connections while reducing the effective sequence length to sqrt(N) for an N-element square map. GSPN is evaluated on ImageNet classification, class-conditional ImageNet generation with DiT-like architectures, and text-to-image generation based on SD-v1.5 and SD-XL. The paper reports state-of-the-art or competitive results (e.g., GSPN-T 83.0% top-1, GSPN-XL/2 FID 15.26) and an over 84x speedup for SD-XL at 16K resolution.

Significance. If the theoretical and empirical claims hold, GSPN would be a practical, sub-quadratic alternative to softmax attention that respects 2D spatial structure, with demonstrated benefits in high-resolution generation. The paper's strengths include a broad experimental evaluation across discriminative and generative tasks, a custom CUDA implementation with reported speed measurements, and a detailed complexity analysis. The central theoretical guarantee, however, rests on a stability proof that is currently incorrect as written, and the headline speedup conflates multiple techniques. The significance of the empirical results is high, but the theoretical foundation needs repair before the claims can be accepted.

major comments (4)
  1. [Supplementary Section 7.2, Theorem 4] The proof of Theorem 4 uses Gershgorin's Circle Theorem to conclude that for a row-stochastic matrix w_i, the largest singular value satisfies sigma_max <= 1. Gershgorin's theorem bounds eigenvalues, not singular values, and the conclusion is false in general: a non-normal row-stochastic matrix such as [[0,1],[0.5,0.5]] has spectral norm approximately 1.144 > 1. Consequently, the stated norm condition ||w_i|| <= sigma_max <= 1 is not established, and the stability guarantee for the recurrence in Eq. (7) is unproven. The theorem may be repairable by using the infinity norm, since every row-stochastic matrix has unit infinity norm and products of row-stochastic matrices remain row-stochastic with unit infinity norm, but the proof as written is invalid.
  2. [Section 3.2, Theorem 2] Theorem 2's proof consists solely of a citation to [62] ('Making w_t row stochastic is a sufficient condition to ensure stability presented in [62]'). The paper's central claim relies on this theorem, and the supplementary material purported to provide comprehensive proofs; a self-contained derivation is needed. The later attempt in Section 7.2 does not fill this gap because it contains the flawed Gershgorin-to-singular-value step identified above.
  3. [Abstract and Section 5.6] The abstract and Section 5.6 state that 'GSPN accelerates SD-XL with softmax-attention by over 84x when generating 16K images.' However, the reported 84x speedup is obtained by building on the DemoFusion pipeline and additionally skipping 60% of the initial denoising steps in the high-resolution stage. These are separate modifications, not intrinsic to replacing softmax attention with GSPN. The paper should report an ablation that isolates the speedup attributable to GSPN alone, and the abstract should be rephrased to avoid attributing the entire speedup to the GSPN mechanism.
  4. [Section 5.3, Table 2] The text states that 'GSPN-B/2 achieves competitive performance at 20.3% of DiT-XL/2's parameter count.' In Table 2, GSPN-B/2 has FID 28.70, which is substantially worse than DiT-XL/2's FID 20.05 (and worse than all other listed models). Calling this 'competitive' is misleading; the claim should be limited to parameter efficiency, not performance parity.
minor comments (5)
  1. [Section 3.3] The phrase 'verse-visa' should be 'vice versa'.
  2. [Supplementary Section 7.2, Eq. (10)] The index 'k ∈ N' in Eq. (10) is ambiguous; it should be clarified as a neighborhood of pixel i (e.g., the three connected pixels in the previous row).
  3. [Abstract and Section 3.2] The phrase 'reduces the effective sequence length to sqrt(N)' is imprecise: the total number of elements is still N, and the total work is O(Nd). What is reduced is the sequential depth (number of propagation steps), which enables parallel row-wise processing. Consider rewording to 'reduces the sequential propagation depth to sqrt(N)'.
  4. [Section 5.4, Table 3] The comparison with Linfusion states that GSPN 'matches the performance of Linfusion without using any pretrained weights,' but SD-v1.5-GSPN w/o init has FID 36.89, which is slightly worse than Linfusion's 36.33. The wording should be adjusted to reflect that the gap is small but non-negligible.
  5. [Supplementary Section 9, Algorithm 1] The CUDA kernel description would benefit from a note on how the row-stochastic constraint is enforced in the gate tensors G1, G2, G3 during the forward pass, since the main text describes the sigmoid-and-normalize procedure in Eq. (6) but the algorithm listing does not show it.

Circularity Check

1 steps flagged · score 4.0 of 10

The Stability-Context Condition's stability guarantee is imported from the authors' SPN paper rather than independently derived; the external empirical benchmarks are not circular.

  1. self citation load bearing [Section 3.2, Theorem 2; Supplementary Section 7.2, Theorem 4]
    "Theorem 2. The stability of Eq. (1) is ensured when all matrices wτ are row stochastic. Proof. Making wτ row stochastic is a sufficient condition to ensure stability presented in [62]. See the Appendix for the complete proof. ... Making wτ row stochastic ensures that the norm constraint ∥wi∥ ≤ 1 holds, which provides a sufficient condition for model stability, as presented in [62]."

    The load-bearing theoretical claim that row-stochastic propagation matrices guarantee stable long-range propagation is not independently derived in this paper. The main-text proof is a citation to the authors' prior SPN paper [62], and the appendix proof again ends by citing [62]. The cited paper shares authors with the present paper, and the appendix's Gershgorin argument does not supply independent support because Gershgorin bounds eigenvalues, not singular values, so the asserted bound ||wi|| ≤ σmax ≤ 1 does not follow. The stability guarantee is therefore carried by a self-citation rather than by an external, machine-checked, or independently reproduced theorem.

full rationale

The paper's headline performance claims are benchmarked against external datasets and models (ImageNet, COCO, DiT, SD-XL), so there is no fitting-as-prediction circularity, and the line-scan complexity analysis is an architectural consequence rather than a disguised fit. However, the Stability-Context Condition has two halves: Theorem 1, that products of row-stochastic matrices remain row-stochastic, is a valid elementary proof, while Theorem 2, that row-stochasticity implies stability, is justified by citation to the authors' own SPN paper [62]. The appendix's attempted proof conflates Gershgorin eigenvalue bounds with singular-value bounds, so the cited prior work is doing the load-bearing work. This does not make the ImageNet, FID, or speedup results circular, because those are external empirical comparisons, and it does not erase the substantial independent engineering content. It does mean the central theoretical premise is not self-contained. The Gershgorin/singular-value issue is primarily a correctness risk; it is relevant to circularity only as evidence that the stability theorem is not independently proven here. Overall score 4 reflects partial self-citation load-bearing with substantial independent empirical content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No constants are fitted to target metrics; the central empirical claims are benchmarked externally. The ledger mostly contains design assumptions inherited from SPN and one mathematically false stability premise, which is the main theoretical risk.

free parameters (3)
  • Local GSPN group size g = 2
    Hand-chosen default in Section 4.1; controls the O(4N*d/g) work and the receptive field of the local variant.
  • High-resolution denoising step skip fraction = 60%
    Section 5.6 skips 60% of initial high-resolution denoising steps; this empirical choice contributes directly to the claimed 84x speedup.
  • Distillation loss weights = 0.5, 0.5
    Section 5.1 sets both distillation hyperparameters to 0.5 by hand; they control the text-to-image quality numbers in Table 3.
assumptions (4)
  • standard math Row-stochastic propagation matrices w_t ensure stability via sigma_max(w_t) <= 1.
    Supplementary Section 7.2 (Theorem 4) claims Gershgorin's circle theorem gives sigma_max <= 1 for row-stochastic matrices. This is false: Gershgorin bounds eigenvalues, and non-normal row-stochastic matrices can have spectral norm greater than 1. The stability guarantee is therefore not established.
  • ad hoc to paper The 3-way tridiagonal connection is the minimal structure whose products become dense.
    Section 7.3 gives an intuitive band-spreading argument but no proof that tridiagonal is minimal for dense products; the dense pairwise connections claim relies on this.
  • domain assumption Sigmoid plus row normalization produces weights satisfying the Stability-Context Condition.
    Section 3.3 Eq. 6 ensures non-negativity and row sums of 1 by construction; whether this preserves input-dependent context is an assumption carried into the architecture.
  • domain assumption Positional information is fully encoded by the scan order, so positional embeddings can be removed.
    Section 4.2 argues this from ablation results in Figure 4; it is an empirical design assumption, not a theorem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Parallel Sequence Modeling via Generalized Spatial Propagation Network." pith.science (2026). https://pith.science/paper/BIN6XPLA

@misc{pith2026250112381,
  author       = {Pith},
  title        = {Pith review of: Parallel Sequence Modeling via Generalized Spatial Propagation Network},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BIN6XPLA}},
  note         = {Machine review of arXiv:2501.12381}
}
abstract

We present the Generalized Spatial Propagation Network (GSPN), a new attention mechanism optimized for vision tasks that inherently captures 2D spatial structures. Existing attention models, including transformers, linear attention, and state-space models like Mamba, process multi-dimensional data as 1D sequences, compromising spatial coherence and efficiency. GSPN overcomes these limitations by directly operating on spatially coherent image data and forming dense pairwise connections through a line-scan approach. Central to GSPN is the Stability-Context Condition, which ensures stable, context-aware propagation across 2D sequences and reduces the effective sequence length to $\sqrt{N}$ for a square map with N elements, significantly enhancing computational efficiency. With learnable, input-dependent weights and no reliance on positional embeddings, GSPN achieves superior spatial fidelity and state-of-the-art performance in vision tasks, including ImageNet classification, class-guided image generation, and text-to-image generation. Notably, GSPN accelerates SD-XL with softmax-attention by over $84\times$ when generating 16K images.

Figures

Figures reproduced from arXiv: 2501.12381 by the authors.

Figure 1
Figure 1. Comparison of attention mechanisms and computa￾tional efficiency. Top: Softmax attention (left), mamba (middle), and our GSPN (right). Bottom: Inference speed comparison across different input sizes on A100 GPU, demonstrating GSPN’s superior efficiency for high-resolution input. Dashed lines indicate quadratic extrapolation due to memory explosion. Global and local GSPN are detailed in Section 4.1. tasks, where main… view at source ↗
Figure 2
Figure 2. 2D Linear Propagation. For the i-th row, each item in the hidden state hi is computed by: (1) a weighted sum of three neighboring values from the hidden layer hi−1, where weights form a normalized tridiagonal matrix wi, and (2) the element-wise product of the current input xi with λ. wi and λ are both learnable and input-dependent parameters. The weights in wi are obtained by applying sigmoid activation followed by … view at source ↗
Figure 3
Figure 3. Architecture of GSPN. We design different GSPN blocks for discriminative and generative task. Both blocks share the same GSPN module with our core 2D Linear Propagation mechanism illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Design choice of GSPN with benchmark methods in classification accuracy and computational efficiency for both classification and generation tasks. A higher Top-1 accuracy and a lower FID indicate better model capability. Our GSPN can outperform other methods under simi…
Figure 5
Figure 5. Figure 5: Qualitative text-to-image results generated from our SD-XL-GSPN model. We enable generation up to 16K resolution on a single A100 GPU while reducing inference time by up to 84× on the SD-XL model. Hybrid Design. Starting with an all-local GSPN baseline achieving 81.2% …
Figure 6
Figure 6. Figure 6: GSPN guarantees Dense Pairwise Connections via 3-way connection and 4-directional scanning, as introduced in Sec 3.2 and detailed in Sec. 7.3. The scanning of each direction corresponds to a lower triangular affinity matrix. The finally full matrix is obtained through …
Figure 7
Figure 7. Figure 7: Illustration of heatmaps for the query patch (marked with an orange star) along different directions and the averaged results. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Qualitative results of class-conditional generation from our 256 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Examples of GSPN at various higher resolutions based upon SD-v1.5 and SDXL. GSPN enables to synthesis of images up to a [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Examples of GSPN at various higher resolutions based upon SDXL. GSPN enables to synthesis of images up to a resolution of [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Examples of GSPN at various higher resolutions based upon SDXL. GSPN enables to synthesis of images up to a resolution of [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

97 extracted references · 50 canonical work pages

  1. [62]

    Learning affinity via spatial propagation networks

    Sifei Liu, Shalini De Mello, Jinwei Gu, Guangyu Zhong, Ming-Hsuan Yang, and Jan Kautz. Learning affinity via spatial propagation networks. NeurIPS, 2017. 2, 3, 4, 5, 8

    work page 2017

  2. [1]

    Xcit: Cross-covariance image transformers

    Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bo- janowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Na- talia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 2021. 2

    2021

  3. [2]

    Vision-lstm: xlstm as generic vision backbone

    Benedikt Alkin, Maximilian Beck, Korbinian P¨oppel, Sepp Hochreiter, and Johannes Brandstetter. Vision-lstm: xlstm as generic vision backbone. arXiv preprint arXiv: 2406.04303,

    arXiv

  4. [3]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. 4

    arXiv 2016

  5. [4]

    Exploring alternatives to softmax function

    Kunal Banerjee, Rishi Raj Gupta, Karthik Vyas, Biswajit Mishra, et al. Exploring alternatives to softmax function. arXiv preprint arXiv:2011.11538, 2020. 2

    arXiv 2011

  6. [5]

    All are worth words: A vit backbone for diffusion models

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2023. 7

    2023

  7. [6]

    Multidiffusion: Fusing diffusion paths for controlled image generation

    Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. In ICML, 2023. 8

    2023

  8. [7]

    2-d ssm: A general spatial layer for visual transformers

    Ethan Baron, Itamar Zimerman, and Lior Wolf. 2-d ssm: A general spatial layer for visual transformers. arXiv preprint arXiv:2306.06635, 2023. 2

    arXiv 2023

Show all 97 references
  1. [8]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 1

    2005 arXiv

  2. [9]

    Scene labeling with lstm recurrent neural net- works

    Wonmin Byeon, Thomas M Breuel, Federico Raue, and Mar- cus Liwicki. Scene labeling with lstm recurrent neural net- works. In CVPR, 2015. 2

    2015

  3. [10]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, 2020. 1, 2

    2020

  4. [11]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. ICCV,

  5. [12]

    Pixart- \α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart- \α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 7

    arXiv

  6. [13]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. 2

    1904 arXiv

  7. [14]

    Xception: Deep learning with depthwise separable convolutions

    Franc ¸ois Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017. 4

    2017

  8. [15]

    Rethink- ing attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethink- ing attention with performers. In ICLR, 2021. 1, 2

    2021

  9. [16]

    Twins: Re- visiting the design of spatial attention in vision transformers

    Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Re- visiting the design of spatial attention in vision transformers. NeurIPS, 2021. 6

    2021

  10. [17]

    Empirical evaluation of gated recurrent neural networks on sequence modeling

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NeurIPS, 2014. 2

    2014

  11. [18]

    Coatnet: Marrying convolution and attention for all data sizes

    Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. NeurIPS, 2021. 2, 6, 7

    2021

  12. [19]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In ICML, 2024. 7

    2024

  13. [20]

    Vision transformers need registers

    Timoth´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR,

  14. [21]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 6

    2009

  15. [22]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 2021. 7

    2021

  16. [23]

    Cswin transformer: A general vision transformer backbone with cross-shaped windows

    Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022. 6, 7

    2022

  17. [24]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2020. 1, 2 10

    2020

  18. [25]

    Demofusion: Democratising high- resolution image generation with no $$$

    Ruoyi Du, Dongliang Chang, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. Demofusion: Democratising high- resolution image generation with no $$$. In CVPR, 2024. 8

    2024

  19. [26]

    Vision-rwkv: Efficient and scalable visual per- ception with rwkv-like architectures

    Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, and Wen- hai Wang. Vision-rwkv: Efficient and scalable visual per- ception with rwkv-like architectures. arXiv preprint arXiv: 2403.02308, 2024. 6, 7

    2024 arXiv

  20. [27]

    Sigmoid- weighted linear units for neural network function approxi- mation in reinforcement learning

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid- weighted linear units for neural network function approxi- mation in reinforcement learning. Neural networks, 2018. 4

    2018

  21. [28]

    Hungry hungry hippos: Towards language modeling with state space models

    Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher R´e. Hungry hungry hippos: Towards language modeling with state space models. In ICLR, 2023. 2

    2023

  22. [29]

    Levit: a vision transformer in convnet’s clothing for faster inference

    Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herv ´e J ´egou, and Matthijs Douze. Levit: a vision transformer in convnet’s clothing for faster inference. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 12259–12269,

  23. [30]

    Multi-dimensional recurrent neural networks

    Alex Graves, Santiago Fern´andez, and J¨urgen Schmidhuber. Multi-dimensional recurrent neural networks. InInternational conference on artificial neural networks, 2007. 2

    2007

  24. [31]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 1, 2, 6, 7

    2023 arXiv

  25. [32]

    Elasticdiffusion: Training-free arbitrary size image generation through global-local content separation

    Moayed Haji-Ali, Guha Balakrishnan, and Vicente Ordonez. Elasticdiffusion: Training-free arbitrary size image generation through global-local content separation. In CVPR, 2024. 8

    2024

  26. [33]

    Flatten transformer: Vision transformer using focused linear attention

    Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5961– 5971, 2023. 1

    2023

  27. [34]

    Mambavision: A hybrid mamba-transformer vision backbone

    Ali Hatamizadeh and Jan Kautz. Mambavision: A hybrid mamba-transformer vision backbone. arXiv preprint arXiv: 2407.08083, 2024. 6

    2024 arXiv

  28. [35]

    Zhang, Shaoqing Ren, and Jian Sun

    Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 2

    2016

  29. [36]

    Scalecrafter: Tuning-free higher- resolution visual generation with diffusion models

    Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, and Ying Shan. Scalecrafter: Tuning-free higher- resolution visual generation with diffusion models. In ICLR,

  30. [37]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arxiv. arXiv preprint arXiv:1606.08415, 2016. 4

    2016 arXiv

  31. [38]

    Tenenbaum, Kfir Aberman, Y

    Amir Hertz, Ron Mokady, J. Tenenbaum, Kfir Aberman, Y . Pritch, and D. Cohen-Or. Prompt-to-prompt image editing with cross attention control. ICLR, 2022. 2

    2022

  32. [39]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bern- hard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017. 7

    2017

  33. [40]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop on Deep Generative Models and Downstream Applications, 2022. 7

    2022

  34. [41]

    Axial attention in multidimensional transformers

    Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180, 2019. 2

    1912 arXiv

  35. [42]

    Untersuchungen zu dynamischen neu- ronalen netzen

    Sepp Hochreiter. Untersuchungen zu dynamischen neu- ronalen netzen. Diploma, Technische Universit¨at M¨unchen,

  36. [43]

    Long short-term memory

    S Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997. 2

    1997

  37. [44]

    Trans- former quality in linear time

    Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Trans- former quality in linear time. In ICML, 2022. 2

    2022

  38. [45]

    Fouriscale: A frequency perspective on training-free high-resolution image synthesis

    Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, and Hongsheng Li. Fouriscale: A frequency perspective on training-free high-resolution image synthesis. arXiv preprint arXiv:2403.12963, 2024. 8

    2024 arXiv

  39. [46]

    Localmamba: Visual state space model with windowed selective scan

    Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan. arXiv preprint arXiv:2403.09338,

    arXiv

  40. [47]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML, 2020. 1, 2, 3

    2020

  41. [48]

    Bk-sdm: Architecturally compressed stable diffusion for efficient text-to-image generation

    Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: Architecturally compressed stable diffusion for efficient text-to-image generation. In Workshop on Efficient Systems for Foundation Models@ ICML2023 ,

  42. [49]

    Re- former: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Re- former: The efficient transformer. In ICLR, 2020. 2

    2020

  43. [50]

    Chatgpt: Jack of all trades, master of none

    Jan Koco ´n, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kan- clerz, et al. Chatgpt: Jack of all trades, master of none. Information Fusion, 2023. 1

    2023

  44. [51]

    Im- agenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Im- agenet classification with deep convolutional neural networks. NeurIPS, 2012. 2

    2012

  45. [52]

    Improved precision and recall metric for assessing generative models

    Tuomas Kynk¨a¨anniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In NeurIPS, 2019. 7

    2019

  46. [53]

    Next-vit: Next generation vision transformer for efficient deployment in real- istic industrial scenarios

    Jiashi Li, Xin Xia, Wei Li, Huixia Li, Xing Wang, Xuefeng Xiao, Rui Wang, Min Zheng, and Xin Pan. Next-vit: Next generation vision transformer for efficient deployment in real- istic industrial scenarios. arXiv preprint arXiv:2207.05501,

    arXiv

  47. [54]

    Savarese, and Steven C

    Junnan Li, Dongxu Li, S. Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML,

  48. [55]

    Uniformer: Unified transformer for efficient spatiotemporal representation learning

    Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning. In ICLR,

  49. [56]

    Distrifusion: Distributed parallel inference for high-resolution diffusion models

    Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In CVPR, 2024. 8 11

    2024

  50. [57]

    Mamba-nd: Selective state space modeling for multi-dimensional data

    Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-nd: Selective state space modeling for multi-dimensional data. arXiv preprint arXiv:2402.05892, 2024. 2, 6, 7

    2024 arXiv

  51. [58]

    Cutdiffusion: A simple, fast, cheap, and strong diffusion extrapolation method

    Mingbao Lin, Zhihang Lin, Wengyi Zhan, Liujuan Cao, and Rongrong Ji. Cutdiffusion: A simple, fast, cheap, and strong diffusion extrapolation method. arXiv preprint arXiv:2404.15141, 2024. 8

    2024 arXiv

  52. [59]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 6

    2014

  53. [60]

    Ac- cdiffusion: An accurate method for higher-resolution image generation

    Zhihang Lin, Mingbao Lin, Meng Zhao, and Rongrong Ji. Ac- cdiffusion: An accurate method for higher-resolution image generation. arXiv preprint arXiv:2407.10738, 2024. 8

    2024 arXiv

  54. [61]

    Transformer-vq: Linear-time transformers via vector quantization

    Lucas D Lingle. Transformer-vq: Linear-time transformers via vector quantization. In ICLR, 2024. 2

    2024

  55. [63]

    Linfusion: 1 gpu, 1 minute, 16k image

    Songhua Liu, Weihao Yu, Zhenxiong Tan, and Xinchao Wang. Linfusion: 1 gpu, 1 minute, 16k image. arXiv preprint arXiv:2409.02097, 2024. 7, 8

    2024 arXiv

  56. [64]

    Vmamba: Visual state space model

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166,

    arXiv

  57. [65]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 1, 2, 4, 6, 7

    2021

  58. [66]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022. 2, 4, 6, 7

    2022

  59. [67]

    Soft: Softmax-free transformer with linear complexity

    Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. Soft: Softmax-free transformer with linear complexity. NeurIPS,

  60. [68]

    Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers

    Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In ECCV, 2024. 7

    2024

  61. [69]

    Sdedit: Guided image synthesis and editing with stochastic differential equations

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021. 4, 8

    2021 arXiv

  62. [70]

    Generating images with sparse representations

    Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.03841, 2021. 7

    2021 arXiv

  63. [71]

    S4nd: Modeling images and videos as multidimensional signals with state spaces

    Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher R´e. S4nd: Modeling images and videos as multidimensional signals with state spaces. NeurIPS, 2022. 2

    2022

  64. [72]

    On aliased resizing and surprising subtleties in gan evaluation

    Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR,

  65. [73]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In ICML,

  66. [74]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023. 1, 5, 7, 8

    2023

  67. [75]

    Random feature attention

    Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong. Random feature attention. arXiv preprint arXiv:2103.02143, 2021. 1, 2

    2021 arXiv

  68. [76]

    Self-attention does not need o(n2) memory

    Markus N Rabe and Charles Staats. Self-attention does not need o(n2) memory. arXiv preprint arXiv:2112.05682, 2021. 1

    2021 arXiv

  69. [77]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022. 7, 8

    2022

  70. [78]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurIPS, 2016. 7

    2016

  71. [79]

    Laion-5b: An open large-scale dataset for training next gener- ation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next gener- ation image-text models. NeurIPS, 2022. 6

    2022

  72. [80]

    Efficient attention: Attention with linear complexities

    Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In WACV, 2021. 2

    2021

  73. [81]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024. 6

    2024

  74. [82]

    Training data-efficient image transformers & distillation through atten- tion

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv´e J´egou. Training data-efficient image transformers & distillation through atten- tion. In ICML, 2021. 1, 2, 6, 7

    2021

  75. [83]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  76. [84]

    Pixel recurrent neural networks

    A¨aron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In ICML,

  77. [85]

    Attention is all you need

    A Vaswani. Attention is all you need. NeurIPS, 2017. 4, 6

    2017

  78. [86]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. 2

    2006 arXiv

  79. [87]

    Pyra- mid vision transformer: A versatile backbone for dense pre- diction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyra- mid vision transformer: A versatile backbone for dense pre- diction without convolutions. In ICCV, 2021. 2, 4

    2021

  80. [88]

    Con- vnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Con- vnext v2: Co-designing and scaling convnets with masked autoencoders. In CVPR, 2023. 4

    2023

  81. [89]

    Cvt: Introducing convolutions to vision transformers

    Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 2

    2021

  82. [90]

    Lite transformer with long-short range attention

    Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. In ICLR,

  83. [91]

    Nystr¨omformer: A nystr ¨om-based algorithm for approximat- ing self-attention

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystr¨omformer: A nystr ¨om-based algorithm for approximat- ing self-attention. In AAAI, 2021. 2

    2021

  84. [92]

    Focal modulation networks

    Jianwei Yang, Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks. NeurIPS, 2022. 6

    2022

  85. [93]

    Emernerf: Emergent spatial- temporal scene decomposition via self-supervision

    Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Se- ung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, and Yue Wang. Emernerf: Emergent spatial- temporal scene decomposition via self-supervision. arXiv preprint arXiv:2311.02077, 2023. 2, 6

    2023 arXiv

  86. [94]

    Mambaout: Do we really need mamba for vision? arXiv preprint arXiv:2405.07992,

    Weihao Yu and Xinchao Wang. Mambaout: Do we really need mamba for vision? arXiv preprint arXiv:2405.07992,

    arXiv

  87. [95]

    Tay, Jiashi Feng, and Shuicheng Yan

    Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021. 6

    2021

  88. [96]

    Biformer: Vision transformer with bi-level routing attention

    Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson Lau. Biformer: Vision transformer with bi-level routing attention. CVPR, 2023. 2

    2023

  89. [97]

    Vision mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024. 2, 6 13

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.