Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Pushing the Boundaries of State Space Models for Image and Video Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 5-billion-parameter diffusion model that replaces most self-attention with a bidirectional state-space token mixer matches Transformer image quality and video consistency, and runs faster on long sequences.

desk verdict A credible 5B-scale study of SSM/attention hybrids for visual generation whose efficiency story holds, but the headline parity claim outruns its own GenEval and VBench evidence. read the letter →

arxiv 2502.00972 v1 pith:ZIX3NVQ7 submitted 2025-02-03 cs.CV cs.LG

Yicong Hong , Long Mai , Yuan Yao , Feng Liu This is my paper · ORCID
classification cs.CVcs.LG
keywords statespacemodelsdiffusiontext-to-imagegenerationtext-to-videohybridarchitecturelinearattentionefficientinferenceHydra
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test whether a state-space model (SSM)—a token mixer that processes sequences in roughly linear rather than quadratic time—can produce images and videos as good as a Transformer-based diffusion model. It builds a 5-billion-parameter denoiser in which 30 of every 33 blocks use Hydra, a bidirectional SSM, with the remaining three using self-attention. Using only simple horizontal and vertical raster scans, the model generates up to 2K images and 8-second 360p videos, and its FID and text-alignment scores are comparable to those of several contemporary diffusion baselines. The paper concludes that sequential, subquadratic token mixers are a viable replacement for Transformer backbones in long-sequence visual generation, with faster inference at very long token counts.

What carries the argument

The object that carries the argument is Hydra, a bidirectional state-space token mixer defined as a quasiseparable matrix: a forward semiseparable sweep plus a reversed sweep plus a learnable diagonal, making each token's output depend on tokens on both sides without quadratic attention. The hybrid recipe wraps this mixer in a 33-block diffusion stack with a 30:3 Hydra-to-attention ratio, and the scanning order—alternating horizontal and vertical bidirectional raster scans, with a temporal-first variant for video—is what gives the sequential mixer enough spatial and temporal receptive field to handle 2D and 3D latents. The few self-attention layers provide global interaction at low cost.

What would settle it

Compare HTH against a same-scale DiT model using the identical VAE, identical training data, and identical diffusion schedule; if HTH's FID/CLIP parity on MJHQ-30K and VBench disappears, or the roughly 3x speedup at 384x384 tokens no longer appears, the paper's central attribution of parity and efficiency to the Hydra mixer is not supported.

Watch

Extended reading notes

Core claim

The central claim is that the boundary of SSM-based visual generation is far larger than prior small-scale studies suggested: at 5B parameters, a Hydra-dominant hybrid matches a same-scale DiT baseline and several public diffusion models on image and video benchmarks, and it does so with subquadratic cost. The model reaches 2K-resolution images (up to 2560x2560 in zero-shot experiments) and 360p 128-frame videos after some Hydra layers are switched from spatial to temporal-first scanning. On MJHQ-30K it reports FID 6.52 versus the DiT baseline's 6.90, and on VBench its 360p model leads in Dynamic Degree and overall consistency among the compared systems. The paper also reports the clear limitation that GenEval object-composition scores are lower than DiT's (0.58 vs 0.63 overall), which it attributes to the limited global modeling of SSM-dense architectures.

Load-bearing premise

The benchmark comparisons assume that HTH's internal VAE and internal training data are quality-matched to the public pipelines used as baselines; if the VAE reconstruction or data richness differs materially, the apparent parity is driven by the pipeline, not by the Hydra token mixer, so the conclusion does not follow.

Editorial extensions

If this is right

  • A 30:3 Hydra-to-self-attention ratio is sufficient for FID and text-alignment parity with Transformer diffusion models at 1K resolution, so the next threshold to probe is a fully SSM backbone.
  • The measured denoiser speed gap widens with token count—around 3x at 384x384 tokens—so SSM backbones become the cheaper option for 8K images, longer videos, and other long visual sequences.
  • Replacing spatial scans with temporal-first scans in 40% of Hydra layers turns an image-trained model into a video generator without adding parameters, suggesting a cheap route from T2I to T2V.
  • The zero-shot generation of images roughly 3.5x larger than the training resolution indicates that SSM locality may soften the resolution-generalization problem that Transformer positional encodings suffer.
  • Object composition remains the known failure mode: GenEval scores fall below the DiT baseline, implying that global attention or better conditioning is still needed for multi-object scenes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because HTH uses an internal 3D VAE and internal training data while the baselines use different public VAE decoders and corpora, the parity claim is pipeline-confounded; a same-VAE, same-data comparison is needed to isolate the Hydra token mixer.
  • If the quasiseparable sweep is the true source of the efficiency and quality, then substituting another bidirectional linear mixer in the same 30:3 schedule would preserve speed but not necessarily the FID/CLIP parity—an experiment the paper does not run.
  • The 3.5x zero-shot resolution extrapolation, if replicated, implies SSM-based pipelines could reduce or eliminate resolution-specific fine-tuning stages in production systems, a consequence the authors leave implicit.
  • The speed advantage only becomes decisive beyond roughly 128x128 tokens, so the practical payoff depends on whether the field pushes image sizes above 2K or video lengths beyond 8 seconds as a standard.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HTH, a 5B-parameter diffusion model that mixes Hydra bidirectional state-space-model layers with a small number of self-attention layers, and applies it to text-to-image generation up to 2K resolution and text-to-video generation at 360p, 128 frames. The authors report FID/CLIP-Score/GenEval numbers for images, a VBench comparison for video, an inference-speed comparison against a Transformer denoiser, and a limitations appendix. The central claim is that models built with sequential, subquadratic token mixers can achieve performance comparable to Transformer-based models while being more efficient.

Significance. If fully supported, this would be a valuable empirical data point: it is the largest reported diffusion SSM-Transformer hybrid for visual generation, it uses deliberately simple raster and temporal scans, it includes a matched DiT baseline for image generation, and it is unusually candid about its limitations in Appendix A. The paper also ships a concrete efficiency comparison and demonstrates zero-shot higher-resolution generalization. However, the significance as stated is not fully established: the matched image comparison shows a clear GenEval deficit and a CLIP-Score deficit, the video comparison lacks a matched baseline and uses a selected subset of VBench dimensions, and no error bars are reported. These gaps affect the strength of the conclusion but appear addressable within the scope of a revision.

major comments (4)
  1. [Section 5 (Conclusion) and Section 4.1.2, Table 2] The unqualified conclusion that SSM-based models 'can achieve performance comparable to Transformer-based models' is not supported by the paper's own matched comparison. At 1024x1024, HTH improves MJHQ FID (6.52 vs 6.90) but is worse on CLIP-Score (27.26 vs 27.37) and clearly worse on GenEval (0.58 vs 0.63); Appendix A explicitly states that the results 'generally exhibit inferior semantic understanding (e.g., worse entity composition and structural coherence) compared to Transformer-based models.' Since text-aligned composition is a core capability for text-to-image and text-to-video generation, the evidence establishes parity in low-level fidelity and dynamics, not overall parity. Please either qualify the central claim to something like 'comparable pixel fidelity with weaker prompt alignment' or add a prompt-following benchmark on which HTH matches the Transformer baseline.
  2. [Section 4.1.2, Table 3] The video evaluation does not establish parity against Transformer-based video models. Table 3 compares HTH against externally trained models with different VAEs, training data, and output resolutions, and there is no matched internal DiT-video baseline. Moreover, the authors 'choose 12 evaluation dimensions' from VBench without stating criteria, and the selected set omits composition-oriented dimensions such as Multiple Objects and Spatial Relationship, which are precisely where the GenEval result and Appendix A predict HTH to be weak. Please report all VBench dimensions or justify the subset, and add a same-pipeline DiT video baseline so the comparison isolates the token mixer rather than the data/VAE pipeline.
  3. [Section 4.1.2, Tables 1 and 2] No error bars or multiple-seed runs are reported, so the differences between HTH and the DiT baseline (e.g., MJHQ FID 6.52 vs 6.90, CLIP-Score 27.26 vs 27.37, GenEval 0.58 vs 0.63) are not shown to be statistically meaningful. Given that the VAE and training data are internal, the comparison to external models is also sensitive to pipeline confounds. Please provide variance estimates or at least report the evaluation protocol, including seed count and sample size, so the reader can assess whether the observed differences exceed run-to-run noise.
  4. [Section 4.1.3, Figure 5] The inference-speed comparison may not isolate the effect of the token mixer. The two 5B models are described only as 'HTH' and 'Transformer (our DiT baseline)'; no details are given about layer counts, block structure, attention heads, batch size, hardware, or whether FlashAttention-2 is used in both. Since HTH replaces most self-attention blocks with Hydra but also adds per-block cross-attention, the measured speed gap could be affected by factors other than the SSM-versus-attention token mixer. Please specify the exact configurations compared and report latency at the actual operating token lengths used for 2K images and 360p videos, including total wall-clock time rather than only the difference.
minor comments (5)
  1. [Section 3.1] The text contains a typo '3D V AE' that should read '3D VAE'.
  2. [Table 3] The header 'Apperance' is misspelled; it should be 'Appearance'. The caption should also list the 12 chosen dimensions explicitly.
  3. [Section 4.1.2] The parenthetical '(score 0.67 vs. 0.75)' in the GenEval discussion is ambiguous: please specify which GenEval subcategory is being reported and confirm that both scores are for HTH and the DiT baseline, respectively.
  4. [Figure 5] The x-axis labels '1282' and '3842' should be typeset as 128^2 and 384^2. The caption and text should also state the absolute times for both models, not only the difference, to make the '10 seconds faster' claim interpretable.
  5. [Appendix B.1] The small-scale token-mixer comparison is presented only through qualitative examples in Figure 6; adding quantitative metrics for this comparison would strengthen the claim that Hydra is the best SSM choice.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation chain; the headline claim is empirical, though the conclusion's 'comparable' wording is broader than the paper's own Appendix A admission supports.

full rationale

This paper is an empirical scaling and benchmarking study, not a derivation of its target result from fitted parameters or from a self-citation chain. The Hydra token mixer is adopted from Hwang et al. (2024), an external source with no author overlap, and that citation is used only to justify the mixer choice, not to set any benchmark number in this paper. The few self-citations (e.g., LRM, Long-LRM) appear in background enumerations of prior visual-generation work and do not carry the SSM claim. The internal DiT baseline is trained under the same pipeline and data, giving a controlled, non-circular comparison of the Hydra mixer. The VBench table selects 12 of 16 dimensions and HTH underperforms on GenEval; these are evaluation-selection and overstatement concerns, not reductions of the prediction to its inputs. Appendix A explicitly states that "the results generally exhibit inferior semantic understanding (e.g., worse entity composition and structural coherence) compared to Transformer-based models," which qualifies the conclusion's 'comparable' wording but does not indicate circularity. No equation in the paper is defined in terms of the quantity it is used to explain, and no fitted coefficient is relabeled as a prediction. The central claim is therefore not circular; it is merely stronger than the paper's own evidence supports.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the choice of Hydra as the token mixer, the internal VAE, standard diffusion assumptions, and the evaluation metrics. No new entities are introduced. The main hand-chosen design parameters are the Hydra/attention ratio, the temporal-first proportion, and unreported sampling settings.

free parameters (4)
  • Hydra-to-Transformer ratio (30:3 per 11 blocks) = 30 Hydra : 3 Self-Attention
    Hand chosen without reported ablation; central architectural decision affecting quality/efficiency trade-off.
  • Proportion of temporal-first Hydra blocks (40%) = 40% of Hydra blocks
    Tuned via 'extensive experiments' without reported ablations; key for video adaptation.
  • Classifier-free guidance scale = not reported
    Affects all generated samples and benchmark scores; omission prevents exact replication.
  • DDIM sampling steps = not reported
    Affects speed and quality; not specified in inference details.
assumptions (4)
  • domain assumption Hydra's quasiseparable matrix formulation is strictly more expressive than additive bidirectional SSMs and is correctly implemented in the Mamba-2 framework.
    The paper's central choice of Hydra as token mixer relies on claims from Hwang et al. (2024), not re-derived here; if Hydra were not as expressive, the benchmark results would be weaker.
  • domain assumption The internal MAGVIT-v2-style 3D VAE preserves sufficient information for text-conditioned generation.
    The VAE is not released and is adapted from prior work; reconstruction fidelity directly bounds generation quality.
  • domain assumption Standard diffusion training and sampling (InstaFlow schedule, classifier-free guidance, DDIM) apply unchanged to SSM-based latent diffusion.
    The paper adapts DiT pipeline without re-validating diffusion assumptions for the hybrid token mixer.
  • domain assumption FID, CLIP-Score, GenEval, and VBench are valid proxies for text-image/video alignment and quality.
    Benchmark comparisons rely on these metrics, which are known to have limitations for structure coherence and temporal dynamics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pushing the Boundaries of State Space Models for Image and Video Generation." pith.science (2026). https://pith.science/paper/ZIX3NVQ7

@misc{pith2026250200972,
  author       = {Pith},
  title        = {Pith review of: Pushing the Boundaries of State Space Models for Image and Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZIX3NVQ7}},
  note         = {Machine review of arXiv:2502.00972}
}
read the original abstract

While Transformers have become the dominant architecture for visual generation, linear attention models, such as the state-space models (SSM), are increasingly recognized for their efficiency in processing long visual sequences. However, the essential efficiency of these models comes from formulating a limited recurrent state, enforcing causality among tokens that are prone to inconsistent modeling of N-dimensional visual data, leaving questions on their capacity to generate long non-causal sequences. In this paper, we explore the boundary of SSM on image and video generation by building the largest-scale diffusion SSM-Transformer hybrid model to date (5B parameters) based on the sub-quadratic bi-directional Hydra and self-attention, and generate up to 2K images and 360p 8 seconds (16 FPS) videos. Our results demonstrate that the model can produce faithful results aligned with complex text prompts and temporal consistent videos with high dynamics, suggesting the great potential of using SSMs for visual generation tasks.

Figures

Figures reproduced from arXiv: 2502.00972 by the authors.

Figure 1
Figure 1. Text-to-1K/2K+ image generation results of our Hydra-Transformer Hybrid model. The resolution of each sample is displayed in the bottom-right corner. Text prompts and additional results are provided in the Appendix. Please zoom in for a clearer visualization. 2 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Text-to-360p 128 frames video generation results produced by our Hydra-Transformer Hybrid model. More results are provided in the Appendix. Please zoom in for a clearer visualization. encouraging results on image generation, our preliminary experiments show that these methods struggle with complex text prompts or video generation, where the latter requires learning much more complicated spatiotemporal patterns. In l… view at source ↗
Figure 3
Figure 3. Illustration of our diffusion Hybrid Hydra (HTH) model for image and video generation. The architecture consists of N stacked blocks, each comprising a cross-attention layer, a token mixer, and a feed-forward network. (a) The token mixer can be implemented as either the Hydra state space model or self-attention. (b) For image data, we use horizontal and vertical bidirectional raster scans on tokens, and for video da… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Illustration of the model adaptation from state 1 (T2I) to state 2 (T2V). For each set of 11 blocks in our HTH model, we change the scanning pattern of certain Hydra layers from spatial-major to temporal-major scan when processing video data. networks (Botev et al., 20…
Figure 5
Figure 5. Figure 5: Denoiser inference speed comparison between HTH and Transformer (our DiT baseline). et al., 2023b), Show-1 (Zhang et al., 2024a), OpenSora￾v1.2 (Open-Sora, 2024), VideoCrafter-2.0 (Chen et al., 2023), T2V-Turbo (VC2) (Li et al., 2024b), CogVideoX￾5B (Yang et al., 2024)…
Figure 6
Figure 6. Figure 6: Example 256ˆ256-resolution T2I generation results of different token mixers. SSMs and Hydra models are based on Mamba￾2 (Dao & Gu, 2024). The Bidirectional SSM and Hydra apply interleaved horizontal and vertical raster scans across model blocks. None of the models in t…
Figure 7
Figure 7. Figure 7: Zero-shot comparison at higher resolution. We compare our HTH model with the same-scale Transformer-based models (SA: Self-Attention) with Absolute Positional Embedding (APE) or Rotary Positional Embedding (RoPE). All models are trained on 256p image data and evaluated…
Figure 8
Figure 8. Figure 8: Text-to-360p 128 frames video generation results produced by our Hydra-Transformer Hybrid model. Please zoom in for a clearer visualization. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Text-to-1K image generation results produced by our Hydra-Transformer Hybrid model. The images are in resolutions 768ˆ1344 and 1344ˆ1344. Please zoom in for clearer visualization. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Text-to-1K image generation results produced by our Hydra-Transformer Hybrid model. The images are in resolutions 1344ˆ768. Please zoom in for clearer visualization. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Text-to-2K image generation results produced by our Hydra-Transformer Hybrid model. The images are in resolutions 1440ˆ2560. Please zoom in for clearer visualization. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Text-to-2K image generation results produced by our Hydra-Transformer Hybrid model. The images are in resolutions 2560ˆ1440. Please zoom in for clearer visualization. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Text-to-2K+ image generation results produced by our Hydra-Transformer Hybrid model. The images are in resolutions 2560ˆ2560. Please zoom in for clearer visualization. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

    cs.CV 2026-07 conditional novelty 6.5 of 10

    A hybrid NoPE visual diffusion backbone with HeteroP scaling yields ~7× pretraining compute efficiency versus matched full attention and near-stable zero-shot 5s→30s video extrapolation.

Reference graph

Works this paper leans on

15 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    A dog that has been meditating all the time

    work page

  2. [2]

    He is seated at an office desk, his melancholic expression in contrast to the elaborate facades of yttrium yellow and zircon blue surrounding him

    In a surreal office setting during a sunlit day, a man in a bizarre jelly mask, reminiscent of an oyster shell, occupies a full shot. He is seated at an office desk, his melancholic expression in contrast to the elaborate facades of yttrium yellow and zircon blue surrounding him

    work page

  3. [3]

    The scene exudes a gritty retro sci-fi aesthetic from the 70s and 80s, with outdated technology and weathered machinery

    Full-body photograph of a beautiful female cyborg, suspended in a worn-out, cassette-futurism science lab. The scene exudes a gritty retro sci-fi aesthetic from the 70s and 80s, with outdated technology and weathered machinery. Dust particles float in the light, enhancing the vintage, worn-down atmosphere. The cyborg’s synthetic skin is torn from her down...

    work page

  4. [4]

    A bouquet of roses made of pastel ice crystals

    work page

  5. [5]

    A cat drinking a beer

    work page

  6. [6]

    Paper artwork, layered paper, colorful Chinese dragon surrounded by clouds

    work page

  7. [7]

    Beautiful neon lights forming the words “Hydra”, glowing vibrantly

    work page

  8. [8]

    Filmic photo of a group of three women on a street downtown, they are holding their hands up the camera

    work page

Show all 15 references
  1. [9]

    Neon lights, and a sense of technological future

    A pink bunny girl in the cyber world, with beautiful long hair, wearing a pink skirt, pink bunny ears, pink headphones. Neon lights, and a sense of technological future

  2. [10]

    Half human, half robot, repaired human, human flesh warrior, mech display, man in mech, cyberpunk

  3. [11]

    Elephant amigurumi walking in savanna, a professional, blurry background

  4. [12]

    Polychrome particles and powders surrounding a whimsical child figure silhouette emerging out of a portal

  5. [13]

    A very cute little Shamrock bird in a mossy forest. 21

  6. [755]

    Liu, S., Yu, W., Tan, Z., and Wang, X

    Springer, 2014. Liu, S., Yu, W., Tan, Z., and Wang, X. Linfusion: 1 gpu, 1 minute, 16k image. arXiv preprint arXiv:2409.02097, 2024. Liu, X., Zhang, X., Ma, J., Peng, J., et al. Instaflow: One step is enough for high-quality diffusion-based text-to- image generation. In The Tw...

    2014 arXiv

  7. [2024]

    Su, J., Ahmed, M., Lu, Y ., Pan, S., Bo, W., and Liu, Y

    URL https://stability.ai/news/ deepfloyd-if-text-to-image-model . Su, J., Ahmed, M., Lu, Y ., Pan, S., Bo, W., and Liu, Y . Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. Sun, P., Jiang, Y ., Chen, S., Zhang, S., Peng, B., Luo,...

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.