Pith. sign in

REVIEW 3 major objections 7 minor 2 cited by

M4V: Multimodal Mamba for Efficient Text-to-Video Generation

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read M4V shows that replacing most attention layers with Mamba state-space blocks cuts mixer-layer FLOPs by 45 percent for 768×1280 text-to-video generation while holding VBench quality essentially equal to the attention-based baseline.

desk verdict Useful Mamba-based T2V architecture with a genuinely new mixer design, but the efficiency claim is mixer-level, not end-to-end; worth a serious referee who asks for controlled timing and clearer data disclosure. read the letter →

arxiv 2506.10915 v2 pith:F4TWELCR submitted 2025-06-12 cs.CV cs.AIcs.LG

Jiancheng Huang , Gengwei Zhang , Zequn Jie , Siyu Jiao , Yinlong Qian , Ling Chen , Yunchao Wei , Lin Ma This is my paper
classification cs.CVcs.AIcs.LG
keywords Mambastate-spacemodelstext-to-videogenerationdiffusionflowmatchingautoregressivevideolinear-timesequencemodelingcomputationalefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that most of the attention layers in a text-to-video diffusion model can be swapped for Mamba state-space blocks without losing generation quality, and that this swap cuts the computational cost of the mixer layers by 45 percent at 768×1280 resolution. On VBench, the resulting system (M4V) scores 81.55 versus 81.61 for the attention-based PyramidFlow baseline trained on the same public data, and 81.91 when reward-based fine-tuning and synthesized training videos are added. The efficiency question matters because attention's quadratic cost over long spatiotemporal token sequences is the main barrier to generating longer, higher-resolution clips. If the claim holds, linear-time sequence models become a practical backbone for high-quality text-to-video generation instead of a niche alternative.

What carries the argument

The carrying object is the MM-DiM (Multi-Modal Diffusion Mamba) block, a drop-in replacement for the attention mixer in the single-stream layers of a multimodal diffusion model. It arranges text and video tokens into one sequence for the selective state-space model: text tokens sit at the front behind zero padding, so the hidden state stays zero until the text is read, and the same text tokens are appended at the end, with the left and right copies summed after the SSM pass to give bidirectional text-video flow without cross-attention. Per-frame registers — learnable tokens inserted between conditional frames at three resolution levels — signal frame boundaries and resolution changes to a model that otherwise infers position only from scan order, and eight zigzag spatial scan paths alternate across layers to preserve 2D structure. A lightweight temporal branch applies causal attention along the compressed time axis to stabilize long-range motion, and a reward-learning fine-tuning stage, scoring one-step denoised frames with an aesthetic reward and a text-image alignment reward, targets the quality decay of later frames in autoregressive generation.

What would settle it

Build the identical model at the same scale and data with attention mixers in place of the MM-DiM blocks, and vice versa, then measure end-to-end wall-clock time for 241-frame, 768×1280 generation on the same GPU; the central claim fails if the Mamba version is not faster in practice once the dual-stream attention blocks and temporal branch are counted. A cheaper check is to profile a single generation and compare time spent in the remaining attention blocks against time spent in the MM-DiM blocks.

Watch

Extended reading notes

Core claim

The central claim is that a text-to-video generator whose single-stream mixer layers are Mamba blocks can match the output quality of an attention-based generator at materially lower compute. The authors keep the autoregressive PyramidFlow pipeline — multi-level latent compression and piecewise flow-matching, with each frame predicted from compressed versions of earlier frames — and replace the attention mixers with their MM-DiM block. The block sequences text and video tokens through a selective state-space model, using token re-composition for bidirectional text-video interaction, per-frame learnable registers to mark frame boundaries and resolution changes, zigzag spatial scans to preserve 2D structure, and a lightweight causal temporal attention branch for long-range motion consistency. Full-vector attention costs $O((TM)^2)$ over $T$ frames and $M$ spatial tokens; the SSM path costs $O(TM)$ and the temporal branch adds $O(T^2)$, and the measured mixer-layer TFLOPs for 241-frame generation fall from 55.44 to 29.52. The paper reports VBench Total Scores of 81.55 for M4V versus 81.61 for PyramidFlow reproduced under the same data, rising to 81.91 with reward learning and generated data, alongside faster reported wall-clock inference (226 seconds versus 296 seconds for 121 frames).

Load-bearing premise

The efficiency claim rests on mixer-layer FLOPs standing in for practical system cost: the final model keeps eight attention-based dual-stream blocks and a causal temporal attention branch, the reported speedup is measured against a different model rather than an identical attention-based twin, and the paper itself reports that replacing the remaining dual-stream blocks with Mamba raises per-iteration training time by about 1.5 times.

Editorial extensions

If this is right

  • An autoregressive flow-matching video model with Mamba mixers matches its attention-based twin on VBench (81.55 versus 81.61 Total Score) while cutting mixer-layer FLOPs from 55.44 to 29.52 TFLOPs for 241-frame clips, so the savings grow with video length.
  • Reward learning with aesthetic and text-image alignment rewards, combined with roughly 80K synthesized training videos, raises the Total Score to 81.91, with the Semantic Score gaining the most.
  • The architecture still leans on attention in two places it does not fully replace — eight dual-stream blocks and the temporal attention branch — and the paper reports that replacing those dual-stream blocks with Mamba would increase per-iteration training time by about 1.5 times.
  • The model's lower Dynamic Degree than attention-based rivals is presented as a known trait of hidden-state-based generation, and the paper names motion diversity as an open direction for Mamba video models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stricter test of the efficiency thesis would swap only the mixer type in an otherwise identical model at the same scale and measure end-to-end latency, since the reported 45% is measured on mixer layers and the paper's own appendix data suggest pure-FLOP accounting flatters the architecture swap.
  • If Mamba-based video backbones mature, the $O(T^2)$ temporal branch becomes the next bottleneck, so the natural follow-up is a linear-time replacement for that branch; the per-frame-register idea would likely transfer to any SSM-based multimodal generator.
  • The authors' own Dynamic Degree observation invites a testable fix: injecting explicit stochasticity or diversity regularization into the SSM hidden state might recover motion diversity without re-introducing attention.
  • Because the gains are stated for one resolution (768×1280) and one hardware setup, the claimed 45% reduction should be re-measured at extreme resolutions and on consumer GPUs, where attention-kernel optimizations may narrow or widen the gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces M4V, a text-to-video generation model that replaces most attention mixers with Mamba-based MultiModal Diffusion Mamba (MM-DiM) blocks, following the autoregressive pyramidal flow matching framework of PyramidFlow. The authors claim a 45% reduction in mixer-layer FLOPs at 768×1280 resolution, VBench scores on par with PyramidFlow (81.55 vs 81.61, rising to 81.91 with reward learning), and faster reported inference (226 vs 296 s for 121 frames). They also propose several design components: MM-Token Re-Composition for text-video interaction, per-frame registers, a lightweight temporal attention branch, and a reward-learning fine-tuning stage. The manuscript includes component ablations, architecture variants, and a user study against SOTA baselines.

Significance. If the efficiency and quality claims hold, M4V would be a meaningful step toward practical T2V generation with sub-quadratic mixers, and the detailed ablations (Tables 2–4, Figures 4–5) and clearly documented training stages are valuable for the community. The paper is commendably transparent about the architecture's remaining attention blocks and the training-latency penalty of a fully Mamba variant (Appendix A.2). However, the central efficiency claim is currently supported only by a partial FLOP analysis and an indirect wall-clock comparison, and the 'public data' statement in the abstract conflicts with the actual training-data description in Section 4.1. The significance of the quality result is therefore bounded by these unresolved issues.

major comments (3)
  1. [§4.2, Table 3, Appendix A.2] The headline efficiency claim ('significantly lowering computational costs') is not established by the reported evidence. The 45% FLOP reduction is computed only for mixer layers (caption of Table 3), excluding MLPs, embeddings, the 3D VAE, and the text encoder. More importantly, the retained L=8 dual-stream attention blocks and the temporal attention branch are part of the deployed model, so the paper never measures the end-to-end cost of M4V against an identical attention-based model at the same scale. Appendix A.2 explicitly reports that replacing all dual-stream blocks with Mamba reduces TFLOPs (29.52 → 22.416) yet increases per-iteration training latency by ~1.5×, demonstrating that the FLOP metric does not track practical speed. The wall-clock numbers in Table 3 compare fast-evaluation variants trained for 20k steps at 384p, not the final progressive-training recipe, and the 226 s vs 296 s comparison is against a different model (PyramidFlow) rather than the same architecture. I request either an end-to-end FLOP count or a reproducible wall-clock/energy benchmark for the final model versus an identical attention baseline, or a revised claim that restricts the savings to mixer-layer arithmetic.
  2. [§4.1, Abstract] The abstract states that M4V is trained 'using only publicly available datasets', but Section 4.1 lists 40M Midjourney-synthetic images, 40M Instagram images, 10M internally curated portrait images, and 80K HunyuanVideo-synthesized clips in addition to public video data. This contradiction matters because the VBench comparison in Table 1 (81.55 vs PyramidFlow† 81.61, and 81.91 with reward learning) is confounded: PyramidFlow† is trained on 10M public data, while M4V benefits from additional synthetic and proprietary data. The authors must either correct the public-data claim or report a same-data comparison that isolates the architectural contribution.
  3. [§4.4, Tables 2–3] The component-wise ablation and architecture-variant results are reported on a 'customized VBench' subset (50 prompts) using an arithmetic mean of seven metrics, but some of these metrics have a very different scale (Overall-Consistency around 20–26 vs other metrics around 90–99). As a result, the 'Avg. Score' in Table 3 is dominated by Overall-Consistency and does not reflect the weighting scheme of VBench (where Dynamic Degree is down-weighted to 0.5 and the 4:1 quality-semantic ratio is used, as described in Section B.1). This makes it hard to interpret the claim that Full+Temp-Branch (76.18) is the best variant; the differences between Full (74.43), Post-half (74.91), and Full Attn (75.86) are within the noise of a 50-prompt, 20k-step fast protocol. I recommend reporting the individual per-metric values (or the official VBench weighting) for the architecture variants, and adding a statement about statistical significance.
minor comments (7)
  1. [Abstract/§3.2] The term 'Mamba-based framework' is misleading given that the final model retains L=8 dual-stream attention blocks and a temporal attention branch; consider using 'hybrid Mamba-attention' throughout.
  2. [Figure 2] The caption contains a typo: 'strcture' should be 'structure'; also 'ommit' should be 'omit' in the same caption.
  3. [§3.3] The per-frame registers are conceptually similar to the registers proposed in Mamba-R [43]; that work should be cited at the introduction of the per-frame register concept.
  4. [Table 1] The notation for M4V⋆ is unclear: the ⋆ footnote says 'with reward learning and including 80K generated samples in the final training stage,' but it is not stated whether the 80K samples are the same as the synthetic data described in Section 4.1; please clarify the exact composition.
  5. [§4.2] The wall-clock comparisons for CogVideoX-5B and HunyuanVideo are quoted at different resolutions and frame counts (720×480×49 and 720×1280×129), which makes the comparison of absolute timings of limited value; at minimum, state the hardware and inference settings (steps, guidance, batch size) for each.
  6. [§4.5] The ablation of reward learning (Table 4) reports a Total Score gain from 81.55 to 81.71 with reward learning alone, but no error bars or confidence intervals are given; given the small differences, a significance test or multiple-seed evaluation would strengthen the claim.
  7. [§B.1] The description of VBench normalization says 'dynamic-degree is down-weighted to 0.5' but the table shows Dynamic Degree as one of the Quality metrics; the text and table are consistent, but a reader could mistake the down-weighting as applying to the Total Score rather than the Quality Score.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the FLOP arithmetic, VBench comparisons, and reward-learning evaluations are self-contained and do not reduce to the paper's own fitted inputs or to load-bearing self-citations.

full rationale

The paper's central claims are (i) a 45% mixer-layer FLOP reduction for the MM-DiM blocks relative to an attention-based alternative, (ii) VBench quality on par with PyramidFlow, and (iii) reward-learning improvements. Each is supported by independent or internally consistent evidence rather than by definitional equivalence. The FLOP comparison in Table 3 is an arithmetic calculation over the paper's own architectures (Full Attn: 55.44 TFLOPs at 241 frames vs. Full+Temp-Branch: 29.52 TFLOPs), so the 45% figure follows from the layer definitions and complexity formulas, not from fitting any parameter to the claimed outcome. The VBench comparison in Table 1 uses an external benchmark and a re-run PyramidFlow baseline. The reward-learning stage in Section 3.4, Eq. 5, optimizes external reward models HPSv2 and CLIP and is then evaluated with VBench and a user study, so the evaluation signal is not identical to the training objective. Self-citations to Matten [13] and FlexVar [20] appear only in related-work context and are not load-bearing. The paper itself flags in Section A.2 and the Conclusion that attention blocks remain and that full Mamba replacement increases training latency by roughly 1.5x per iteration, which weakens the practical efficiency claim but is a correctness and scope concern, not circularity. No equation in the paper reduces to its own input, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central efficiency claim is essentially an arithmetic consequence of replacing attention mixers with Mamba plus a small temporal attention branch. The central quality claim rests on training choices and evaluation assumptions: a specific architecture ratio, data mixture, reward weights, and the validity of VBench comparisons across unequal training sets. The only genuinely new postulated component is the Per-Frame Register tokens, which are supported only by an internal ablation.

free parameters (5)
  • L:N block ratio = 1:2 (L=8 dual-stream attention blocks, N=16 single-stream Mamba blocks, approximate)
    Chosen by hand to balance multimodal alignment against computational savings; Section 3.2 notes full Mamba replacement increases training latency by about 1.5x per iteration.
  • Image:video training mix ratio = 1:8
    Selected after observing that 1:1 ratio causes subject deformation in later frames and low ratio causes forgetting of first-frame quality (Section 4.5).
  • Reward loss weight = 0.1
    Chosen for reward fine-tuning with HPSv2 and CLIP; Section D.1.
  • Corruption noise schedule parameters = not reported (maximum and minimum corruption scales sampled in [0, eta])
    Linear per-frame corruption noise is introduced for training stability, but the actual eta range is not given (Section A.3).
  • Self-guidance weights = not reported for w_p and w_sqg
    Optional inference-time guidance (Eq. 7-9); not used for main results but adds unstated hyperparameters.
assumptions (5)
  • domain assumption Mamba's selective state-space model can model 2D spatial and temporal video structure when tokens are reordered into 1D sequences
    The whole method relies on this; Section 3.3 assumes zigzag scans plus per-frame registers are sufficient for spatiotemporal awareness.
  • domain assumption PyramidFlow's autoregressive compressed-latent conditioning and piecewise flow matching are a valid base for high-quality video generation
    M4V inherits the PyramidFlow pipeline (Eq. 1, Eq. 2) without re-validating it.
  • domain assumption VBench scores are comparable across models trained on different data and are a reliable proxy for generation quality
    Main comparisons use VBench leaderboard numbers from diverse models and data (Section 4.1, B.1).
  • domain assumption HPSv2 and CLIP reward models provide gradients that improve real video frame quality
    Reward learning (Eq. 5) assumes these image-based reward models transfer to video frames.
  • standard math Standard complexity counts for attention and SSM are valid
    Used in Section 4.2 for the 45% FLOP claim: attention O((TM)^2) vs SSM O(TM) plus temporal attention O(T^2).
invented entities (1)
  • Per-Frame Registers
    purpose: Learnable tokens inserted between conditional frames to signal next-frame boundaries and resolution-stage changes; added to help Mamba track temporal position and autoregressive order.
    The only evidence is the ablation in Table 2, which is internal to this paper and uses the same VBench-style evaluation; no independent external test is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of M4V: Multimodal Mamba for Efficient Text-to-Video Generation." pith.science (2026). https://pith.science/paper/F4TWELCR

@misc{pith2026250610915,
  author       = {Pith},
  title        = {Pith review of: M4V: Multimodal Mamba for Efficient Text-to-Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F4TWELCR}},
  note         = {Machine review of arXiv:2506.10915}
}
read the original abstract

Text-to-video generation has significantly enriched content creation and holds the potential to evolve into powerful world simulators. However, modeling the vast spatiotemporal space remains computationally demanding, particularly when employing Transformers, which incur quadratic complexity in sequence processing and thus limit practical applications. Recent advancements in linear-time sequence modeling, particularly the Mamba architecture, offer a more efficient alternative. Nevertheless, its plain design limits its direct applicability to multimodal and spatiotemporal video generation tasks. To address these challenges, we introduce M4V, a multimodal Mamba framework for efficient text-to-video generation. Specifically, a MultiModal diffusion Mamba (MM-DiM) block is designed within the framework to enable seamless integration of multimodal information and spatiotemporal modeling. In detail, we introduce a novel multimodal token re-composition design, which employs a bidirectional scheme for multimodal information integration through simple token arrangement, along with visual registers to enhance spatialtemporal consistency. As a result, the MM-DiM blocks in M4V reduce FLOPs by 45% compared with the attention-based alternative when generating videos at 768x1280 resolution. Additionally, several training strategies are explored in this work to provide a better understanding of training text-to-video models using only publicly available datasets. Extensive experiments on text-to-video benchmarks demonstrate M4V's ability to produce high-quality videos while significantly lowering computational costs. Project page: https://huangjch526.github.io/M4V_project/.

Figures

Figures reproduced from arXiv: 2506.10915 by the authors.

Figure 1
Figure 1. Comparison of FLOPS between full attention [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Overview of the generation architecture. (b) Detailed strcture of our MM-DiM Block. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. User study between Ours, T2V-Turbo, CogvideoX, HunyanVideo and Pyramidflow. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Ablation study of different ratio for image/video hybrid training and reward learning. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visualization of text-to-video generation results which are generated at 5s, 768p, 24fps. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Sptial scan paths for Mamba. A.2 Overall Architecture We retain L = 8 dual-stream blocks from MM-DiT [9], which use separate parameter sets for different modalities. While replacing all dual-stream modules with Mamba blocks could theoretically reduce TFLOPs further (fr…
Figure 7
Figure 7. Figure 7: Detail results of user study (A: PyramidFlow, B: ours, C: HuanyuanVideo, D: CogVideoX, [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Visualization of text-to-video generation results which are generated at 5s, 768p, 24fps. [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Visualization of image-to-video generation results which are generated at 5s, 768p, 24fps. [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MobileWan: Closing the Quality Gap for Mobile Video Diffusion

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A 5B-parameter video diffusion transformer is made deployable on mobile hardware via recurrence distillation, learnable head pruning, step distillation, and decoder optimization, achieving 83.79 VBench at 20s latency.

  2. SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Parameter-free centroid compensation plus error-aware block routing yields a better quality–density Pareto frontier for sparse attention in video DiTs than score-based sparsification.

Reference graph

Works this paper leans on

55 extracted references · 14 canonical work pages · cited by 2 Pith papers

  1. [1]

    Pika art.https://pika.art, 2024

    Pika art. Pika art.https://pika.art, 2024

    work page 2024

  2. [2]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. InProceedings of the IEEE/CVF international conference on computer vision, pages 1728–1738, 2021

    2021

  3. [3]

    Flux.https://github.com/black-forest-labs/flux, 2023

    Labs Black Forest. Flux.https://github.com/black-forest-labs/flux, 2023

    work page 2023

  4. [4]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. 2024. URL https://openai. com/research/video-generation-models-as-world-simulators, 3, 2024

    2024

  5. [5]

    Video mamba suite: State space model as a versatile alternative for video understanding.arXiv preprint arXiv:2403.09626, 2024

    Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei, Zhe Chen, Zhiqi Li, Jiahao Wang, Kunchang Li, Tong Lu, and Limin Wang. Video mamba suite: State space model as a versatile alternative for video understanding.arXiv preprint arXiv:2403.09626, 2024

    arXiv 2024

  6. [6]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7310–7320, 2024

    2024

  7. [7]

    Goku: Flow based video generative foundation models.arXiv preprint arXiv:2502.04896, 2025

    Shoufa Chen, Chongjian Ge, Yuqi Zhang, Yida Zhang, Fengda Zhu, Hao Yang, Hongxiang Hao, Hui Wu, Zhichao Lai, Yifei Hu, Ting-Che Lin, Shilong Zhang, Fu Li, Chuan Li, Xing Wang, Yanghua Peng, Peize Sun, Ping Luo, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Goku: Flow based video generative foundation models.arXiv preprint arXiv:2502.04896, 2025

    arXiv 2025

  8. [8]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    arXiv 2024

Show all 55 references
  1. [9]

    Scaling rectified flow transform- ers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. InForty-first International Conference on Machine...

    2024

  2. [10]

    Vchitect-2.0: Parallel transformer for scaling up video diffusion models.arXiv preprint arXiv:2501.08453, 2025

    Weichen Fan, Chenyang Si, Junhao Song, Zhenyu Yang, Yinan He, Long Zhuo, Ziqi Huang, Ziyue Dong, Jingwen He, Dongwei Pan, et al. Vchitect-2.0: Parallel transformer for scaling up video diffusion models.arXiv preprint arXiv:2501.08453, 2025

    2025 arXiv

  3. [11]

    Dimba: Transformer-mamba diffusion models.arXiv preprint arXiv:2406.01159, 2024

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, Youqiang Zhang, and Junshi Huang. Dimba: Transformer-mamba diffusion models.arXiv preprint arXiv:2406.01159, 2024. 10

    2024 arXiv

  4. [12]

    Lamamba-diff: Linear-time high-fidelity diffusion models based on local attention and mamba.arXiv preprint arXiv:2408.02615, 2024

    Yunxiang Fu, Chaoqi Chen, and Yizhou Yu. Lamamba-diff: Linear-time high-fidelity diffusion models based on local attention and mamba.arXiv preprint arXiv:2408.02615, 2024

    2024 arXiv

  5. [13]

    Matten: Video generation with mamba-attention.arXiv preprint arXiv:2405.03025, 2024

    Yu Gao, Jiancheng Huang, Xiaopeng Sun, Zequn Jie, Yujie Zhong, and Lin Ma. Matten: Video generation with mamba-attention.arXiv preprint arXiv:2405.03025, 2024

    2024 arXiv

  6. [14]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  7. [15]

    Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396, 2021

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396, 2021

    2021 arXiv

  8. [16]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  9. [17]

    Video diffusion models.Advances in Neural Information Processing Systems, 35:8633–8646, 2022

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models.Advances in Neural Information Processing Systems, 35:8633–8646, 2022

    2022

  10. [18]

    Zigma: A dit-style zigzag mamba diffusion model

    Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Björn Ommer. Zigma: A dit-style zigzag mamba diffusion model. In European Conference on Computer Vision, pages 148–166. Springer, 2024

    2024

  11. [19]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  12. [20]

    Flexvar: Flexible visual autoregressive modeling without residual prediction.arXiv preprint arXiv:2502.20313, 2025

    Siyu Jiao, Gengwei Zhang, Yinlong Qian, Jiancheng Huang, Yao Zhao, Humphrey Shi, Lin Ma, Yunchao Wei, and Zequn Jie. Flexvar: Flexible visual autoregressive modeling without residual prediction.arXiv preprint arXiv:2502.20313, 2025

    2025

  13. [21]

    Pyramidal flow matching for efficient video generative modeling.arXiv preprint arXiv:2410.05954, 2024

    Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling.arXiv preprint arXiv:2410.05954, 2024

    2024

  14. [22]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  15. [23]

    Kling ai: Next-generation ai creative studio

    Kling Ai Kuaishou. Kling ai: Next-generation ai creative studio. https://kling.kuaishou. com, 2024

    2024

  16. [24]

    T2v-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback, 2024

    Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sugato Basu, Wenhu Chen, and William Yang Wang. T2v-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback, 2024

    2024

  17. [25]

    Video- mamba: State space model for efficient video understanding

    Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Video- mamba: State space model for efficient video understanding. InEuropean Conference on Computer Vision, pages 237–255, 2024

    2024

  18. [26]

    Mamba-nd: Selective state space modeling for multi-dimensional data

    Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-nd: Selective state space modeling for multi-dimensional data. InEuropean Conference on Computer Vision, pages 75–92. Springer, 2025

    2025

  19. [27]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  20. [28]

    Step-video-t2v technical report: The practice, challenges, and future of video foundation model.arXiv preprint arXiv:2502.10248, 2025

    Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, et al. Step-video-t2v technical report: The practice, challenges, and future of video foundation model.arXiv preprint arXiv:2502.10248, 2025. 11

    2025 arXiv

  21. [29]

    Openvid-1m: A large-scale high-quality dataset for text-to-video generation

    Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02371, 2024

    2024 arXiv

  22. [30]

    Ssm meets video diffusion models: Efficient video generation with structured state spaces

    Yuta Oshima, Shohei Taniguchi, Masahiro Suzuki, and Yutaka Matsuo. Ssm meets video diffusion models: Efficient video generation with structured state spaces. In5th Workshop on practical ML for limited/low resource settings, 2024

    2024

  23. [31]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023

    2023

  24. [32]

    Open-sora plan.https://github.com/PKU-YuanGroup, 2024

    Tuzhan AI PKU-Yuan Lab. Open-sora plan.https://github.com/PKU-YuanGroup, 2024

    2024

  25. [33]

    Long-context state-space video world models.arXiv preprint arXiv:2505.20171, 2025

    Ryan Po, Yotam Nitzan, Richard Zhang, Berlin Chen, Tri Dao, Eli Shechtman, Gordon Wetzstein, and Xun Huang. Long-context state-space video world models.arXiv preprint arXiv:2505.20171, 2025

    2025 arXiv

  26. [34]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  27. [35]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  28. [36]

    Runway gen-3 alpha

    Runway. Runway gen-3 alpha. https://runwayml.com/research/ introducing-gen-3-alpha, 2024

    2024

  29. [37]

    Magi-1: Autoregressive video generation at scale, 2025

    Sand-AI. Magi-1: Autoregressive video generation at scale, 2025

    2025

  30. [38]

    Laion- 5b: An open large-scale dataset for training next generation image-text models.Advances in Neural Information Processing Systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models.Advances in Neural Informa...

    2022

  31. [39]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction.Advances in neural information processing systems, 37:84839–84865, 2024

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction.Advances in neural information processing systems, 37:84839–84865, 2024

    2024

  32. [40]

    Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024

    2024 arXiv

  33. [41]

    Attention is all you need.Advances in Neural Information Processing Systems, 2017

    A Vaswani. Attention is all you need.Advances in Neural Information Processing Systems, 2017

    2017

  34. [42]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, and Di Chen. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

  35. [43]

    Mamba-r: Vision mamba also needs registers.arXiv preprint arXiv:2405.14858, 2024

    Feng Wang, Jiahao Wang, Sucheng Ren, Guoyizhe Wei, Jieru Mei, Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie. Mamba-r: Vision mamba also needs registers.arXiv preprint arXiv:2405.14858, 2024

    2024 arXiv

  36. [44]

    Lingen: Towards high-resolution minute-length text-to-video generation with linear computational complexity.arXiv preprint arXiv:2412.09856, 2024

    Hongjie Wang, Chih-Yao Ma, Yen-Cheng Liu, Ji Hou, Tao Xu, Jialiang Wang, Felix Juefei-Xu, Yaqiao Luo, Peizhao Zhang, Tingbo Hou, et al. Lingen: Towards high-resolution minute-length text-to-video generation with linear computational complexity.arXiv preprint arXiv:2412.09856, 2024

    2024 arXiv

  37. [45]

    The mamba in the llama: Distilling and accelerating hybrid models.arXiv preprint arXiv:2408.15237, 2024

    Junxiong Wang, Daniele Paliotta, Avner May, Alexander M Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models.arXiv preprint arXiv:2408.15237, 2024

    2024 arXiv

  38. [46]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341, 2023

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341, 2023. 12

    2023 arXiv

  39. [47]

    Easyanimate: A high-performance long video generation method based on transformer architecture.arXiv preprint arXiv:2405.18991, 2024

    Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. Easyanimate: A high-performance long video generation method based on transformer architecture.arXiv preprint arXiv:2405.18991, 2024

    2024

  40. [48]

    Perflow: Piecewise rectified flow as universal plug-and-play accelerator.arXiv preprint arXiv:2405.07510, 2024

    Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal plug-and-play accelerator.arXiv preprint arXiv:2405.07510, 2024

    2024 arXiv

  41. [49]

    Perflow: Piecewise rectified flow as universal plug-and-play accelerator.Advances in Neural Information Processing Systems, 37:78630–78652, 2025

    Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal plug-and-play accelerator.Advances in Neural Information Processing Systems, 37:78630–78652, 2025

    2025

  42. [50]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  43. [51]

    From slow bidirectional to fast autoregressive video diffusion models.arXiv preprint arXiv:2412.07772, 2, 2024

    Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models.arXiv preprint arXiv:2412.07772, 2, 2024

    2024

  44. [52]

    Slca: Slow learner with classifier alignment for continual learning on a pre-trained model

    Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 19148–19158, 2023

    2023

  45. [53]

    Slca++: Unleash the power of sequential fine-tuning for continual learning with pre-training.arXiv preprint arXiv:2408.08295, 2024

    Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca++: Unleash the power of sequential fine-tuning for continual learning with pre-training.arXiv preprint arXiv:2408.08295, 2024

    2024 arXiv

  46. [54]

    Open-sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024

    2024

  47. [55]

    downward first, then rightward

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In Forty-first International Conference on Machine Learning, 2024. 13 A Implementation Details A.1 ...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.