Pith. sign in

REVIEW 2 major objections 6 minor 89 references

MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MoMa claims that a frozen image foundation model can learn full spatial-temporal video dynamics by modulating its features with lightweight Mamba state-space layers, beating prior parameter-efficient adapters while using fewer FLOPs.

desk verdict A sensible Mamba-based PEFT adapter for video that deserves serious review, but the SSv2 advantage over DiST rests on a protocol-mismatched comparison and needs a matched-protocol rerun. read the letter →

arxiv 2506.23283 v1 pith:ROQMLI4T submitted 2025-06-29 cs.CV

Yuhuan Yang , Chaofan Ma , Zhenjie Mao , Jiangchao Yao , Ya Zhang , Yanfeng Wang This is my paper
classification cs.CV
keywords videorecognitionparameter-efficientfine-tuningimagefoundationmodelsMambastatespacemodelspatial-temporalmodelingCLIPadaptersequencemodulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes MoMa, a parameter-efficient adapter that lets a frozen image foundation model such as CLIP do video recognition without full fine-tuning. The central claim is that a lightweight Mamba state-space layer can inject genuine spatial-temporal modeling into the frozen transformer, provided its output is used as a per-token scale and bias that modulates the transformer's own features rather than replacing them. This matters because prior adapters tend to split spatial and temporal processing into separate branches, which the authors argue misses the full joint dynamics of video. MoMa reports higher accuracy than the main adapter baselines on Kinetics-400, Something-Something-v2, long-video, and zero-shot benchmarks, with lower FLOPs and only the SSM layers trainable.

What carries the argument

The load-bearing object is SeqMod, a sequence-to-sequence modulation operation that extends adaptive normalization (AdaN/FiLM) from scalars to full tensors: $\mathrm{SeqMod}(x,y_1,y_2)=y_1\odot x+y_2+x$. Here $x$ is the window-attended output of the frozen CLIP layer and $(y_1,y_2)$ are the two channel-split outputs of a Mamba SSM layer that scans the video sequence bidirectionally in spatial and temporal dimensions. SeqMod is what carries the argument: it lets the trainable SSM change each token's importance and offset without directly writing a new feature map into the frozen transformer, and the skip connection keeps the pre-trained CLIP forward pass intact. The surrounding Divide-and-Modulate schedule (windowed attention, then SSM modulation, then the frozen feed-forward block) is the packaging that makes the injection cheap.

What would settle it

A controlled experiment that shuffles the temporal order of frames during training on Something-Something-v2 would settle the claim: if MoMa's accuracy barely drops, then the Mamba scans are not learning temporal correspondences, and the full spatial-temporal modeling claim fails even if the adapter still helps.

Watch

Extended reading notes

Core claim

On its own terms, MoMa's discovery is that the right way to attach Mamba to a pre-trained transformer is to modulate, not merge. The framework rewrites each frozen CLIP layer as $V_{i+1}=\mathrm{FFN}(\mathrm{SeqMod}(\mathrm{Divide}(V_i)))$, where Divide runs the frozen attention over small non-overlapping 2D windows and SeqMod is $y_1 \odot x + y_2 + x$, with $y_1,y_2$ produced by bidirectional Mamba scans over the flattened spatial-temporal sequence. Because the modulation is element-wise and the skip connection preserves the original CLIP features, the SSM information is added without disturbing the pre-trained distribution. The paper argues that this is what makes full spatial-temporal modeling possible at linear cost, and it supports the argument with ablations showing SeqMod outperforms additive, max, concatenation, and scalar AdaN fusion schemes.

Load-bearing premise

The method assumes that sweeping a state-space model back and forth over the whole flattened sequence of video patches is enough to capture true temporal dependencies, even though the paper gives no explicit ordering or temporal positional encoding for the scan.

Editorial extensions

If this is right

  • On Kinetics-400 with CLIP ViT-L/14 and 16 frames, MoMa reports 87.8 Top-1 at 4152 GFLOPs, compared with DiST's 87.6 at 4534 GFLOPs.
  • On Something-Something-v2 with 32 frames, MoMa reaches 73.8 Top-1 versus DiST's 73.1, on a dataset that stresses temporal reasoning.
  • On long-video benchmarks, a K400-fine-tuned MoMa reaches 96.9 on Breakfast and 90.0 on COIN at 64 frames, above the VideoMamba numbers reported under the same protocol.
  • Zero-shot transfer from K400 to HMDB51 and UCF101 gives 59.1 and 76.2 with ViT-L/14, ahead of the DiST baseline.
  • Ablations show the alternating [TM]12 layer pattern and the 8x8 window size are the best choices among the variants tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One extension the paper leaves implicit: the same frozen-backbone-plus-SSM-modulation pattern could adapt other pre-trained transformers to long-sequence inputs such as audio, point clouds, or medical video without retraining the backbone.
  • The paper does not specify the token flattening order for the Mamba scans nor add temporal positional encoding, so the full spatial-temporal claim would be strengthened by an ablation that shuffles frames or changes scan order; if accuracy does not drop, the SSM might be doing spatial feature adaptation rather than temporal correspondence.
  • Because only the SSM layers are trainable, the gains might be partly due to task-specific feature rescaling rather than new temporal knowledge; probing the learned scale and bias patterns across frames would test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper presents MoMa, a parameter-efficient fine-tuning method that adapts frozen CLIP image encoders to video recognition. For each transformer layer, a Divide stage splits each frame into non-overlapping 2D windows and applies the frozen CLIP attention within each window; a Modulate stage feeds the sequence through a Mamba SSM layer and applies a learned sequence-wise scale/bias modulation (SeqMod) before the FFN. The method is evaluated on Kinetics-400, Something-Something-v2, Breakfast, COIN, and zero-shot HMDB51/UCF101, with ablations on fusion operation, window size, and layer arrangement.

Significance. If the results hold, the contribution is a useful and inexpensive way to inject global spatiotemporal modeling into frozen image transformers, using linear-complexity SSM rather than quadratic attention. The ablation study is thorough for fusion designs and window/layer patterns, and the reported gains on Breakfast/COIN and zero-shot transfer are informative. However, the headline SSv2 comparison against DiST uses unmatched test views, and the exact SSM scan order is not specified; both issues need to be resolved before the central claims of 'full spatial-temporal modeling' and 'superior performance' can be accepted.

major comments (2)
  1. [Table 2 / Section 4.1] The SSv2 comparison with DiST is not protocol-matched: MoMa ViT-L/14 is evaluated with 32×1×3 views (1 temporal clip, 3 spatial crops) while DiST ViT-L/14 is reported with 32×3×1 views (3 temporal clips, 1 spatial crop). On a temporal-heavy benchmark such as SSv2, the number of temporal clips can substantially change Top-1 accuracy, so the 73.8 vs. 73.1 gap cannot be attributed to the method. Please report a matched-protocol comparison, e.g., both models with 32×1×3 or both with 32×3×1, or explicitly state that the SSv2 comparison is not apples-to-apples.
  2. [Section 3.3 / Figure 2] The paper states that the SSM performs 'multiple times of bidirectional scanning operations through both spatial and temporal dimensions' but never defines the 1D token ordering used for the scan, nor how the 3D feature tensor of shape H×W×T is flattened. Without this, the central claim of 'full spatial-temporal modeling' cannot be verified: a scan that orders all tokens of each frame contiguously and then concatenates frames produces different temporal interactions than a scan that interleaves spatial and temporal indices. Please specify the exact linearization (e.g., t-major vs. hw-major), state whether positional embeddings are added or interpolated for the 40×30 resolution used in Table 6, and clarify how the CLS token is handled in the Divide and Modulate stages.
minor comments (6)
  1. [Section 4.4 / Table 6] The column labels 'Attn(%)' and 'FFN(%)' are not defined, and for the 'Full' row the two values sum to 80.7 rather than 100; please clarify whether these are percentages of total inference time or of some other quantity, and correct the table accordingly.
  2. [Section 3.4 / Implementation Details] The 'CLIP distillation loss' is mentioned but no formula, coefficient, or target (e.g., CLIP text-embedding similarity or logit distillation) is given; please specify this loss, along with the number of epochs and learning-rate schedule for SSv2 and the long-video benchmarks.
  3. [Table 7] The layer-pattern notation is not self-explanatory; for example, [T]12[M]12 appears to place SSM blocks after the entire transformer stack, which conflicts with the per-layer insertion described by Eq. (2). Please clarify what 'M' contains and how the parameter counts compare across patterns.
  4. [Section 3.5] The phrase 'by drawing inspiration from multi-modal fusion techniques, specifically from AdaN' is misleading, because FiLM and AdaIN are feature-wise conditioning layers rather than multi-modal fusion techniques; please rephrase to describe the connection accurately.
  5. [Tables 1-4] The paper reports single-run accuracies without error bars or code; given that some headline margins are under one point (e.g., 87.8 vs. 87.6 on K400), reporting multiple seeds or confidence intervals would substantially strengthen the claims.
  6. [Implementation Details] There is a typo in 'learning rage 3e-4'; it should read 'learning rate 3e-4'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MoMa's claims rest on external benchmarks; the one self-citation is non-load-bearing.

full rationale

MoMa is an empirical adapter architecture; there is no derived quantity that is predicted from fitted parameters in a way that reduces to its own inputs by construction. The central equations (Divide in Eq. 5 and SeqMod in Eq. 10) define forward operations explicitly, and the headline numbers in Tables 1-4 are external evaluations on K400, SSv2, Breakfast, COIN, HMDB51, and UCF101 against published methods. Hyperparameters such as the window size w=8 and the [TM]12 layer pattern are chosen through ablations on the same benchmarks, which is standard hyperparameter selection rather than a fitted parameter renamed as a prediction; no 'prediction' is obtained by feeding the tuned value back into the benchmark as an input. The only self-citation appearing in the method section, 'Similarly, as evidenced in [80], Mamba-based module does not inherently fit Transformer well' (Section 3.5), is redundant with the paper's own Table 5 ablation and is not load-bearing; removing or externalizing that citation would not change the architecture's empirical support. The SSv2 comparison with DiST uses mismatched test views (32x1x3 for MoMa vs 32x3x1 for DiST), which is a protocol-matching concern and a legitimate correctness risk, but it is not circularity: the quoted numbers are measured external results and no reduction-by-construction is present. No circular step can be exhibited under the required standard, so the score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities; it combines existing components (CLIP, Mamba, feature-wise modulation). The main unstated assumptions concern the adequacy of the SSM scan, the handling of temporal order, and the non-disruption of pre-trained features. Free parameters are standard hyperparameters or empirically chosen design values.

free parameters (3)
  • Divide stage window size w = 8
    Chosen from ablations on SSv2 (Table 6) balancing speed and accuracy; directly controls the attention receptive field and computational cost of the Divide stage.
  • SSM hidden dimension = 384
    Set to match CLIP's feature dimension; standard configuration, not fitted to the target datasets.
  • SSM hidden state size = 16
    State dimension for the selective SSM, taken from prior Mamba configurations; no tuning analysis is provided.
assumptions (4)
  • domain assumption Frozen CLIP weights retain useful spatial semantics that can be modulated for video without full fine-tuning.
    This is the motivating premise of PEFT; Section 3.4 states that only SSM layers are trainable and all CLIP parameters are frozen.
  • domain assumption A 1D Mamba SSM with bidirectional scans over a flattened spatial-temporal sequence captures full spatial-temporal interactions.
    Section 3.3 asserts that multiple bidirectional scans through spatial and temporal dimensions are sufficient, but no proof or analysis of coverage is given; this is critical for the 'full spatial-temporal modeling' claim.
  • ad hoc to paper Applying pre-trained CLIP attention within non-overlapping 2D windows preserves the model's feature distribution and does not disrupt pre-trained knowledge.
    Section 3.2 proposes windowing to reduce cost and claims it does not interfere with CLIP features; this is validated indirectly by ablations but not theoretically.
  • domain assumption Temporal order is implicitly captured by the SSM scan without explicit temporal positional encoding.
    The paper never mentions adding temporal position information; the construction of the flattened spatial-temporal sequence in Equations (3) and (5) implicitly assumes the scan order encodes time.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition." pith.science (2026). https://pith.science/paper/ROQMLI4T

@misc{pith2026250623283,
  author       = {Pith},
  title        = {Pith review of: MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ROQMLI4T}},
  note         = {Machine review of arXiv:2506.23283}
}
read the original abstract

Video understanding is a complex challenge that requires effective modeling of spatial-temporal dynamics. With the success of image foundation models (IFMs) in image understanding, recent approaches have explored parameter-efficient fine-tuning (PEFT) to adapt IFMs for video. However, most of these methods tend to process spatial and temporal information separately, which may fail to capture the full intricacy of video dynamics. In this paper, we propose MoMa, an efficient adapter framework that achieves full spatial-temporal modeling by integrating Mamba's selective state space modeling into IFMs. We propose a novel SeqMod operation to inject spatial-temporal information into pre-trained IFMs, without disrupting their original features. By incorporating SeqMod into a Divide-and-Modulate architecture, MoMa enhances video understanding while maintaining computational efficiency. Extensive experiments on multiple video benchmarks demonstrate the effectiveness of MoMa, achieving superior performance with reduced computational cost.

Figures

Figures reproduced from arXiv: 2506.23283 by the authors.

Figure 1
Figure 1. Overview of our proposed MoMa. The Divide stage aim to cut down the computational cost by narrowing the attention range. We utilize the original CLIP attention layers, but splits the input video sequence into smaller windows and processes attention independently for each window. In Modulate stage, we aim to capture full spatial-temporal dynamics using lightweight Mamba. We first forward the sequence through an SSM l… view at source ↗
Figure 2
Figure 2. Detailed architecture of State Space Model (SSM) for￾warding layer. The SSM module is designed to learn two sequences y1 and y2 that are used in further modulation operation. We con￾duct multiple times of bidirectional scanning through both spatial and temporal dimensions. The final projection channel is then doubled, and we split the output into two sequences. By restricting the attention range within each window, … view at source ↗
Figure 3
Figure 3. Some other designs of sequence fusion operations to integrate two sequences x and y. (a) Weighted average operation with hyperparameter w1 and w2; (b) Element-wise max pooling operation; (c) Concatenation in channel dimension. where y1 and y2 are the output of SSM, serving as se￾quential scale and bias; ⊙ is element-wise multiplication. The modulated sequence is then fed into the original CLIP feed-forward layer for… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of speed performance. Compared with full attention method UMT and spatial-temporal separate method AIM, our method, powered by the SeqMod operation, achieves faster processing with a more gradual increase in memory usage. temporal attention separately, first…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

89 extracted references · 49 canonical work pages

  1. [1]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Luˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 6836–6846, 2021

    2021

  2. [2]

    BEiT: BERT Pre-Training of Image Transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT Pre-Training of Image Transformers. arXiv preprint arXiv:2106.08254, 2021

    arXiv 2021

  3. [3]

    Is space-time attention all you need for video under- standing? In Proceedings of the International Confer- ence on Machine Learning (ICML), July 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video under- standing? In Proceedings of the International Confer- ence on Machine Learning (ICML), July 2021

    2021

  4. [4]

    Coyo-700m: Image-text pair dataset, 2022

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset, 2022

    2022

  5. [5]

    Quo vadis, ac- tion recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, ac- tion recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6299–6308, 2017

    2017

  6. [6]

    Video mamba suite: State space model as a versatile alternative for video understand- ing

    Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei, Zhe Chen, Zhiqi Li, Jiahao Wang, Kunchang Li, Tong Lu, and Limin Wang. Video mamba suite: State space model as a versatile alternative for video understand- ing. arXiv preprint arXiv: 2403.09626, 2024

    arXiv 2024

  7. [7]

    A simple framework for con- trastive learning of visual representations, 2020

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations, 2020

    2020

  8. [8]

    Feature-wise transformations

    Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 2018. 9 MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition

    2018

Show all 89 references
  1. [9]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yang- hao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 6824–6835, 2021

    2021

  2. [10]

    X3d: Expanding architec- tures for efficient video recognition

    Christoph Feichtenhofer. X3d: Expanding architec- tures for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 203–213, 2020

    2020

  3. [11]

    Masked autoencoders as spatiotemporal learners

    Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners. arXiv preprint arXiv:2205.09113, 2022

    2022 arXiv

  4. [12]

    Slowfast networks for video recog- nition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recog- nition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6202–6211, 2019

    2019

  5. [13]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yian- ilos, Moritz Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In ...

    2017

  6. [14]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  7. [15]

    Effi- ciently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R ´e. Effi- ciently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021

    2021 arXiv

  8. [16]

    Open-vocabulary object detection via vision and language knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. International Confer- ence on Learning Representations, 2021

    2021

  9. [17]

    Trustworthy machine learning: From data to models

    Bo Han, Jiangchao Yao, Tongliang Liu, Bo Li, Sanmi Koyejo, Feng Liu, et al. Trustworthy machine learning: From data to models. Foundations and Trends® in Privacy and Security, 7(2-3):74–246, 2025

    2025

  10. [18]

    Turbo training with token dropout

    Tengda Han, Weidi Xie, and Andrew Zisserman. Turbo training with token dropout. British Machine Vision Conference, 2022

    2022

  11. [19]

    Learning spatio-temporal features with 3d residual networks for action recognition

    Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learning spatio-temporal features with 3d residual networks for action recognition. In Proceedings of the IEEE international conference on computer vision workshops, pages 3154–3160, 2017

    2017

  12. [20]

    Mambavision: A hybrid mamba-transformer vision backbone

    Ali Hatamizadeh and Jan Kautz. Mambavision: A hybrid mamba-transformer vision backbone. arXiv preprint arXiv:2407.08083, 2024

    2024 arXiv

  13. [21]

    Clipscore: A reference- free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ro- nan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. Con- ference on Empirical Methods in Natural Language Processing, 2021

    2021

  14. [22]

    Arbitrary style trans- fer in real-time with adaptive instance normalization

    Xun Huang and Serge Belongie. Arbitrary style trans- fer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017

    2017

  15. [23]

    Noureldien Hussein, Efstratios Gavves, and Arnold W. M. Smeulders. Videograph: Recognizing minutes- long human activities in videos. arXiv preprint arXiv: 1905.05143, 2019

    1905 arXiv

  16. [24]

    Smeulders

    Noureldien Hussein, Efstratios Gavves, and Arnold W.M. Smeulders. Timeception for com- plex action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019

    2019

  17. [25]

    Long movie clip classification with state-space video mod- els

    Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video mod- els. In Computer Vision – ECCV 2022, pages 87–104. Springer Nature Switzerland, 2022

    2022

  18. [26]

    Scaling up visual and vision- language representation learning with noisy text su- pervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- language representation learning with noisy text su- pervision. In International Conference on Machine Learning, pages 4904–49...

    2021

  19. [27]

    Laine, and Timo Aila

    Tero Karras, S. Laine, and Timo Aila. A style-based generator architecture for generative adversarial net- works. Computer Vision and Pattern Recognition , 2018

    2018

  20. [28]

    The kinetics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017

    2017 arXiv

  21. [29]

    Kuehne, H

    H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. Hmdb: A large video database for human motion recognition. In 2011 International Conference on Computer Vision, pages 2556–2563, 2011

    2011

  22. [30]

    The lan- guage of actions: Recovering the syntax and semantics of goal-directed human activities

    Hilde Kuehne, Ali Arslan, and Thomas Serre. The lan- guage of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014. 10 MoMa: Modulating Mamba for Adapt...

    2014

  23. [31]

    F-vlm: Open-vocabulary object detection upon frozen vision and language models

    Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vlm: Open-vocabulary object detection upon frozen vision and language models. arXiv preprint arXiv: 2209.15639, 2022

    2022 arXiv

  24. [32]

    Mipmap- gs: Let gaussians deform with scale-specific mipmap for anti-aliasing rendering

    Jiameng Li, Yue Shi, Jiezhang Cao, Bingbing Ni, Wen- jun Zhang, Kai Zhang, and Luc Van Gool. Mipmap- gs: Let gaussians deform with scale-specific mipmap for anti-aliasing rendering. arXiv preprint arXiv: 2408.06286, 2024

    2024 arXiv

  25. [33]

    Videomamba: State space model for efficient video understanding, 2024

    Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding, 2024

    2024

  26. [34]

    Unmasked teacher: Towards training-efficient video foundation models

    Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 19948–19960, 2023

    2023

  27. [35]

    Uniformer: Uni- fied transformer for efficient spatial-temporal repre- sentation learning

    Kunchang Li, Yali Wang, Gao Peng, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Uni- fied transformer for efficient spatial-temporal repre- sentation learning. In International Conference on Learning Representations, 2021

    2021

  28. [36]

    Mvitv2: Improved multiscale vision transformers for classification and detection

    Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In 2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), p...

    2022

  29. [37]

    Fo- caldreamer: Text-driven 3d editing via focal-fusion assembly

    Yuhan Li, Yishun Dou, Yue Shi, Yu Lei, Xuanhong Chen, Yi Zhang, Peng Zhou, and Bingbing Ni. Fo- caldreamer: Text-driven 3d editing via focal-fusion assembly. In Proceedings of the AAAI conference on artificial intelligence, pages 3279–3287, 2024

    2024

  30. [38]

    Pointmamba: A simple state space model for point cloud analysis

    Dingkang Liang, Xin Zhou, Wei Xu, Xingkui Zhu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. In Advances in Neural Information Processing Systems, 2024

    2024

  31. [39]

    Jamba: A hybrid transformer-mamba language model

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024

    2024 arXiv

  32. [40]

    Learning to recognize procedural activities with dis- tant supervision

    Xudong Lin, Fabio Petroni, Gedas Bertasius, Mar- cus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning to recognize procedural activities with dis- tant supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 138...

    2022

  33. [41]

    Frozen clip models are efficient video learners

    Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Ger- ard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. arXiv preprint arXiv:2208.03550, 2022

    2022 arXiv

  34. [42]

    Annotation-free audio-visual segmentation

    Jinxiang Liu, Yu Wang, Chen Ju, Chaofan Ma, Ya Zhang, and Weidi Xie. Annotation-free audio-visual segmentation. arXiv preprint arXiv: 2305.11019, 2023

    2023 arXiv

  35. [43]

    Vmamba: Visual state space model

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024

    2024 arXiv

  36. [44]

    Video swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 3202– 3211, 2022

    2022

  37. [45]

    Freesegdiff: Annotation-free saliency segmentation with diffusion models

    Chaofan Ma, Yuhuan Yang, Chen Ju, Yue Shi, Ya Zhang, and Yanfeng Wang. Freesegdiff: Annotation-free saliency segmentation with diffusion models. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2025

    2025

  38. [46]

    Dif- fusionseg: Adapting diffusion towards unsupervised object discovery

    Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Jinxi- ang Liu, Yu Wang, Ya Zhang, and Yanfeng Wang. Dif- fusionseg: Adapting diffusion towards unsupervised object discovery. arXiv preprint arXiv: 2303.09813, 2023

    2023 arXiv

  39. [47]

    Open-vocabulary semantic segmenta- tion with frozen vision-language models

    Chaofan Ma, Yuhuan Yang, YanFeng Wang, Ya Zhang, and Weidi Xie. Open-vocabulary semantic segmenta- tion with frozen vision-language models. In British Machine Vision Conference, 2022

    2022

  40. [48]

    Attrseg: open- vocabulary semantic segmentation via attribute decomposition-aggregation

    Chaofan Ma, Yang Yuhuan, Chen Ju, Fei Zhang, Ya Zhang, and Yanfeng Wang. Attrseg: open- vocabulary semantic segmentation via attribute decomposition-aggregation. Advances in neural in- formation processing systems, 36:10258–10270, 2023

    2023

  41. [49]

    Scaling open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. Advances in Neural Information Processing Systems, 36:72983–73007, 2023

    2023

  42. [50]

    Ron Mokady, Amir Hertz, and Amit H. Bermano. Clip- cap: Clip prefix for image captioning. arXiv preprint arXiv: 2111.09734, 2021

    2021 arXiv

  43. [51]

    Expanding language-image pre- trained models for general video recognition

    Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pre- trained models for general video recognition. arXiv preprint arXiv:2208.02816, 2022. 11 MoMa: Modulating Mamba for Adapting Image Found...

    2022 arXiv

  44. [52]

    Text-only training for image captioning using noise- injected clip

    David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise- injected clip. arXiv preprint arXiv: 2211.00575, 2022

    2022 arXiv

  45. [53]

    Maxime Oquab, Timoth´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicola...

    2023

  46. [54]

    St-adapter: Parameter-efficient image- to-video transfer learning for action recognition

    Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient image- to-video transfer learning for action recognition. arXiv preprint arXiv:2206.13559, 2022

    2022 arXiv

  47. [55]

    Mambasci: Ef- ficient mamba-unet for quad-bayer patterned video snapshot compressive imaging

    Zhenghao Pan, Haijin Zeng, Jiezhang Cao, Yongy- ong Chen, Kai Zhang, and Yong Xu. Mambasci: Ef- ficient mamba-unet for quad-bayer patterned video snapshot compressive imaging. arXiv preprint arXiv: 2410.14214, 2024

    2024 arXiv

  48. [56]

    Dual-path adaptation from image to video transform- ers

    Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Dual-path adaptation from image to video transform- ers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2203– 2213, 2023

    2023

  49. [57]

    Peebles and Saining Xie

    William S. Peebles and Saining Xie. Scalable diffu- sion models with transformers. IEEE International Conference on Computer Vision, 2022

    2022

  50. [58]

    Ethan Perez, Florian Strub, H. D. Vries, Vincent Du- moulin, and Aaron C. Courville. Film: Visual reason- ing with a general conditioning layer. AAAI Confer- ence on Artificial Intelligence, 2017

    2017

  51. [59]

    Dis- entangling spatial and temporal learning for efficient image-to-video transfer learning

    Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, Yingya Zhang, Changxin Gao, Deli Zhao, and Nong Sang. Dis- entangling spatial and temporal learning for efficient image-to-video transfer learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13934–1...

    2023

  52. [60]

    Learning transferable visual models from natural lan- guage supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. In International Conference on Machine Learning...

    2021

  53. [61]

    Token- learner: Adaptive space-time tokenization for videos

    Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Token- learner: Adaptive space-time tokenization for videos. In A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wort- man Vaughan, editors, Advances in Neural Informa- tion Processing Systems, 2021

    2021

  54. [62]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion...

    arXiv

  55. [63]

    Only time can tell: Discovering temporal data for temporal modeling

    Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torre- sani. Only time can tell: Discovering temporal data for temporal modeling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vi- sion, pages 535–544, 2021

    2021

  56. [64]

    Darf: Depth- aware generalizable neural radiance field

    Yue Shi, Dingyi Rong, Chang Chen, Chaofan Ma, Bingbing Ni, and Wenjun Zhang. Darf: Depth- aware generalizable neural radiance field. Displays, 88:102996, 2025

    2025

  57. [65]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.0402, 2012

    2012 arXiv

  58. [66]

    Coin: A large-scale dataset for comprehensive instruc- tional video analysis

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instruc- tional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), June 2019

    2019

  59. [67]

    Dim: Diffusion mamba for efficient high-resolution image synthesis, 2024

    Yao Teng, Yue Wu, Han Shi, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. Dim: Diffusion mamba for efficient high-resolution image synthesis, 2024

    2024

  60. [68]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203.12602, 2022

    2022 arXiv

  61. [69]

    An empirical study of mamba-based language models

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Bran- don Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba-based language models. arXiv preprint arXiv:2406.07887, 2024

    2024 arXiv

  62. [70]

    Sigma: Siamese mamba network for multi-modal semantic 12 MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition segmentation

    Zifu Wan, Pingping Zhang, Yuhao Wang, Silong Yong, Simon Stepputtis, Katia Sycara, and Yaqi Xie. Sigma: Siamese mamba network for multi-modal semantic 12 MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition segmentation. arXiv preprint arXiv: 2404.0...

    2024 arXiv

  63. [71]

    Ac- tionclip: A new paradigm for video action recognition

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Ac- tionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472, 2021

    2021 arXiv

  64. [72]

    Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

    Wenhui Wang et al. Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks. arXiv preprint arXiv:2208.10442, 2022

    2022 arXiv

  65. [73]

    Pointramba: A hybrid transformer-mamba framework for point cloud analysis

    Zicheng Wang, Zhenghao Chen, Yiming Wu, Zhen Zhao, Luping Zhou, and Dong Xu. Pointramba: A hybrid transformer-mamba framework for point cloud analysis. arXiv preprint arXiv: 2405.15463, 2024

    2024 arXiv

  66. [74]

    Dualnerf: Text-driven 3d scene editing via dual- field representation

    Yuxuan Xiong, Yue Shi, Yishun Dou, and Bingbing Ni. Dualnerf: Text-driven 3d scene editing via dual- field representation. arXiv preprint arXiv: 2502.16302, 2025

    2025 arXiv

  67. [75]

    Clip- vip: Adapting pre-trained image-text model to video- language representation alignment

    Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip- vip: Adapting pre-trained image-text model to video- language representation alignment. arXiv preprint arXiv:2209.06430, 2022

    2022 arXiv

  68. [76]

    Multiview transformers for video recognition

    Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3333–3343, 2022

    2022

  69. [77]

    Mambamil: Enhancing long sequence modeling with sequence reordering in computational pathology

    Shu Yang, Yihui Wang, and Hao Chen. Mambamil: Enhancing long sequence modeling with sequence reordering in computational pathology. arXiv preprint arXiv:2403.06800, 2024

    2024 arXiv

  70. [78]

    AIM: Adapting image mod- els for efficient video action recognition

    Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. AIM: Adapting image mod- els for efficient video action recognition. In The Eleventh International Conference on Learning Repre- sentations, 2023

    2023

  71. [79]

    Multi- modal prototypes for open-world semantic segmen- tation

    Yuhuan Yang, Chaofan Ma, Chen Ju, Fei Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Multi- modal prototypes for open-world semantic segmen- tation. International Journal of Computer Vision , 132(12):6004–6020, 2024

    2024

  72. [80]

    Remamber: Referring image segmentation with mamba twister

    Yuhuan Yang, Chaofan Ma, Jiangchao Yao, Zhun Zhong, Ya Zhang, and Yanfeng Wang. Remamber: Referring image segmentation with mamba twister. European Conference on Computer Vision (ECCV) , 2024

    2024

  73. [81]

    Learning with multi- class auc: Theory and algorithms

    Zhiyong Yang, Qianqian Xu, Shilong Bao, Xiaochun Cao, and Qingming Huang. Learning with multi- class auc: Theory and algorithms. IEEE Transac- tions on Pattern Analysis and Machine Intelligence , 44(11):7747–7763, 2021

    2021

  74. [82]

    Optimizing two-way partial auc with an end-to-end framework

    Zhiyong Yang, Qianqian Xu, Shilong Bao, Yuan He, Xiaochun Cao, and Qingming Huang. Optimizing two-way partial auc with an end-to-end framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):10228–10246, 2023

    2023

  75. [83]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. arXiv preprint arXiv: 2303.15343, 2023

    2023 arXiv

  76. [84]

    Point cloud mamba: Point cloud learning via state space model

    Tao Zhang, Xiangtai Li, Haobo Yuan, Shunping Ji, and Shuicheng Yan. Point cloud mamba: Point cloud learning via state space model. arXiv preprint arXiv:2403.00762, 2024

    2024 arXiv

  77. [85]

    G4seg: Generation for inexact segmentation refinement with diffusion models

    Tianjiao Zhang, Fei Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. G4seg: Generation for inexact segmentation refinement with diffusion models. arXiv preprint arXiv:2506.01539, 2025

    2025 arXiv

  78. [86]

    Vidtr: Video transformer without convo- lutions

    Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. Vidtr: Video transformer without convo- lutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13577–13587, 2021

    2021

  79. [87]

    Regionclip: Region-based language-image pretraining

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chun- yuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2022

  80. [88]

    Graph-based high-order relation modeling for long-term action recognition

    Jiaming Zhou, Kun-Yu Lin, Haoxin Li, and Wei-Shi Zheng. Graph-based high-order relation modeling for long-term action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8984–8993, June 2021

    2021

  81. [89]

    Vi- sion mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xin- long Wang, Wenyu Liu, and Xinggang Wang. Vi- sion mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024. 13

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.