Pith. sign in

REVIEW 3 major objections 6 minor 43 references

MOOSE: Pay Attention to Temporal Dynamics for Video Understanding via Optical Flows

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A video encoder that fuses frozen image and optical-flow features beats heavier models on Kinetics-400 and SSv2.

desk verdict Sensible architecture, but the missing flow-removal ablation and overclaimed SOTA make the central result unsupported as written. read the letter →

arxiv 2506.01119 v1 pith:7GPQ4VJ3 submitted 2025-06-01 cs.CV

Hong Nguyen , Dung Tran , Hieu Hoang , Phong Nguyen , Shrikanth Narayanan This is my paper
classification cs.CV
keywords videounderstandingopticalflowtemporalmodelingvisiontransformercross-attentionactionrecognitioninterpretabilitycomputationalefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MOOSE is a video encoder that treats motion as its own modality: it computes optical flow between consecutive frames and feeds those flow fields through a small trainable vision transformer, while a frozen image transformer encodes the frames themselves. The paper's central claim is that this explicit, interpretable motion pathway lets the model match or beat much larger video models on action recognition, reaching 70.84% top-1 on Kinetics-400 and 65.23% top-1 on Something-Something v2 using only 132 GPU hours of training. With the spatial and flow backbones frozen, the only learned parts are the flow encoder, the fusion layer, and the classification head. The authors argue this makes temporally aware video understanding affordable and inspectable, with attention maps that track moving actors instead of scattering over background.

What carries the argument

The load-bearing mechanism is treating optical flow as a second image-like modality and aligning it patch-to-patch with a frozen spatial vision transformer. Each frame is paired with the velocity field computed by the frozen RAFT flow estimator; both are split into equal-size patches, and bidirectional cross-attention lets each spatial patch exchange information with its corresponding flow patch. Arrow masking enforces the inductive bias that a visual patch should attend only to its own motion vector, with the first row and column allowing the class tokens to view the full patch spaces. A causal self-attention layer then aggregates the fused space-time unit embeddings into a clip-level representation, preserving temporal order instead of averaging it away.

What would settle it

Replace the true optical-flow inputs with random vector fields matched in patch magnitude while keeping every other component of the architecture, training data, and schedule identical, then compare Kinetics-400 and SSv2 top-1 accuracy; if accuracy does not drop substantially, the claimed dependence on optical-flow content is not load-bearing.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a video's temporal dynamics can be decomposed into a sequence of velocity fields — optical flow images — that are fused with spatial embeddings at matching patch positions, so a pretrained image encoder keeps its strong static semantics while a lightweight flow encoder contributes motion. The implementation builds space-time units, each holding one frame plus its flow field; arrow-masked bidirectional cross-attention lets each spatial patch read its own flow patch while the classification tokens see both modalities, and a causal self-attention over the sequence of fused units produces the video-level embedding. The paper reports this design beats TimeSformer, VideoMAE, ViViT, and 3D CNN baselines on Kinetics-400 and SSv2 with dramatically lower training cost, and its attention visualizations show consistent focus on moving limbs and objects. The authors also report that performance falls sharply when optical flow is unreliable, as in low-resolution real-time MRI of speech.

Load-bearing premise

The method assumes that optical flow between consecutive frames captures all meaningful change — that motion is local intensity transitions and the frozen flow estimator gets them right — so when flow is weak or noisy, as in low-resolution medical MRI, the video representation degrades.

Editorial extensions

If this is right

  • A competitive video model can be trained from frozen image and optical-flow encoders, cutting training cost to a fraction of standard video pretraining while improving accuracy on motion-heavy benchmarks.
  • The memory footprint scales to long clips and high resolution: the paper reports supporting 128 input frames and 1024×1024 inputs on a 48 GB GPU, where TimeSformer hits memory limits at 16 frames.
  • Attention heatmaps concentrate on actors' moving parts across frames, indicating the learned representation is temporally grounded and usable for interpretable analysis in clinical or behavioral settings.
  • The accuracy gain is largest on fine-grained temporal reasoning (Something-Something v2) relative to scene-driven Kinetics-400, suggesting the method is specifically suited to motion-centric actions.
  • On low-resolution real-time MRI of speech articulation, MOOSE trails TimeSformer by 7.65 points, a concrete boundary case when optical flow estimation is unreliable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer the architecture implies a general recipe: any frozen image encoder can be upgraded to a video encoder by attaching a small trainable flow encoder aligned through patch correspondence, which would make video understanding feasible without video-scale pretraining.
  • This suggests the arrow-masking constraint — forcing each spatial patch to read only its own motion vector — is likely the main source of both the interpretability and part of the accuracy gain; ablating the mask by allowing unrestricted cross-patch attention would test that directly.
  • The drop in performance on weak-flow domains like rtMRI implies the method's reach is bounded by optical-flow quality, so pairing it with an adaptive or trainable flow estimator could extend the gains to low-contrast and medical imagery.
  • The 132-GPU-hour training cost suggests per-task video encoders may become trainable on modest hardware, which could shift how motion-centric benchmarks are approached by groups without large compute budgets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces MOOSE, a video encoder that combines a frozen DINOv2 spatial Vision Transformer with a frozen RAFT optical-flow estimator and a trainable temporal ViT that processes flow patches. Spatial and flow patch embeddings are fused through cross-attention variants (visual-prior, motion-prior, and bidirectional concat), and clip-level embeddings are aggregated with causal attention. The authors report top-1/top-5 accuracies on Kinetics-400 (70.84/89.56) and Something-Something v2 (65.23/89.96), claim state-of-the-art performance with only 132 GPU hours, and present attention heatmaps, t-SNE visualizations, memory-scaling experiments, and additional results on rtMRI-Phoneme, AV-ASD, HAA500, and UCF-101. A Limitations section acknowledges fragility to low frame rates, camera shake, and atypical motion patterns.

Significance. The approach is practically attractive: reusing frozen pretrained image and flow encoders is a plausible route to cheap temporal modeling, and the cross-attention design can produce interpretable flow-aligned attention maps. The paper ships code, reports memory-scaling comparisons, and includes qualitative visualizations, which are useful contributions. However, the central empirical claim—that explicit optical-flow integration is responsible for the reported accuracy gains—is not supported without a no-flow control, and the state-of-the-art comparison table contains implausible baseline numbers. The significance is therefore conditional on the authors adding the missing ablation and correcting the comparison protocol.

major comments (3)
  1. [Section 4.4, Tables 2 and 3; Section 4.2, Table 1] The paper never ablates the flow pathway. Every configuration in Tables 2 and 3 (Bidirectional Concat, Visual-Prior, Motion-Prior, Mean, Causal, Mamba) retains the frozen DINOv2 spatial encoder plus the optical-flow temporal branch. The reported improvement over TimeSformer (70.84 vs 68.72 on K400; 65.23 vs 63.88 on SSv2) could therefore be entirely due to the stronger spatial backbone rather than to temporal flow modeling. A control that removes or zeroes the flow branch (for example, DINOv2 spatial embeddings alone with a classifier, or flow patches replaced by zeros) is required before the paper's central attribution claim is supported; the Mean-aggregation row in Table 3 is not such a control because it still includes the flow branch.
  2. [Section 4.2, Table 1] Several baseline entries are inconsistent with the cited publications and appear to be protocol-mismatched or mis-transcribed. VideoMAE is listed at 59.18% top-1 on K400 and 49.85% on SSv2, whereas the published VideoMAE ViT-B results are substantially higher; ViViT-B is listed with 0.73M parameters and VideoMAE with 0.72M parameters, which are implausible for ViT-B video transformers. Because the 'state-of-the-art' claim is quantified against these numbers, the comparison must be redone with standard reported results or rerun under a matched protocol, and the parameter and TFLOP entries must be corrected.
  3. [Abstract, Section 4.2, Table 4, and Limitations] The abstract claims broad applicability across clinical and medical domains, but on the rtMRI-Phoneme dataset MOOSE underperforms TimeSformer by 7.65 points (77.62 vs 85.27). The authors attribute this to weak optical flow estimation at low spatio-temporal resolution. This is a legitimate limitation, but it directly qualifies the broad-applicability claim. The paper should either soften the claim or provide evidence, beyond the current qualitative discussion, that the failure is an implementation-level issue that can be resolved rather than a fundamental limitation of the flow-based approach.
minor comments (6)
  1. [Section 4.1] The text contains a typo, 'DIONv2', which should read 'DINOv2'; the appendix identifies the backbone as a ViT-B/14 distilled with registers, so the body should use consistent naming.
  2. [Table 4 caption and Section 4.2] The statement that comparisons are made under 'identical parameter and compute budgets' is inaccurate because TimeSformer has 121.57M parameters and MOOSE has 101.01M; 'comparable' would be appropriate, or the budgets should actually be matched.
  3. [Section 3.2] The sentence 'As shown in Figure [?]' contains an unresolved figure reference; this should point to an actual figure that depicts the proposed fusion architectures.
  4. [Section 4.2] The phrase '8 frames with an 8×8 frame × sampling rate' is garbled; the actual sampling protocol should be stated unambiguously (for example, 8 frames sampled at 8 frames per second, or the equivalent specification used by the baselines).
  5. [Appendix A.3] The sentence 'Notice that the embeddings dimension of ViTs for spatial much smaller than temporal' appears to be reversed: the spatial ViT has embedding dimension 768 and the temporal ViT has dimension 192.
  6. [Section 4.4 and Table 3] The text refers to 'identical parameter budgets (101.01G)' while Table 3 reports 101.01M for Mean and Mamba and 104.70M for Causal; the units and values should be made consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MOOSE's reported gains come from a trained encoder evaluated on external benchmarks, and the only author-overlapping citation is a benchmark dataset that does not force any result.

full rationale

Walking the paper's derivation chain: Section 3 defines spatial and temporal pathways, and Equations 1-4 give fusion and aggregation formulas, but no equation defines the Kinetics-400 or SSv2 accuracies in terms of the fitted parameters by construction. The temporal ViT, fusion module, and classification head are trained with SGD against ground-truth labels on K400/SSv2 (§4.1, A.5) and evaluated on held-out test sets; the optical-flow pathway is a frozen RAFT model pretrained on Sintel, and the spatial encoder is a frozen DINOv2 model. Neither backbone is fitted to reproduce the benchmark labels, so the reported numbers are genuine empirical outcomes rather than restatements of inputs. Assumption 1 is an explicit modeling assumption about frame sampling, not a self-referential definition, and the paper tests its failure mode on rtMRI (Table 4, Limitations). The only author-overlapping citation is the rtMRI corpus [10], used as an external evaluation benchmark; this is not load-bearing and is externally falsifiable data, not a theorem or fitted parameter, so per the hard rules it does not raise the circularity score. The missing no-flow ablation (a spatial-only DINOv2 baseline without the flow pathway) is a genuine experimental-design confound for attributing the gain to optical flow, but it is a control omission, not a circular reduction. Similarly, the dangling 'Figure [?]' in Section 3.2 is a formatting/correctness issue. The paper's own Limitations and Appendix A.1 acknowledge motion blur, camera shake, and weak rtMRI flow estimation, which are robustness concerns rather than circularity. Under the specified circularity definitions, there is no equation-level or self-citation-chain circularity; the derivation is self-contained against external benchmarks, and the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities; it is a neural architecture using existing pretrained components. The central benchmark claims rest on learned model weights and hand-chosen hyperparameters, listed above.

free parameters (5)
  • 8-frame input clip = 8 frames at 1/8 sampling rate
    Hand-selected default; the paper says 'primarily consider frames sampled at a low rate, typically 1/8 of the original frame rate, with an input size of 8x224x224'.
  • Flow patch size = 14x14
    Matched to DINOv2 patch size; chosen by hand.
  • Temporal ViT embedding dimension = 192
    Chosen as ViT-tiny for flow to reduce compute; not derived.
  • Training hyperparameters (learning rate, epochs, batch size) = LR 0.005 cosine, 20 epochs, batch 96 on A6000 and 16 on GTX 3090
    Standard choices in Appendix A.5; not derived from theory.
  • All trainable weights in temporal ViT, fusion module, and classifier = Not listed; millions of parameters trained on each benchmark
    These parameters are fitted to task data and are what make the benchmark numbers possible; they are not derived from first principles.
assumptions (3)
  • domain assumption Assumption 1: Changes in image intensity between consecutive frames are due only to local intensity transition; context is preserved within frames.
    Stated in Section 3 as the validity condition for optical flow; the whole temporal pathway depends on flow being physically meaningful.
  • domain assumption The frozen DINOv2 encoder trained on ImageNet-1K provides spatial features adequate for action recognition across target domains.
    The spatial pathway is never fine-tuned; the paper relies on ImageNet-pretrained features transferring to K400, SSv2, rtMRI, AV-ASD, HAA500, and UCF-101.
  • domain assumption RAFT optical flow pretrained on Sintel generalizes to each target video domain.
    RAFT is frozen; the authors rely on its flow estimates for clinical MRI and web video, though they note failure on rtMRI.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MOOSE: Pay Attention to Temporal Dynamics for Video Understanding via Optical Flows." pith.science (2026). https://pith.science/paper/7GPQ4VJ3

@misc{pith2026250601119,
  author       = {Pith},
  title        = {Pith review of: MOOSE: Pay Attention to Temporal Dynamics for Video Understanding via Optical Flows},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7GPQ4VJ3}},
  note         = {Machine review of arXiv:2506.01119}
}
read the original abstract

Many motion-centric video analysis tasks, such as atomic actions, detecting atypical motor behavior in individuals with autism, or analyzing articulatory motion in real-time MRI of human speech, require efficient and interpretable temporal modeling. Capturing temporal dynamics is a central challenge in video analysis, often requiring significant computational resources and fine-grained annotations that are not widely available. This paper presents MOOSE (Motion Flow Over Spatial Space), a novel temporally-centric video encoder explicitly integrating optical flow with spatial embeddings to model temporal information efficiently, inspired by human perception of motion. Unlike prior models, MOOSE takes advantage of rich, widely available pre-trained visual and optical flow encoders instead of training video models from scratch. This significantly reduces computational complexity while enhancing temporal interpretability. Our primary contributions includes (1) proposing a computationally efficient temporally-centric architecture for video understanding (2) demonstrating enhanced interpretability in modeling temporal dynamics; and (3) achieving state-of-the-art performance on diverse benchmarks, including clinical, medical, and standard action recognition datasets, confirming the broad applicability and effectiveness of our approach.

Figures

Figures reproduced from arXiv: 2506.01119 by the authors.

Figure 1
Figure 1. (a) Conventional attention-based methods perceive temporal dynamics with cross-spatial [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The proposed video model for atomic activity classification treating time and space as [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Eight continuous frames of original image followed by attention heatmap of (from top [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of attention heatmaps over eight consecutive video frames for [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Attention heatmap of flow space between no masking (left) and arrow masking (right) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Images show (a) TSNE plots of VideoMAE, ViViT, TimeSFormer, MOOSE, respectively and the [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Eight consecutive heatmaps from top to bottom for TimeSformer, ViViT, and MOOSE, [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Eight consecutive heatmaps from top to bottom for TimeSformer, ViViT, and MOOSE, [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Attention heatmap of MOOSE on rtMRI-Phoneme dataset. The articulator is producing [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 30 canonical work pages

  1. [1]

    Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J

    Long Zhao, Nitesh B. Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J. Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, Rachel Hornung, Florian Schroff, Ming-Hsuan Yang, David A. Ross, Huisheng Wang, Hartwig Adam, Mikhail Sirotenko, Ting Liu, and Boqing Gong. Videoprism: a foundational visual encoder for video understanding. InProceedings ...

    work page 2024

  2. [2]

    The kinetics human action video dataset.ArXiv, abs/1705.06950, 2017

    Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Apostol Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset.ArXiv, abs/1705.06950, 2017

    arXiv 2017

  3. [3]

    The human visual system and its role in motion perception

    Frank Cardullo, Barbara Sweet, Ruud Hosman, and Craig Coon. The human visual system and its role in motion perception. InAIAA Modeling and Simulation Technologies Conference, page 6422, 2011

    work page 2011

  4. [4]

    Is space-time attention all you need for video understanding?CoRR, abs/2102.05095, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding?CoRR, abs/2102.05095, 2021

    arXiv 2021

  5. [5]

    Vivit: A video vision transformer.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6816–6826, 2021

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. Vivit: A video vision transformer.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6816–6826, 2021

    work page 2021

  6. [6]

    De Gruyter, Berlin, Boston, 2016

    Sigmund Exner: Explorations into Kinesthetics, Sensation of Movement and Apparent Motion, pages 87–88. De Gruyter, Berlin, Boston, 2016

    work page 2016

  7. [7]

    Action recognition for surveillance applications using optic flow and svm

    Somayeh Danafar and Niloofar Gheissari. Action recognition for surveillance applications using optic flow and svm. In Yasushi Yagi, Sing Bing Kang, In So Kweon, and Hongbin Zha, editors,Computer Vision – ACCV 2007, pages 457–466, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg

    work page 2007

  8. [8]

    Conv3d-based video violence detection network using optical flow and rgb data.Sensors, 24(2):317, 2024

    Jae-Hyuk Park, Mohamed Mahmoud, and Hyun-Soo Kang. Conv3d-based video violence detection network using optical flow and rgb data.Sensors, 24(2):317, 2024

    work page 2024

Show all 43 references
  1. [9]

    A multi-modal egocentric activity recognition approach towards video domain generalization.Sensors, 24(8):2491, 2024

    Antonios Papadakis and Evaggelos Spyrou. A multi-modal egocentric activity recognition approach towards video domain generalization.Sensors, 24(8):2491, 2024

    2024

  2. [10]

    Nayak, and Shrikanth S

    Yongwan Lim, Asterios Toutios, Yannick Bliesener, Ye Tian, Sajan Goud Lingala, Colin Vaz, Tanner Sorensen, Miran Oh, Sarah Harper, Weiyi Chen, Yoonjeong Lee, Johannes Töger, Mairym Lloréns Montesserin, Caitlin Smith, Bianca Godinez, Louis Goldstein, Dani Byrd, Krishna S. Nay...

    2021

  3. [11]

    Kosloski, Siddhi Patel, Zeke A

    Shijian Deng, Erin E. Kosloski, Siddhi Patel, Zeke A. Barnett, Yiyang Nan, Alexander Kaplan, Sisira Aarukapalli, William T. Doan, Matthew Wang, Harsh Singh, Pamela R. Rollins, and Yapeng Tian. Hear me, see me, understand me: Audio-visual autism behavior recognition. IEEE Trans...

    2025

  4. [12]

    Childplay: A new benchmark for understanding children’s gaze behaviour

    Samy Tafasca, Anshul Gupta, and Jean-Marc Odobez. Childplay: A new benchmark for understanding children’s gaze behaviour. pages 20878–20889, 10 2023

    2023

  5. [13]

    Barner, and Roghayeh Leila Barmaki

    Jicheng Li, Vuthea Chheang, Pinar Kullu, Eli Brignac, Zhang Guo, Anjana Bhat, Kenneth E. Barner, and Roghayeh Leila Barmaki. Mmasd: A multimodal dataset for autism intervention analysis. InProceedings of the 25th International Conference on Multimodal Interaction, ICMI ’23, pa...

    2023

  6. [14]

    Reversible vision transformers

    Karttikeya Mangalam, Haoqi Fan, Yanghao Li, Chao-Yuan Wu, Bo Xiong, Christoph Feichten- hofer, and Jitendra Malik. Reversible vision transformers. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10820–10830, 2022

    2022

  7. [15]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6804–6815, 2021. 10

    2021

  8. [16]

    X3d: Expanding architectures for efficient video recognition

    Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 200–210, 2020

    2020

  9. [17]

    A large-scale study on unsupervised spatiotemporal representation learning

    Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3298–3308, 2021

    2021

  10. [18]

    Spatio- temporal collaborative module for efficient action recognition.IEEE Transactions on Image Processing, 31:7279–7291, 2022

    Yanbin Hao, Shuo Wang, Yi Tan, Xiangnan He, Zhenguang Liu, and Meng Wang. Spatio- temporal collaborative module for efficient action recognition.IEEE Transactions on Image Processing, 31:7279–7291, 2022

    2022

  11. [19]

    Quo vadis, action recognition? a new model and the kinetics dataset

    João Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724–4733, 2017

    2017

  12. [20]

    Batch transformer: Look for attention in batch, 2024

    Myung Beom Her, Jisu Jeong, Hojoon Song, and Ji-Hyeong Han. Batch transformer: Look for attention in batch, 2024

    2024

  13. [21]

    Videomae: masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: masked autoencoders are data-efficient learners for self-supervised video pre-training. InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY , USA, 2022. ...

    2022

  14. [22]

    Videomae v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14549– 14560, 2023

    2023

  15. [23]

    Video swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3192–3201, 2022

    2022

  16. [24]

    Pyslowfast

    Haoqi Fan, Yanghao Li, Bo Xiong, Wan-Yen Lo, and Christoph Feichtenhofer. Pyslowfast. https://github.com/facebookresearch/slowfast, 2020

    2020

  17. [25]

    Jampani, Andreas Geiger, and Michael J

    Laura Sevilla-Lara, Yiyi Liao, Fatma Güney, V . Jampani, Andreas Geiger, and Michael J. Black. On the integration of optical flow and action recognition. InGerman Conference on Pattern Recognition, 2017

    2017

  18. [26]

    Henriques

    Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Fe- ichtenhofer, Andrea Vedaldi, and João F. Henriques. Keeping your eye on the ball: trajectory attention in video transformers. InProceedings of the 35th International Conference on Neural Inf...

    2021

  19. [27]

    Memflow: Optical flow estimation and prediction with memory, 2024

    Qiaole Dong and Yanwei Fu. Memflow: Optical flow estimation and prediction with memory, 2024

    2024

  20. [28]

    Reformer: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020

    2001 arXiv

  21. [29]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  22. [30]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II, page 402–419, Berlin, Heidelberg, 2020. Springer-Verlag

    2020

  23. [31]

    D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.), editor,European Conf. on Computer Vision (ECCV), Part IV , LNCS 7577, pages 611–625. Springer-Verlag, October 2012

    2012

  24. [32]

    Vision transformers need registers, 2023

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2023. 11

    2023

  25. [33]

    Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas...

    2023

  26. [34]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The “something something” video datab...

    2017

  27. [35]

    Haa500: Human-centric atomic action dataset with curated videos.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 13445–13454, 2020

    Jihoon Chung, Cheng hsin Wuu, Hsuan ru Yang, Yu-Wing Tai, and Chi-Keung Tang. Haa500: Human-centric atomic action dataset with curated videos.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 13445–13454, 2020

    2021

  28. [36]

    Ucf101: A dataset of 101 human actions classes from videos in the wild.ArXiv, abs/1212.0402, 2012

    Khurram Soomro, Amir Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild.ArXiv, abs/1212.0402, 2012

    2012 arXiv

  29. [37]

    Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  30. [38]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  31. [39]

    Boqiang Zhang

    et al. Boqiang Zhang. Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    2025 arXiv

  32. [40]

    Xu Mingze

    et al. Xu Mingze. Slowfast-llava: A strong training-free baseline for video large language models.arXiv:2407.15841, 2024

    2024 arXiv

  33. [41]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  34. [42]

    Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment.arXiv preprint arXiv:2310.01852, 2023

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment.arXiv preprint arXiv:2310.01852, 2023

    arXiv

  35. [43]

    running” or “jumping

    Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. InInternational Conference on Machine Learning (ICML), 2024. 12 A Supplemental material This section details the experimental setup for training, e...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.