Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that adding masked channel mixing around CLIP's attention, plus LLM-built spatial and temporal prompts, yields state-of-the-art zero-shot action recognition on UCF101, HMDB51, and Kinetics-600.

desk verdict Competent, well-ablated ZSAR paper with a real but fixable efficiency-claim problem; the accuracy gains are small, yet the work is better than the usual SOTA churn and deserves a serious referee. read the letter →

arxiv 2412.09895 v2 pith:MKUILP33 submitted 2024-12-13 cs.CV

Yating Yu , Congqi Cao , Yueran Zhang , Qinyi Lv , Lingtong Min , Yanning Zhang This is my paper · ORCID
classification cs.CV
keywords zero-shotactionrecognitionCLIPSpace-timeCrossAttentionSemanticKnowledgeGraphspatiotemporaltextaugmentationvideounderstandinglargelanguagemodelscontrastivelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Zero-shot action recognition asks a model to label video actions it never saw during training, so the model must rely on general visual and semantic knowledge. This paper claims that CLIP, a strong image-text model, can be turned into a spatiotemporal action expert without adding any trainable parameters on the vision side: within each transformer block, a masked, channel-swapped set of tokens is passed through the existing spatial attention, then padded and shortcut back, letting the network see cross-frame motion cues. On the text side, the paper builds an Action Semantic Knowledge Graph by prompting a large language model for each action's objects, sub-actions, and their order, then turns those relations into separate spatial and temporal text prompts. Training aligns frame-level video features with these prompt-level text features, while a frozen CLIP distills its general visual knowledge into the video encoder. On Kinetics-600, UCF101, and HMDB51 under zero-shot protocols, the paper reports consistent gains over prior CLIP-based methods.

What carries the argument

The load-bearing object is Space-time Cross Attention (STCA), a four-step rewrite of CLIP's existing spatial attention. Before the multi-head self-attention, (1) Window Shift Masking drops a fraction of spatial tokens with a mask that shifts along time; (2) Multi-scale Channel Mixing swaps channel slices between the surviving tokens and their neighbors at several temporal offsets, reusing the established 'shift trick' for zero-cost motion exchange. After attention, (3) Spatial Padding restores the original token positions and (4) a residual Short-cutting adds the mixed temporal information back into the main stream. The companion machinery is the Action Semantic Knowledge Graph (ASKG), an LLM-generated graph in which action categories connect to object nodes and sub-action nodes via relations such as 'used to shoot' and 'precedes'; parsing this graph yields the spatial prompts [S] and temporal prompts [T] that the contrastive alignment uses as fine-grained text targets.

What would settle it

Time and profile STDD against its own spatial-only baseline on the same 8-frame clips: if Eq. (7) executes a second multi-head self-attention on the half-masked tokens, total attention FLOPs should be about $(1 + S r^2)$ times spatial-only (with $S=2$, $r=0.5$, roughly 1.5x), and the measured wall-clock or FLOP ratio will show whether the 'no increased computational complexity' claim holds. Separately, zero-shot accuracy on a held-out split with and without Space-time Cross Attention would show how much of the reported gain comes from the temporal mechanism rather than from text augmentation.

Watch

Extended reading notes

Core claim

The central claim is that the two main weaknesses of directly finetuning CLIP for zero-shot action recognition—missing temporal dynamics and overly coarse class-level text—can be fixed simultaneously. The vision fix, Space-time Cross Attention, applies window-shift masking and multi-scale channel mixing to a fraction of the spatial tokens before the existing multi-head self-attention, then pads and residual-shortcuts the result, so temporal information flows through the same parameters CLIP already has. The text fix, spatiotemporal text augmentation from an Action Semantic Knowledge Graph, generates prompts that name the objects involved in an action (spatial prompts) and the ordered sub-actions that compose it (temporal prompts), giving the contrastive alignment finer-grained anchors than a single class name. With these two pieces, the paper claims state-of-the-art zero-shot accuracy on UCF101, HMDB51, and Kinetics-600, and shows the text prompts alone can also lift a baseline model's accuracy when swapped in without retraining.

Load-bearing premise

The load-bearing premise is that the extra attention pass over the masked, channel-mixed tokens costs no meaningful compute, so the total stays 'equal to spatial-only attention'; if that extra pass is a real second forward step, the no-complexity increase selling point weakens.

Editorial extensions

If this is right

  • If the accuracy gains hold, STDD becomes the new state of the art for CLIP-based zero-shot action recognition on UCF101, HMDB51, and Kinetics-600 under the standard protocols.
  • Because the vision-side change adds no parameters, the same Space-time Cross Attention can be dropped into other ViT-based video models, potentially improving action localization or video captioning without retraining a custom architecture.
  • The ASKG prompt set is generated offline by an LLM, so it can be extended to new action categories simply by prompting, without collecting new video data or updating the text encoder.
  • The frame-to-prompt fine-grained alignment used for training also serves as the zero-shot inference score, so the method needs no extra classifier or video-level aggregation at test time beyond averaging frame-prompt matches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test is to measure wall-clock latency and FLOPs on identical hardware: since the mixed-token MHSA of Eq. (7) is a second attention pass, the exact cost is roughly $(1 + S r^2)$ times the spatial-only attention at the same token count; the 'no increased computational complexity' claim is asymptotic and may not match practical timing.
  • The paper's own limitation note—LLM hallucination makes prompt quality vary per category—suggests a testable extension: ground the generated prompts in the video's actual detected objects, or filter low-confidence LLM relations, to reduce variance across action classes.
  • The method aligns frame-level features to prompt-level text; that design should transfer to few-shot action recognition, where a handful of labeled examples could replace or calibrate the LLM prompts, and to open-vocabulary action spotting, where the same score could localize when an unseen action occurs.
  • Because the text prompts decompose actions into ordered sub-actions, the learned alignment might support action anticipation—predicting the next sub-action from the video frames—without any new supervision, a consequence the paper does not draw.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes STDD, a CLIP-based framework for zero-shot action recognition. On the vision side, it augments the spatial self-attention of each ViT block with a four-step operation (Window Shift Masking, Multi-scale Channel Mixing, Spatial Padding, and Short-cutting) to capture cross-frame dynamics without adding parameters. On the text side, it constructs an Action Semantic Knowledge Graph (ASKG) by prompting GPT-3.5 to generate spatial and temporal text prompts. During training, frame-level video representations are aligned with prompt-level text representations and regularized by feature distillation from frozen CLIP. Experiments on UCF101, HMDB51, and Kinetics-600 under three evaluation protocols are reported against prior CLIP-based ZSAR methods.

Significance. The paper's main strength is the systematic combination of a parameter-free temporal attention mechanism with structured LLM-based semantic augmentation, evaluated under a clean K400-training/novel-class protocol. The public code and the detailed appendices, including full LLM prompts and ablations of masking and temporal scales, are useful for reproducibility. If the efficiency claim were accurate, the method would be practically attractive. However, the efficiency claim hides a non-trivial constant factor, and several improvements over prior state-of-the-art are within one standard deviation; both issues are addressable.

major comments (4)
  1. [Abstract / §3 Computational Complexity] The claim that STCA increases no computational complexity rests on the asymptotic expression O(T N^2 + S T (N')^2) = O(T N^2). Asymptotically this is true, but the paper's stronger claim 'without increasing computational complexity' is not supported: Eq. (7) performs a second MHSA on N' tokens for each of S temporal scales, in addition to the full MHSA of Eq. (1). With the implemented r=0.5 and S=2 (scales [±1, ±2]), the attention FLOPs are about (1 + S r^2) = 1.5x spatial-only, plus extra LayerNorm, residual additions, and memory. Please qualify the claim as asymptotic and report actual FLOPs or runtime, or revise the comparison against AIM and full space-time attention.
  2. [§3 Training Objectives] The final training objective is defined only as L_CE 'together with feature distillation loss proposed by (Huang et al. 2024)'; the loss is not defined, its weighting is not given, and no ablation isolates its contribution. Since this term is part of the optimization, the reported results are not fully reproducible. Please provide the loss expression, the hyperparameter value, and an ablation with and without this term.
  3. [§3 Multi-scale Channel Mixing / Implementation Details] The channel-index hyperparameter gamma in Eq. (5), where d_delta = gamma * D, is never reported. It controls how much channel information is mixed at each temporal scale and should be listed in Implementation Details, ideally with an ablation.
  4. [Table 2 / Main Results] The statement that STDD 'consistently surpasses state-of-the-art' is weakened by the reported variances: the gains over the previous best on UCF (85.2 +/- 1.2 vs. 84.8 +/- 1.1), K600 (75.1 +/- 0.7 vs. 74.8 +/- 0.9), and HMDB (55.9 +/- 0.2 vs. 55.6 +/- 1.4) are all within one standard deviation. Please report multiple seeds or statistical significance, or soften the claim accordingly.
minor comments (5)
  1. [Supplementary Material, Table 5] The text refers to 'our MSM' when discussing the masking strategy, but the method is called WSM; this should be corrected.
  2. [Eq. (3)] The periodic function phi(·|·) that generates masking maps is described only in words; the appendix would benefit from pseudocode or an exact procedural definition.
  3. [Figure 3(c)] The caption uses the term 'Window view' without defining the coordinate axes or the meaning of the numbers (1 and 3); please make the figure self-contained.
  4. [References] In the Related Work section on semantic knowledge, one citation is rendered as '?' rather than an actual reference; this needs to be fixed.
  5. [Table 1 caption] The abbreviation 'HMBD' appears in the caption while the text uses 'HMDB51'; please use a single consistent abbreviation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the reported zero-shot gains come from a clean K400-trained evaluation on disjoint benchmarks, not from a fitted input or a load-bearing self-citation.

full rationale

The STDD derivation chain is self-contained with respect to the benchmarks it targets. The vision module (STCA) is built from parameter-free masking/mixing/padding/shortcut operations around the CLIP MHSA, and the claimed efficiency comparison is an asymptotic complexity statement rather than a fitted prediction. The text side uses GPT-generated ASKG prompts derived from class names; these prompts are generated independently of the training videos and of the test labels' visual content, so the zero-shot evaluation on UCF/HMDB/K600 novel classes is not predetermined by a fitted parameter. Training on K400 with disjoint test classes follows the standard Brattoli/Ni protocols, and the alignment losses are standard cross-entropy plus feature distillation. The only self-citations (e.g., Task-Adapter and the gesture-recognition word-embedding works) appear in related-work context and are not used to justify the central claim. The Big-O complexity claim, O(T N^2 + S T (N')^2) = O(T N^2), may be over-generous in constant-factor terms because Eq. (7) adds an extra MHSA pass over mixed tokens, but that is a correctness or consistency concern rather than circular reasoning. No step in the paper reduces by construction to its own input.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method relies on standard transformer components, CLIP transfer from images to video, K400-to-target generalization, the FROSTER distillation loss being valid, and LLM outputs being semantically accurate. It introduces no fitted scalar parameters beyond hand-chosen hyperparameters such as mask ratio, window size, temporal scales, and the undisclosed gamma.

free parameters (6)
  • mask ratio r = 0.5
    Chosen by hand; ablation on UCF and HMDB EP3 shows 50% works best among 25%, 50%, and 75%.
  • spatial window size w1 x w2 = 2 x 2
    Set as the default window for WSM and MCM; the paper does not report a sweep over other window sizes.
  • temporal scales = [±1, ±2]
    Selected via ablation; the combination [±1, ±2] gives the best UCF-HMDB accuracy trade-off.
  • gamma in d_delta = gamma * D = not stated
    Eq. (5) defines d_delta = gamma * D as a hyperparameter, but the paper never specifies gamma.
  • feature distillation loss weight = not stated
    The training objective uses a distillation loss from Huang et al. 2024, but the loss formula and weight are not reported.
  • top K objects per action = 5 to 10
    The LLM prompt requests 5 <= K <= 10 relevant objects, so the number of generated prompts per class varies.
assumptions (5)
  • domain assumption CLIP's frozen text encoder and vision-language alignment transfer from images to video frames.
    The entire framework assumes CLIP zero-shot knowledge generalizes to video inputs and that frame-level features can be aligned with text prompts.
  • domain assumption Fine-tuning the vision encoder on Kinetics-400 with cross-entropy and distillation preserves zero-shot generalization to unseen classes.
    Section 4 uses K400-trained models directly on UCF, HMDB, and K600; if training destroys CLIP alignment, the approach would not work.
  • domain assumption GPT-3.5 generated action descriptions are semantically accurate across seen and unseen classes.
    The ASKG and text prompts come from GPT-3.5; Appendix E acknowledges hallucination and says prompt quality varies by category, which affects final performance.
  • standard math Standard transformer machinery (LayerNorm, residual connections, MLP) remains valid under masked and mixed tokens.
    Equations (1), (2), and (7) rely on standard ViT block structure with insertions of masking, channel mixing, and padding.
  • standard math The complexity expression O(TN^2 + ST(N')^2) = O(TN^2) treats N' = rN and S, r as constants.
    This is the basis for the 'no increased complexity' claim, but the exact FLOP count includes a second MHSA pass on masked tokens.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP." pith.science (2026). https://pith.science/paper/MKUILP33

@misc{pith2026241209895,
  author       = {Pith},
  title        = {Pith review of: Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MKUILP33}},
  note         = {Machine review of arXiv:2412.09895}
}
read the original abstract

Zero-shot action recognition (ZSAR) requires collaborative multi-modal spatiotemporal understanding. However, finetuning CLIP directly for ZSAR yields suboptimal performance, given its inherent constraints in capturing essential temporal dynamics from both vision and text perspectives, especially when encountering novel actions with fine-grained spatiotemporal discrepancies. In this work, we propose Spatiotemporal Dynamic Duo (STDD), a novel CLIP-based framework to comprehend multi-modal spatiotemporal dynamics synergistically. For the vision side, we propose an efficient Space-time Cross Attention, which captures spatiotemporal dynamics flexibly with simple yet effective operations applied before and after spatial attention, without adding additional parameters or increasing computational complexity. For the semantic side, we conduct spatiotemporal text augmentation by comprehensively constructing an Action Semantic Knowledge Graph (ASKG) to derive nuanced text prompts. The ASKG elaborates on static and dynamic concepts and their interrelations, based on the idea of decomposing actions into spatial appearances and temporal motions. During the training phase, the frame-level video representations are meticulously aligned with prompt-level nuanced text representations, which are concurrently regulated by the video representations from the frozen CLIP to enhance generalizability. Extensive experiments validate the effectiveness of our approach, which consistently surpasses state-of-the-art approaches on popular video benchmarks (i.e., Kinetics-600, UCF101, and HMDB51) under challenging ZSAR settings.

Figures

Figures reproduced from arXiv: 2412.09895 by the authors.

Figure 1
Figure 1. Illustration of the challenges without collabora￾tive multi-modal spatiotemporal understanding. (a) A model lacking static context alignment may misidentify the novel class due to the ambiguity associated with the barbell. (b) It might also struggle generalizing to other novel weightlifting actions, due to the subtle dynamic differences and strong vi￾sual similarities. would inevitably lead to an ambiguous comprehen… view at source ↗
Figure 2
Figure 2. Overview of our framework. With a four-step operation applied within each block, we transform the spa￾tial attention into novel Space-time Cross Attention. Spa￾tiotemporal text augmentation is conducted to derive spatial and temporal text prompts, where multi-modal dynamics are meticulously aligned in a fine-grained manner. video of {archery}”, thus faltering when encountering novel actions with fine-grained spatiot… view at source ↗
Figure 3
Figure 3. Illustration of our method. (1) We extend the spatial attention block to perform Space-time Cross Attention by applying Window Shift Masking to the input spatial tokens, and perform Multi-Scale Channel Mixing to capture temporal dynamics before MHSA. Then, we employ the spatial padding strategy to fill in the masked positions for seamless short￾cutting, fusing additional dynamics effortlessly. (2) We conduct spatiot… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Effect of different combinations of text augmenta [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Visualizations of the attention maps and frame [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Prompts for Action Semantic Knowledge Graph (ASKG) construction. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Prompts for Spatiotemporal Text Augmentation. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Effect of different combinations of text augmenta [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Task-Adapter++: Task-specific Adaptation with Order-aware Alignment for Few-shot Action Recognition

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Task-Adapter++ adapts frozen CLIP encoders with task-specific visual adapters and order-aware semantic adapters, and reports state-of-the-art results on five few-shot action recognition benchmarks.

Reference graph

Works this paper leans on

81 extracted references · 74 canonical work pages · cited by 1 Pith paper

  1. [1]

    bow: This is a video of archery, which requires a bow

    work page

  2. [2]

    arrow: This is a video of archery, which uses an arrow

    work page

  3. [3]

    Advances in neural information processing systems, 33: 1877–1901

    Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901. Bulat, A.; Perez Rua, J. M.; Sudhakaran, S.; Martinez, B.; and Tz- imiropoulos, G. 2021. Space-time mixing attention for video trans- former. Advances in neural information processing systems , 34: 19594–19607. Cao, C.; Zhang, Y .; Yu, Y .; Lv, Q.; Min...

    arXiv 1901

  4. [4]

    quiver: This is a video of archery, which utilizes a quiver

    work page

  5. [5]

    armguard: This is a video of archery, which requires an arm- guard

    work page

  6. [6]

    finger tab : This is a video of archery, which needs a finger tab

    work page

  7. [7]

    In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6545–6554

    Fine-tuned clip models are efficient video learners. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6545–6554. Sanghi, A.; Chu, H.; Lambourne, J. G.; Wang, Y .; Cheng, C.- Y .; Fumero, M.; and Malekshan, K. R. 2022. Clip-forge: To- wards zero-shot text-to-shape generation. In Proceedings of the IEEE/CVF Conference ...

    arXiv 2022

  8. [8]

    In Proceedings of the IEEE/CVF International Conference on Computer Vision, 19936–19947

    Implicit temporal modeling with learnable alignment for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 19936–19947. Wang, H.; Liu, F.; Jiao, L.; Wang, J.; Hao, Z.; Li, S.; Li, L.; Chen, P.; and Liu, X. 2024. ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Op- tim...

    arXiv 2024

Show all 81 references
  1. [9]

    commonsense knowledge base for human actions

    Bidirectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, 6620–6630. Wu, Z.; Weng, Z.; Peng, W.; Yang, X.; Li, A.; Davis, L. S.; and Jia...

    2024 arXiv

  2. [12]

    target: This is a video of archery, which aims at a target

  3. [16]

    Sub-action list:

    bullseye: This is a video of archery, which involves a bulls- eye. Sub-action list:

  4. [17]

    drawing the bowstring

  5. [18]

    aiming at the target

  6. [19]

    following through Object relation triples for spatial text prompts [S]:

  7. [20]

    <bow, used to shoot, arrow> : This is a video of archery, where a bow is used to shoot an arrow

  8. [21]

    <arrow, aimed at, target> : This is a video of archery, where an arrow is aimed at a target

  9. [22]

    <quiver, holds, arrows> : This is a video of archery, where arrows are held in a quiver

  10. [23]

    <armguard, protects, arm> : This is a video of archery, where an armguard protects the arm

  11. [24]

    <finger tab, protects, fingers> : This is a video of archery, where a finger tab protects the fingers

  12. [25]

    Sub-action relation triples for temporal text prompts [T]:

    <target, has, bullseye> : This is a video of archery, where a target has a bullseye. Sub-action relation triples for temporal text prompts [T]:

  13. [26]

    <archery, starts with, gripping the bow> : This is a video of archery, starting with gripping the bow

  14. [27]

    <gripping the bow, precedes, nocking the arrow>: This is a video of archery, where gripping the bow precedes nocking the arrow

  15. [28]

    <nocking the arrow, precedes, drawing the bowstring>: This is a video of archery, where nocking the arrow comes before drawing the bowstring

  16. [29]

    <drawing the bowstring, precedes, aiming at the target> : This is a video of archery, where drawing the bowstring precedes aiming at the target

  17. [30]

    <aiming at the target, precedes, releasing the arrow> : This is a video of archery, where aiming at the target comes before releasing the arrow

  18. [31]

    Responses for action category “Surfing”: Object list for spatial text prompts [S]:

    <releasing the arrow, precedes, following through>: This is a video of archery, where releasing the arrow precedes following through. Responses for action category “Surfing”: Object list for spatial text prompts [S]:

  19. [32]

    surfboard: This is a video of surfing, which requires a surf- board

  20. [33]

    waves: This is a video of surfing, which takes place on waves

  21. [34]

    ocean: This is a video of surfing, which takes place in the ocean

  22. [35]

    beach: This is a video of surfing, which takes place near a beach

  23. [36]

    wetsuit: This is a video of surfing, which requires a wetsuit

  24. [37]

    leash: This is a video of surfing, which uses a leash

  25. [38]

    Sub-action list:

    fins: This is a video of surfing, which uses fins. Sub-action list:

  26. [39]

    given action name

    paddling out You are a commonsense knowledge base, especially for human actions. You will be provided with an action entity name below, which is delimited with square brackets. Use the following step-by-step instructions to respond to user inputs: Conditions 1 -Return the obje...

  27. [40]

    standing up on the surfboard

  28. [41]

    exiting the wave Object relation triples for spatial text prompts [S]:

  29. [42]

    <surfboard, used on, waves> : This is a video of surfing, using a surfboard on waves

  30. [43]

    <waves, found in, ocean> : This is a video of surfing, finding waves in the ocean

  31. [44]

    <ocean, located near, beach> : This is a video of surfing, the ocean being located near a beach

  32. [45]

    <wetsuit, worn during, surfing> : This is a video of surfing, wearing a wetsuit during surfing

  33. [46]

    <leash, attached to, surfboard> : This is a video of surfing, attaching a leash to the surfboard

  34. [47]

    Sub-action relation triples for temporal text prompts [T]:

    <fins, attached to, surfboard> : This is a video of surfing, attaching fins to the surfboard. Sub-action relation triples for temporal text prompts [T]:

  35. [48]

    <surfing, starts with, paddling out> : This is a video of surfing, starting with paddling out

  36. [49]

    <paddling out, precedes, catching a wave> : This is a video of surfing, where paddling out precedes catching a wave

  37. [50]

    <catching a wave, precedes, standing up on the surfboard> : This is a video of surfing, attaching fins to the surfboard

  38. [51]

    <standing up on the surfboard, precedes, riding the wave> : This is a video of surfing, where standing up on the surfboard precedes riding the wave

  39. [52]

    <riding the wave, precedes, performing tricks>: This is a video of surfing, where riding the wave comes before performing tricks

  40. [53]

    Clean and jerk

    <performing tricks, precedes, exiting the wave>: This is a video of surfing, where performing tricks precedes exiting the wave. Responses for action category “Clean and jerk ”: Object list for spatial text prompts [S]:

  41. [54]

    barbell: This is a video of clean and jerk, which involves a barbell

  42. [55]

    platform: This is a video of clean and jerk, which requires a platform

  43. [56]

    chalk: This is a video of clean and jerk, which uses chalk

  44. [57]

    weightlifting belt : This is a video of clean and jerk, which requires a weightlifting belt

  45. [58]

    weightlifting shoes : This is a video of clean and jerk, which requires weightlifting shoes

  46. [59]

    grip: This is a video of clean and jerk, which involves grip

  47. [60]

    shoulders: This is a video of clean and jerk, which utilizes the shoulders

  48. [61]

    legs: This is a video of clean and jerk, which utilizes the legs

  49. [62]

    Sub-action list:

    arms: This is a video of clean and jerk, which utilizes the arms. Sub-action list:

  50. [63]

    setting up the barbell

  51. [64]

    gripping the barbell

  52. [65]

    cleaning the barbell

  53. [66]

    jerking the barbell Object relation triples for spatial text prompts [S]:

  54. [67]

    <barbell, loaded with, weights> : This is a video of clean and jerk, where a barbell is loaded with weights

  55. [68]

    <barbell, placed on, platform> : This is a video of clean and jerk, where a barbell is placed on a platform

  56. [69]

    <chalk, used for, grip> : This is a video of clean and jerk, where chalk is used for grip

  57. [70]

    <weightlifting belt, worn on, waist> : This is a video of clean and jerk, where a weightlifting belt is worn on the waist

  58. [71]

    <weightlifting shoes, worn on, feet> : This is a video of clean and jerk, where weightlifting shoes are worn on the feet

  59. [72]

    <grip, on, barbell> : This is a video of clean and jerk, where there is a grip on the barbell

  60. [73]

    <shoulders, support, barbell> : This is a video of clean and jerk, where the shoulders support the barbell

  61. [74]

    <legs, provide power for, lift> : This is a video of clean and jerk, where the legs provide power for the lift

  62. [75]

    Sub-action relation triples for temporal text prompts [T]:

    <arms, lift, barbell> : This is a video of clean and jerk, where the arms lift the barbell. Sub-action relation triples for temporal text prompts [T]:

  63. [76]

    <clean and jerk, starts with, setting up the barbell> : This is a video of clean and jerk, starting with setting up the barbell

  64. [77]

    <setting up the barbell, precedes, gripping the barbell> : This is a video of clean and jerk, where setting up the barbell precedes gripping the barbell

  65. [78]

    <gripping the barbell, precedes, cleaning the barbell> : This is a video of clean and jerk, where gripping the barbell precedes cleaning the barbell

  66. [79]

    B Details of Datasets and Evaluation Protocols Datasets We conduct the training process on Kinetics-400 (Kay et al

    <cleaning the barbell, precedes, jerking the barbell> : This is a video of clean and jerk, where cleaning the barbell precedes jerking the barbell. B Details of Datasets and Evaluation Protocols Datasets We conduct the training process on Kinetics-400 (Kay et al

  67. [80]

    jump”, “kiss

    dataset and perform evaluations on other three popu- lar benchmarks: UCF101 (Soomro, Zamir, and Shah 2012), HMDB51 (Kuehne et al. 2011), and Kinetics-600 (Carreira et al. 2018). Kinectics-400 and Kinectics-600 are both comprehensive video datasets for human action recognition....

    2012

  68. [81]

    This can be explained by the fact that larger temporal scales result in sparser interactions for boundary frames during channel mixing

    It can be observed that the performance consistently decreases when expanding the single temporal scale from [±1] to [±3]. This can be explained by the fact that larger temporal scales result in sparser interactions for boundary frames during channel mixing. Better outcomes ar...

  69. [2011]

    In 2011 International conference on computer vision, 2556–

    HMDB: a large video database for human motion recogni- tion. In 2011 International conference on computer vision, 2556–

    2011

  70. [2017]

    arXiv preprint arXiv:1705.06950

    The kinetics human action video dataset. arXiv preprint arXiv:1705.06950. Kim, T. S.; Jones, J.; Peven, M.; Xiao, Z.; Bai, J.; Zhang, Y .; Qiu, W.; Yuille, A.; and Hager, G. D. 2021. Daszl: Dynamic action signatures for zero-shot learning. In Proceedings of the AAAI con- feren...

    2021 arXiv

  71. [2020]

    InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 4613–4623

    Rethinking zero-shot video classification: End-to-end train- ing for realistic applications. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 4613–4623. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhari- wal, P.; Neelakanta...

  72. [2023]

    arXiv preprint arXiv:2303.08774

    Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Ahmad, S.; Chanda, S.; and Rawat, Y . S. 2023. EZ-CLIP: Efficient Zeroshot Video Action Recognition. arXiv preprint arXiv:2312.08010. Arnab, A.; Dehghani, M.; Heigold, G.; Sun, C.; Lu ˇci´c, M.; and Schmid, C. 2021. Vivi...

    2023 arXiv

  73. [2563]

    Lee, D.; Lee, J.; and Choi, J

    IEEE. Lee, D.; Lee, J.; and Choi, J. 2024. CAST: Cross-Attention in Space and Time for Video Action Recognition.Advances in Neural Information Processing Systems, 36. Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. Blip-2: Bootstrapping language-image pre-training with frozen ...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.