Pith. sign in

REVIEW 3 major objections 6 minor 36 references

Kronecker Mask and Interpretive Prompts are Language-Action Video Learners

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CLAVER claims that adapting both CLIP branches — Kronecker mask temporal attention on the vision side and LLM-generated interpretive prompts on the text side — shifts CLIP from object-noun alignment to action-verb alignment, and reports…

desk verdict Solid incremental CLIP-to-video adaptation with a clean attention mask and real gains; the homogenization diagnostic is over-claimed but doesn't sink the paper. read the letter →

arxiv 2502.03549 v3 pith:AV3NOVAC submitted 2025-02-05 cs.CV

Jingyi Yang , Zitong Yu , Xiuming Ni , Jia He , Hui Li This is my paper · ORCID
classification cs.CV
keywords CLIPvideoactionrecognitionKroneckermaskattentiontemporalmodelinginterpretivepromptszero-shotfew-shotspatiotemporalhomogenization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that adapting CLIP to video requires changing both branches at once: the visual branch must model time, and the text branch must describe actions, not objects. To do this it introduces a Kronecker mask temporal attention that lets every patch interact with patches in other frames while blocking same-frame interactions, plus LLM-generated 'interpretive prompts' that decompose actions, list synonyms, and name body parts. The claim is that together these shift CLIP from aligning static objects with concrete nouns to aligning dynamic behaviors with abstract verbs. If true, a frozen-image CLIP plus a small temporal transformer and cheap text generation can match or exceed larger video-language models.

What carries the argument

The central object is the Kronecker mask, an attention mask built as a Kronecker product of an identity matrix over frames and a ones-minus-identity matrix over patches: $M = [I_{(T,T)} \otimes (J_{(L,L)} - I_{(L,L)})]$ with 1 replaced by $-\infty$, applied inside softmax attention. This blocks intra-frame attention while allowing every token to attend to all tokens in other frames, widening the temporal receptive field relative to pipeline or class-token-only temporal attention; the causal variant adds $(U_{(T,T)} - I_{(T,T)}) \otimes J_{(L,L)}$ to enforce time causality and guarantee a full-rank attention matrix. The second mechanism is the interpretive prompt pipeline, which uses ChatGPT to draft examples and LLaMA-3 to generate diverse action decompositions, synonym conversions, and body-part descriptions for every action class.

What would settle it

Run the paper's token-shuffling test on a model that processes frames in a deliberately order-invariant way, such as average-pooled frame features, and on a strong 3D video model; if the shuffling metric does not separate order-sensitive from order-insensitive models, the homogenization claim fails. Alternatively, replace the LLM interpretive prompts with shuffled or nonsensical long texts of the same length; if zero-shot accuracy does not drop, the verb-focus claim is unsupported.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that CLAVER — a Contrastive Language-Action Video Learner — achieves the noun-to-verb shift by jointly adapting the visual and textual branches of CLIP, and that this yields state-of-the-art or competitive results across fully supervised, few-shot and zero-shot settings. Specifically, the Kronecker mask temporal attention (KMTA) and its causal variant (KMCTA) expand each token's temporal receptive field to all other frames while serving as a spatiotemporal heterogeneity inductive bias that fights the 'spatiotemporal homogenization' problem. The interpretive prompts provide sentence-, phrase- and word-level descriptions that move text-side focus toward verbs. The paper backs the shift with word-importance visualizations and token-shuffling sensitivity experiments, and reports improvements such as 86.0% top-1 on Kinetics-400 with ViT-B/16.

Load-bearing premise

The claim that Kronecker mask attention mitigates 'spatiotemporal homogenization' rests on a self-defined metric: sensitivity to random token shuffling. If that metric is not a meaningful proxy for genuine temporal understanding, the interpretability argument collapses and the method stands only on accuracy numbers.

Editorial extensions

If this is right

  • CLIP-based video action recognition can be improved by adapting both branches rather than one, with gains of several accuracy points over single-branch baselines.
  • The Kronecker mask attention is plug-and-play for any transformer-based video model, so it can be reused outside CLIP.
  • LLM-generated descriptive text at training time improves zero-shot and few-shot transfer, suggesting text diversity replaces manual prompt engineering.
  • The causal variant's full-rank guarantee points to a practical recipe for long-frame video where attention matrices risk a low-rank bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's spatiotemporal homogenization metric (sensitivity to random token shuffling) could be tested as a general diagnostic: if it tracks accuracy on genuinely temporal tasks, it gives model designers a cheap probe for temporal structure without labels.
  • The interpretive-prompt recipe suggests a scalable way to build verb-centric text for any action taxonomy, since the prompts are generated by LLMs without manual labor, the approach extends to new datasets or long-tail actions automatically.
  • One could test whether the noun-to-verb shift is causal by ablating the mask and prompts separately on a verb-heavy benchmark; the paper's ablations suggest prompts alone add about 1% while the mask adds about 3.5%, so the mask likely carries more of the effect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CLAVER, a CLIP-based video action recognition method with two complementary components: (i) Kronecker mask temporal attention (KMTA) and its causal variant (KMCTA), which restrict joint attention over video tokens so that each token attends across frames but not within its own frame (or, for KMCTA, only to past frames), and (ii) LLM-generated interpretive prompts that decompose actions, provide synonyms, and describe involved body parts. Experiments report strong or competitive results on Kinetics-400 and Kinetics-600 in fully supervised settings, and on HMDB-51 and UCF-101 in zero-shot and few-shot settings. The paper further claims that the Kronecker mask mitigates 'spatiotemporal homogenization' and that KMCTA avoids a low-rank attention bottleneck.

Significance. If the mechanistic claims were fully supported, this would be a useful contribution: it demonstrates the value of adapting both the visual and textual branches of CLIP for action recognition, introduces a simple structured-mask family for temporal attention, and provides a reproducible recipe for LLM-generated text augmentation. Strengths include public code, an appendix proof of the KMCTA full-rank property, extensive ablations over components and temporal attention variants, and evaluation across fully supervised, zero-shot, and few-shot protocols. The accuracy gains over CLIP-based baselines are consistent, though modest in several configurations. The main weakness is that the interpretability and low-rank narratives are supported by diagnostics that are either confounded or too weak to carry the claims.

major comments (3)
  1. [Sec. 5, Fig. 6, Table 9, Eqs. (6) and (8)] The spatiotemporal homogenization diagnostic does not establish that KMTA/KMCTA encode temporal structure. For joint attention over a flattened token sequence, the attention operation is permutation-equivariant up to positional encodings, so after mean pooling the class tokens, invariance to token shuffling is largely expected rather than evidence of a failure. For KMTA and KMCTA, the masks in Eqs. (6) and (8) are fixed functions of the token ordering, so randomly shuffling tokens changes which pairs are masked; sensitivity follows even if the mask carries no temporal semantics. A random fixed mask with the same sparsity pattern would likely show the same shuffling sensitivity. The authors should add a control with such a mask and, more directly, a temporal-order probe (e.g., reversed or permuted frame order without token shuffling) before attributing the sensitivity to genuine temporal understanding. As written, the interpretability claim in Sec. 5 is confounded; the accuracy results in Tables 1-5 remain the primary support for the method.
  2. [Sec. 4.3, Table 8, Appendix A] The claimed low-rank advantage of KMCTA over KMTA is not established by the presented evidence. In Table 8, most differences among joint attention, KMTA, and KMCTA are within 0.1-0.5 percentage points, and no significance or variance information is reported. When the token count is increased by reducing patch size, the authors explicitly observe no low-rank bottleneck and attribute the gains to fine-grained features, which undercuts the frame-length argument. When frame length is increased, KMTA also improves in several rows (e.g., UCF-101 with patch size 16: 96.3 to 96.6), so the statement that 'only the performance of KMCTA can steadily improve' is not uniformly supported by the table. The full-rank proof in Appendix A.2 is mathematically reasonable under Assumption 1, but its practical relevance to the small accuracy differences is not demonstrated. The authors should either temper the low-rank bottleneck claim or provide direct evidence, such as measured rank or attention-output homogenization under controlled frame-length increases.
  3. [Sec. 5, Fig. 5, Sec. 3.3] The central claim that CLAVER shifts CLIP's focus from nouns to verbs is supported only by qualitative word-importance visualizations on a few examples. No aggregate statistics, no part-of-speech-tagged probe over the test set, and no control experiment replacing interpretive prompts with matched-length generic sentences are provided. The accuracy gains are consistent with the method, but the mechanistic 'shift from concrete nouns to abstract verbs' claim is not quantitatively validated. This matters because the interpretive prompt component in Table 10 could in principle improve performance simply by providing longer or more diverse text, without changing the model's noun-versus-verb bias in the way described.
minor comments (6)
  1. [Sec. 4.1] The dataset name is misspelled as 'Kientics-600'; it should be 'Kinetics-600'.
  2. [Sec. 3.2 and throughout] The notation is inconsistent: the text and tables sometimes refer to 'KMCT transformer' while the equations define 'KMCTA'. Please standardize the abbreviation.
  3. [Eq. (9)] The notation 'sub{M}' is undefined, and the text does not state how many text descriptions are sampled per training step or how the temperature parameter tau is set during inference.
  4. [Appendix D, Table 14] The zero-shot training epoch entry '0 (20 on K400)' is confusing; clarify whether the model is trained on Kinetics-400 for 20 epochs or not trained at all.
  5. [Table 1] The reference 'Li et al.' for MViTv2-L is incomplete; it lacks a year and venue.
  6. [Appendix A.3] The phrase 'When we pray for a longer frame length' appears to be a typo; it should likely be 'pay' or 'strive for'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the Kronecker mask is an explicit architectural construction, the interpretive prompts come from external LLMs, and the low-rank claim is proven from an external theorem; the self-defined homogenization diagnostic is an external-validity concern, not a circular reduction.

full rationale

The paper's derivation chain is self-contained with respect to its own claims. The Kronecker mask temporal attention and the causal variant are explicit algebraic constructions (Eqs. 6 and 8) whose forms are not fitted to the reported benchmarks, and the claimed benefit of avoiding the low-rank bottleneck is argued in Appendix A from the external representation theorem of Bhojanapalli et al. plus a determinant argument for triangular attention matrices, rather than from the paper's own performance numbers. The interpretive prompts are generated by external LLMs (ChatGPT and LLaMA-3) and are not derived from the model's predictions. Accuracy claims are evaluated on external benchmarks (Kinetics-400, Kinetics-600, HMDB-51, UCF-101) and compared against external baselines, so no reported result reduces by construction to a fitted input. The only self-citations (Yu et al. 2021, Xie et al. 2024) appear in related-work enumerations and are not load-bearing for any central premise. Section 5's spatiotemporal-homogenization diagnostic is indeed self-defined and arguably confounded, since any fixed non-permutation-invariant mask will be shuffling-sensitive by construction; however, this is a validity or interpretability concern rather than circularity, because the metric is not used to fit a parameter that is later renamed as a prediction. Under the enumerated circularity patterns, no specific reduction can be exhibited, so the appropriate finding is no significant circularity.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

No new physical or ontological entities are introduced. The Kronecker mask is a mathematical attention pattern, not a postulated entity. The central claim rests on hyperparameters (depth, learning rates, LLM sampling), standard math theorems, and domain assumptions about CLIP features, sparse sampling, and the shuffling-sensitivity metric.

free parameters (7)
  • KMT/KMCT depth fraction = 1/3 of image encoder layers (4 for ViT-B, 8 for ViT-L)
    Selected by ablation in Table 6; 1/3 scale gives the best Top-1 among tested depths.
  • Learning rate multiplier for KMT = 10x the image/text encoder LR
    Stated in Appendix D; heuristic to train the randomly initialized temporal transformer faster.
  • LLM sampling temperature (tau) = 0.90
    Appendix E: chosen to produce diverse interpretive prompts while keeping the central concept.
  • LLM nucleus sampling p = 0.95
    Appendix E: diversity control for LLaMA-3 prompt generation.
  • Interpretive prompt word limit = 76 words
    Prompt constraint used in ChatGPT/LLaMA-3 generation; appears in Figure 4 and Appendix E.
  • Number of ChatGPT demonstration examples = 3
    Appendix E: small set of examples used to seed LLaMA-3 generation.
  • Text descriptions per class (M) = not fully specified; subset sampled per step
    Eq. (9) samples a subset of all text descriptions; the exact M is not reported.
assumptions (6)
  • standard math Representation Theorem of Bhojanapalli et al. (Theorem 1 in Appendix A.1)
    Used to argue joint attention cannot guarantee full rank when d < n.
  • standard math Softmax outputs are strictly positive except where masked by -inf (Assumption 1, Appendix A.2)
    Needed to ensure the KMCTA attention matrix is lower triangular with positive diagonal, hence full rank.
  • domain assumption CLIP image and text encoders provide strong per-frame and per-word features
    The whole design inherits CLIP's pretrained representations; invoked in Section 3.1 and Appendix D.
  • domain assumption Sparse sampling of 8 to 32 frames is sufficient for action recognition
    All experiments use sparse sampling; the method does not model long-range video beyond the sampled frames.
  • ad hoc to paper Sensitivity to token shuffling is a valid measure of spatiotemporal understanding
    The paper defines spatiotemporal homogenization via shuffling sensitivity in Section 5 and Appendix B, without external validation.
  • ad hoc to paper Action decomposition, synonym conversion, and body-part descriptions are the right semantic axes for verb understanding
    Section 3.3 assumes these three prompt types improve alignment; Table 10 ablates them jointly, not individually.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Kronecker Mask and Interpretive Prompts are Language-Action Video Learners." pith.science (2026). https://pith.science/paper/AV3NOVAC

@misc{pith2026250203549,
  author       = {Pith},
  title        = {Pith review of: Kronecker Mask and Interpretive Prompts are Language-Action Video Learners},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AV3NOVAC}},
  note         = {Machine review of arXiv:2502.03549}
}
read the original abstract

Contrastive language-image pretraining (CLIP) has significantly advanced image-based vision learning. A pressing topic subsequently arises: how can we effectively adapt CLIP to the video domain? Recent studies have focused on adjusting either the textual or visual branch of CLIP for action recognition. However, we argue that adaptations of both branches are crucial. In this paper, we propose \textbf{CLAVER}: a \textbf{C}ontrastive \textbf{L}anguage-\textbf{A}ction \textbf{V}ideo Learn\textbf{er}, designed to shift CLIP's focus from the alignment of static visual objects and concrete nouns to the alignment of dynamic action behaviors and abstract verbs. Specifically, we introduce a novel Kronecker mask attention for temporal modeling. Our tailored Kronecker mask offers three benefits 1) it expands the temporal receptive field for each token, 2) it serves as an effective spatiotemporal heterogeneity inductive bias, mitigating the issue of spatiotemporal homogenization, and 3) it can be seamlessly plugged into transformer-based models. Regarding the textual branch, we leverage large language models to generate diverse, sentence-level and semantically rich interpretive prompts of actions, which shift the model's focus towards the verb comprehension. Extensive experiments on various benchmarks and learning scenarios demonstrate the superiority and generality of our approach.

Figures

Figures reproduced from arXiv: 2502.03549 by the authors.

Figure 1
Figure 1. An overview of CLAVER. (Right) Image encoder and KMT transformer are assembled as a video encoder. (Left) How to get the interpretive prompts for actions. In addressing the issue 1), several studies Wang et al. (2021); Ju et al. (2022); Chen et al. (2023); Rasheed et al. (2023) implement straightforward and simple strategies such as mean pooling or 1D-temporal convolution across the temporal dimension, or employing … view at source ↗
Figure 2
Figure 2. (Left) Red indicates the currently focal patch, green patches are visible in spatial attention, orange patches are visible in temporal attention, purple patches are visible in joint attention. (Right) Kronecker mask attention: Several attentions can be seen as employing tailored Kronecker masks for joint attention. aims to align text representations with the video representations. Concretely, given a video clip V = … view at source ↗
Figure 3
Figure 3. Kronecker mask causal temporal attention. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: The Interpretive Prompt scheme. To address the issue 2) in Sec. 1, we prompt LLMs to generate interpretive texts that align the text semantics of video-text pairs with the rich text semantics in VLM’s pre-training dataset. This design aids the model in understanding ab…
Figure 5
Figure 5. Figure 5: Word importance of CLIP, X-CLIP, ILA and CLAVER. Darker color, higher importance. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Spatiotemporal Homogenization. (Upper Left) Token shuffling. (Vertical bar chart) Word importance refers to the degree of correlation between each word in a sentence description and the semantics of the video content, while (Horizontal bar chart) Similarity refers to t…
Figure 7
Figure 7. Figure 7: Visualizing spatiotemporal attention maps of different spatio-temporal modeling. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: We visualize the impact of token shuffling on the ILA, XCLIP. In most cases, token [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Testing on synthetic videos from Imagen ( [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: Attention map of synthetic videos from Imagen. [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: Attention map of videos from Kinetics400. [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: Attention map of synthetic videos from Sora. [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: More visualization examples of transition of CLIP’s attention on nouns to CLAVER’s [PITH_FULL_IMAGE:figures/full_fig_p029_13.png]
Figure 14
Figure 14. Figure 14: The rank of Kronecker temporal attention (KMT), Kronecker causal temporal attention [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 18 canonical work pages

  1. [1]

    The low-rank bottleneck is vital, because it may lead to that many rows of the attention map are seriously homogenized

    Because increasing the frame length will further increase n, the expressive power of self attention may encounter bottlenecks. The low-rank bottleneck is vital, because it may lead to that many rows of the attention map are seriously homogenized. As the output of self-attention is the weighted sum of the same set of value vectors, the homogenization of at...

    work page 2017

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    URL https://openai.com/research/ video-generation-models-as-world-simulators . Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    work page 1901

  3. [4]

    24 Published as a conference paper at ICLR 2025 5

    Jumping: Legs bending at the knees, feet pushing off the ground, arms swinging for momentum, body soaring through the air momentarily before landing. 24 Published as a conference paper at ICLR 2025 5

    work page 2025

  4. [5]

    Question to ChatGPT: 1 Cutting in the kitchen

    Dancing: Feet tapping, legs extending and bending, hips swaying, arms gesturing and flowing in sync with music, body moving rhythmically with expression and grace. Question to ChatGPT: 1 Cutting in the kitchen. Answered by ChatGPT: 1 Using a sharp knife, fingers gripping the handle, hand guiding the blade 2 through ingredients on a cutting board, wrist co...

    work page 2024

  5. [8]

    Open-vocabulary object detection via vision and language knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921,

    arXiv

  6. [9]

    Ssan: Separable self-attention network for video representation learning

    12 Published as a conference paper at ICLR 2025 Xudong Guo, Xun Guo, and Yan Lu. Ssan: Separable self-attention network for video representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12618–12627,

    work page 2025

  7. [10]

    Probing image-language transformers for verb under- standing

    Lisa Anne Hendricks and Aida Nematzadeh. Probing image-language transformers for verb under- standing. arXiv preprint arXiv:2106.09141,

    arXiv

  8. [13]

    Frozen clip models are efficient video learners

    13 Published as a conference paper at ICLR 2025 Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In European Con- ference on Computer Vision, pp. 388–404. Springer,

    work page 2025

Show all 36 references
  1. [15]

    Fine-tuned clip models are efficient video learners

    14 Published as a conference paper at ICLR 2025 Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2025

  2. [16]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402,

    arXiv

  3. [18]

    Learning video representations using contrastive bidirectional transformer

    Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using contrastive bidirectional transformer. arXiv preprint arXiv:1906.05743, 2019a. Chen Sun, Austin Myers, Carl V ondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model ...

    1906 arXiv

  4. [19]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. Hugo Touvron...

    arXiv

  5. [20]

    Clipasso: Semantically-aware object sketching

    15 Published as a conference paper at ICLR 2025 Yael Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically-aware object sketching. ACM Transactions on Graphics (TOG), 41(4):1–11,

    2025

  6. [22]

    Alternative semantic representations for zero-shot human action recog- nition

    Qian Wang and Ke Chen. Alternative semantic representations for zero-shot human action recog- nition. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18–22, 2017, Proceedings, Part I 10 , pp. 87–102. Springer,

    2017

  7. [23]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084,

    arXiv

  8. [24]

    Multi-task zero-shot action recognition with prioritised data augmentation

    Xun Xu, Timothy M Hospedales, and Shaogang Gong. Multi-task zero-shot action recognition with prioritised data augmentation. In Computer Vision–ECCV 2016: 14th European Conference, Am- sterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 343–359. Springer,

    2016

  9. [25]

    Aim: Adapting image models for efficient video action recognition

    Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. arXiv preprint arXiv:2302.03024,

    arXiv

  10. [26]

    Florence: A new foundation model for computer vision

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432,

    arXiv

  11. [27]

    Co-training transformer with videos and images improves action recognition

    Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M Dai, Ruoming Pang, and Fei Sha. Co-training transformer with videos and images improves action recognition. arXiv preprint arXiv:2112.07175,

    arXiv

  12. [28]

    Learning a deep embedding model for zero-shot learning

    16 Published as a conference paper at ICLR 2025 Li Zhang, Tao Xiang, and Shaogang Gong. Learning a deep embedding model for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 2021– 2030,

    2025

  13. [30]

    always be full rank

    Thus, the attention matrix ˜A of KMCTA is always reversible, i.e. always be full rank. 19 Published as a conference paper at ICLR 2025 A.3 P ROOF OF KMTA CANNOT GUARANTEE FULL RANK According to the Assumption

    2025

  14. [32]

    In the zero-shot scenario, the performance of KMCTA and KMTA has its own wins and losses, which is due to the lack of further fine-tuning, and the model’s ability is relatively dependent on previous training. In the few-shot scenario, we can observe that in most cases, the per...

    2025

  15. [36]

    For cases generated by Imagen, in Fig

    The results shows the robustness and generalization of CLA VER. For cases generated by Imagen, in Fig. 9 Upper, we denote the Kinetics-400 action text set as A. The action category ”lift a cup” in Fig. 9 (a) does not appear in A, while, (b) ”washing dishes”, (c) 25 Published a...

    2017

  16. [600]

    The evaluation is conducted three times

    Following, the 220 new categories outside Kinetics-400 in Kinetics-600 are used for evaluation. The evaluation is conducted three times. For each iteration, we randomly sampled 160 categories for evaluation from the 220 categories in Kinetics-600. E I NTERPRETIVE PROMPT DETAIL...

    2024

  17. [2009]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  18. [2011]

    Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training

    Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 11336–11344, 2020a. Kunchang Li, Yali Wang, Peng G...

    2021 arXiv

  19. [2012]

    Vl-bert: Pre-training of generic visual-linguistic representations

    Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530,

    1908 arXiv

  20. [2014]

    The kinetics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950,

    arXiv

  21. [2016]

    Actionclip: A new paradigm for video action recognition

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472,

    arXiv

  22. [2017]

    A short note about kinetics-600

    Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01340,

    arXiv

  23. [2018]

    17 Published as a conference paper at ICLR 2025 8 A PPENDIX Roadmap In the Appendix, we present proof in A, Spatiotemporal homogenization study details in B, some additional experiment results in C, experiment setting details in D, interpretive prompt technique details in E, a...

    2020

  24. [2019]

    All about knowledge graphs for actions

    Pallabi Ghosh, Nirat Saini, Larry S Davis, and Abhinav Shrivastava. All about knowledge graphs for actions. arXiv preprint arXiv:2008.12432,

    2008 arXiv

  25. [2021]

    Ost: Refining text knowledge with optimal spatio-temporal descriptor for general video recognition

    11 Published as a conference paper at ICLR 2025 Tongjia Chen, Hongshan Yu, Zhengeng Yang, Zechuan Li, Wei Sun, and Chen Chen. Ost: Refining text knowledge with optimal spatio-temporal descriptor for general video recognition. In Pro- ceedings of the IEEE/CVF Conference on Comp...

    2025

  26. [2022]

    Gpt-4 technical report, https://arxiv.org/abs/2303.08774,

    OpenAI. Gpt-4 technical report, https://arxiv.org/abs/2303.08774,

    arXiv

  27. [2023]

    Imagenet: A large-scale hi- erarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee,

    2009

  28. [2024]

    Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid

    URL https://arxiv.org/abs/2407.21783. Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classification. IEEE transactions on pattern analysis and machine intelligence , 38(7): 1425–1438, 2015a. Zeynep Akata, Scott Reed, Daniel Walt...

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.