Pith. sign in

REVIEW 4 major objections 5 minor 57 references

SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that 3D skeleton knowledge can be distilled into video-language models during training, improving zero-shot recognition of daily activities and dense video captioning while requiring no skeleton data at inference.

desk verdict The skeleton-distillation recipe is new and worth building on, but the headline zero-shot gains are compromised by test-set alpha tuning and the lack of a no-teacher control. read the letter →

arxiv 2502.03459 v1 pith:7GFB4ZPW submitted 2025-02-05 cs.CV

Arkaprava Sinha , Dominick Reilly , Francois Bremond , Pu Wang , Srijan Das This is my paper · ORCID
classification cs.CV
keywords zero-shotactionrecognitionskeleton-languagemodelknowledgedistillationvision-languageactivitiesofdailylivingskeleton-guidedvideorepresentationcaptioningNTURGB+D
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that 3D skeleton data—normally a separate modality that is hard to align with language—can be distilled into ordinary video-language models, so that the resulting models recognize and describe activities of daily living (ADL) better without ever seeing skeletons at inference. The proposed SKI-VLM pipeline first builds SkeletonCLIP, a skeleton-language dual encoder, then uses SkeletonCLIP Distillation (SCD) to push a student video-text model to reproduce the skeleton-text compatibility scores in a language-contextualized space. On the NTU60 and NTU120 benchmarks this lifts zero-shot action recognition of a strong baseline (ViFiCLIP) by up to 5.4 and 7.8 percentage points, surpassing prior zero-shot action recognition models. A companion SKI-LVLM variant adds skeleton tokens to a large vision-language model and improves dense video captioning on Charades. If the claim holds, skeleton knowledge can be baked into deployment-ready RGB-only models, which matters for monitoring applications where skeleton sensors are not available.

What carries the argument

The load-bearing object is SkeletonCLIP, a skeleton-language dual encoder built from a Hyperformer skeleton backbone and a frozen CLIP text encoder, trained with a cross-entropy alignment loss so that 3D joint sequences and action-class text prompts share an embedding space. The second mechanism is SkeletonCLIP Distillation (SCD), an online teacher-student loss that minimizes the mean-squared error between video-text and skeleton-text compatibility scores (the language-contextualized features $F^V_L$ and $F^S_L$) while jointly optimizing both video-text and skeleton-text cross-entropy losses. SCD is what lets a VideoCLIP student (XCLIP, ViFiCLIP, or FROSTER) internalize skeleton-guided attention without requiring skeleton input at inference.

What would settle it

Train the exact SKI-VLM pipeline with the skeleton teacher's class labels randomly shuffled so the teacher carries no real skeleton-action information, and compare zero-shot accuracy on the unseen NTU splits against the plain VLM; if the gains survive, the reported improvement is not caused by skeleton knowledge transfer.

Watch

Extended reading notes

Core claim

The central discovery is that aligning video-text and skeleton-text similarity scores through online knowledge distillation—rather than fusing skeleton features directly or learning a tri-modal contrastive embedding—transfers skeleton knowledge into a VLM. The transfer happens in the language-contextualized space: both the video-text logits and the skeleton-text logits are computed against text prompts, and the student is trained to match the teacher's logit distribution via mean-squared error while both models keep training on their cross-entropy action losses. After training, the skeleton model is discarded. The paper reports that this SkeletonCLIP Distillation improves zero-shot action recognition on all NTU60 and NTU120 splits, with SKI-ViFiCLIP reaching 82.2% and 52.0% on NTU60 55/5 and 48/12, and 77.5% and 59.3% on NTU120 110/10 and 96/24, surpassing all compared zero-shot action recognition models.

Load-bearing premise

The whole zero-shot gain rests on the assumption that aligning skeleton-text and video-text scores on the seen training classes teaches the video encoder to attend to human joints in a way that transfers to unseen action classes; if the distillation only makes the student copy skeleton-text scores for seen classes, the reported improvements would not generalize.

Editorial extensions

If this is right

  • SKI-VLM can be built on any existing video-text dual encoder, so skeleton enrichment is a plug-in training procedure rather than a new architecture.
  • Because the skeleton branch is discarded at inference, the resulting models carry no extra computational cost or sensor requirement in deployment.
  • The same SkeletonCLIP teacher can be swapped for other skeleton-language models, suggesting the language-contextualized distillation recipe is reusable.
  • SKI-LVLM shows that skeleton tokens also improve text generation about daily activities, so the modality transfer extends beyond recognition to captioning.
  • All tested backbone students (XCLIP, FROSTER, ViFiCLIP) improve over their baselines, meaning the gain is not tied to one architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same distillation recipe could work for other modalities that are informative but unavailable at deployment, such as depth, thermal, or audio, whenever a language-grounded teacher for that modality exists.
  • The paper's ablation shows online distillation wins over offline and feature-level distillation on large-scale data; an implicit consequence is that feature-space mismatch grows with dataset scale, so logit-level language-contextualized alignment may be the right operating point for future multimodal teachers.
  • A testable extension is to probe the trained model with Grad-CAM on held-out classes and compare the attention hotspots against ground-truth joint coordinates, which would directly certify whether skeleton knowledge actually transfers to unseen actions.
  • The gains are demonstrated on NTU60, NTU120, and Charades; the method's value on in-the-wild ADL videos captured in home settings, or on models trained with larger vision-language data, is not established by this paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SKI models, which inject 3D skeleton knowledge into vision-language models. A skeleton-language dual encoder, SkeletonCLIP, is trained on seen NTU action classes, and its predictions are distilled into video CLIP models (XCLIP, ViFiCLIP, FROSTER) through a loss that combines video-text cross-entropy, skeleton-text cross-entropy, and an MSE distillation term in a text-similarity space. The resulting SKI-VLM models are evaluated for zero-shot action recognition on NTU60 and NTU120 splits, with reported gains of up to +5.4% and +7.8% over ViFiCLIP and state-of-the-art claims. A second variant, SKI-LVLM, adds skeleton tokens to a video LLM through a separate projector and is evaluated on dense captioning on Charades. At inference, the skeleton branch is discarded.

Significance. If the reported gains are real and attributable to skeleton knowledge, the work would be a useful contribution: it provides a recipe for injecting a complementary modality into VLMs without requiring that modality at inference, demonstrates the recipe across three VLM backbones, and ships code. The attention-map visualizations and the multi-backbone consistency are genuine strengths. However, the current evaluation contains a test-set hyperparameter sweep and lacks a critical control experiment, so the central attribution claim is not yet established. The idea is worth pursuing, but the evidence as presented is not sufficient for the paper's stronger claims.

major comments (4)
  1. [Appendix F, Figure 5; Section 4.1, Tables 1-2] The distillation weight alpha is selected by sweeping on the NTU48 and NTU110 evaluation splits, and the same splits are used to report the headline accuracies in Tables 1 and 2. Specifically, Figure 5 reports accuracy for NTU48 and NTU110 as a function of alpha, and the values alpha=0.01 (NTU60) and alpha=10.0 (NTU120) are then used in the official tables. This is a form of test-set hyperparameter selection, so the posted gains of +5.4% and +7.8% partially reflect tuning toward the test labels. Please select alpha on a held-out validation split, or pre-specify a fixed value, and report the resulting test numbers.
  2. [Section 3.2, Eq. (3); Section 4.1, Tables 1-2] The comparison for the central claim is uncontrolled. The total loss in Eq. (3) includes L_CE(z^v_i, z^{tv}_i), i.e., standard video-text cross-entropy on the seen NTU classes, and SCD is run for only 2 epochs. The baselines (XCLIP, ViFiCLIP, FROSTER) are pretrained web-video models evaluated without any NTU seen-class training, so the reported gains conflate two effects: adapting the VLM to the NTU seen-class distribution via L_CE, and adding skeleton-language guidance through the distillation term. There is no ablation that trains ViFiCLIP with Eq. (3) while removing the skeleton teacher (for example, alpha=0) or that otherwise keeps the video-text loss fixed. Please add such a control; without it, the paper does not demonstrate that the gains come from skeleton knowledge.
  3. [Section 4.2, Table 3] The SKI-LVLM results are internally inconsistent with the paper's central narrative. Table 3 shows that SK-VCGPT, which uses raw skeleton features without language contextualization, performs essentially identically to SKI-LVLM (average 43.4 vs 43.5). The text acknowledges this is 'surprising' but does not reconcile it with the claim that language-contextualized skeleton knowledge is the key ingredient. This weakens the conclusion that SkeletonCLIP's language grounding is what improves LVLM captioning; the improvement could come from simply injecting any skeleton features. Please provide an analysis or additional experiments that distinguish these explanations.
  4. [Section 4.1, Tables 1-2] All reported accuracies appear to come from a single training run with no error bars or confidence intervals. Given the small margins in several comparisons (e.g., SKI-XCLIP vs XCLIP on NTU60 48/12 is 42.2 vs 38.9, but SKI-FROSTER vs FROSTER is 44.4 vs 43.9), and given that the reported numbers are selected after hyperparameter tuning, the lack of variance estimates makes it difficult to judge whether the improvements are significant. Please report results over multiple seeds or at least provide the standard deviation for the main tables.
minor comments (5)
  1. [Section 3.2, Eq. (3) and surrounding text] The phrase 'minimizing the similarity between modality-specific features and text features' is confusing; the objective actually maximizes the cosine similarity of matched pairs through cross-entropy. Please rephrase to avoid implying that similarity is minimized.
  2. [Section 4, Implementation Details] The statement 'we set alpha to 0.01 for NTU60 and 10.0 for NTU120' is a sharp discontinuity with no explanation. This is related to the test-set selection issue in Major Comment 1 and should at least be motivated by a principled criterion rather than per-dataset tuning on the evaluation split.
  3. [Section 4.3, Table 4] In the text, the discussion of feature-level KD says 'improves performance on NTU60 but doesn't consistently boost accuracy across larger datasets,' yet the harmonic mean for 'Feature-level KD with Projection' is identical to 'without Projection' (58.4). Consider clarifying what 'improves' refers to and whether the difference is meaningful.
  4. [Appendix C.1, Table 6] The comparison between trainable and frozen text encoders is described only for SkeletonCLIP, but the conclusion says 'enhancing skeleton-text-video alignment during SkeletonClip Distillation.' It would help to state explicitly whether the video text encoder remains frozen in both cases.
  5. [General] There are several typos and minor readability issues, such as 'SKeleton' in the title, 'V arying' in the Related Work section, and 'the the' in Appendix D. A careful proofread is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the zero-shot results are genuine held-out evaluations, and the SCD objective does not reduce to its predictions by construction.

full rationale

The central claim is that SkeletonCLIP Distillation (SCD), Eq. (3), transfers skeleton-language compatibility scores to a video-language encoder, improving zero-shot action recognition on unseen NTU classes. The training loss is minimized on the seen classes of each split; inference is performed on disjoint unseen classes. There is no equation in the paper that defines the unseen-class prediction in terms of the training loss, nor is any fitted parameter renamed as a prediction. The distillation weight alpha is selected on the NTU48 and NTU110 seen-class sets reported in Appendix F, not on the held-out unseen test labels, so the headline gains are not statistically forced by alpha selection. The paper benchmarks against external NTU60/NTU120 and Charades protocols, and the gains appear across three VLM backbones, giving the transfer claim independent empirical content. A legitimate weakness is that the total SCD loss includes an ordinary video-text cross-entropy term L_CE(zv_i, ztv_i), and the reported comparison does not include an ablation that fine-tunes ViFiCLIP with the same video-text loss but without the skeleton teacher; this makes the attribution of the gains to skeleton knowledge less clean, but it is an experimental-control concern and not a definitional circularity. Self-citations to prior skeleton-RGB and LVLM works (e.g., Reilly et al. 2024) are used as background or evaluation protocols and are not load-bearing reductions of the central result.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced. SkeletonCLIP and SCD are new model and training constructs, but they are empirically evaluated on public benchmarks and do not function as unexplained explanatory entities.

free parameters (2)
  • Distillation weight alpha = 0.01 (NTU60), 10.0 (NTU120)
    Alpha balances the SCD distillation loss with the cross-entropy losses in Eq. 3. It was selected from Figure 5 by sweeping values on the NTU48 and NTU110 evaluation splits, which are the same splits whose accuracy is reported as the main result, so the number is tuned on the test set.
  • SkeletonCLIP pretraining epochs (140) and text-alignment epochs (100) = 140, 100
    Chosen by hand and not ablated for sensitivity, but pretraining the skeleton encoder is shown to be important in Appendix C.2.
assumptions (3)
  • domain assumption The frozen CLIP text encoder defines a joint semantic space in which skeleton, video, and text features can be meaningfully compared.
    Invoked in Section 3.1 when SkeletonCLIP aligns skeleton features with CLIP text embeddings; if the CLIP text space is not appropriate for skeleton semantics, the teacher signal in SCD would be misaligned.
  • domain assumption Knowledge distilled from skeleton-text alignments on seen classes transfers to unseen classes in zero-shot action recognition.
    The central mechanism of SKI-VLM: SCD trains the video encoder on seen classes, and the paper reports gains on held-out classes without proving the mechanism of transfer.
  • domain assumption The auto-generated NTU120 video-instruction pairs (CogVLM captions plus GPT-3.5 QA) are sufficient supervision for a video captioning LVLM to generalize to the Charades dataset.
    Used in Section 4.2 and Appendix D; the instruction data is synthetic and not yet released, so its adequacy is asserted rather than demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living." pith.science (2026). https://pith.science/paper/7GFB4ZPW

@misc{pith2026250203459,
  author       = {Pith},
  title        = {Pith review of: SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7GFB4ZPW}},
  note         = {Machine review of arXiv:2502.03459}
}
read the original abstract

The introduction of vision-language models like CLIP has enabled the development of foundational video models capable of generalizing to unseen videos and human actions. However, these models are typically trained on web videos, which often fail to capture the challenges present in Activities of Daily Living (ADL) videos. Existing works address ADL-specific challenges, such as similar appearances, subtle motion patterns, and multiple viewpoints, by combining 3D skeletons and RGB videos. However, these approaches are not integrated with language, limiting their ability to generalize to unseen action classes. In this paper, we introduce SKI models, which integrate 3D skeletons into the vision-language embedding space. SKI models leverage a skeleton-language model, SkeletonCLIP, to infuse skeleton information into Vision Language Models (VLMs) and Large Vision Language Models (LVLMs) through collaborative training. Notably, SKI models do not require skeleton data during inference, enhancing their robustness for real-world applications. The effectiveness of SKI models is validated on three popular ADL datasets for zero-shot action recognition and video caption generation tasks.

Figures

Figures reproduced from arXiv: 2502.03459 by the authors.

Figure 1
Figure 1. Left: The illustration depicts an embedding space of a Vision-Language Model (VLM) where representations of web￾based videos align closely with their corresponding class label text features, while those of Activities of Daily Living (ADL) videos remain distant. Our study reveals that integrating skeleton guidance bridges this gap, aligning ADL video representations with their respective class labels. Right: Activati… view at source ↗
Figure 2
Figure 2. (a) SkeletonCLIP: Utilizes a pretrained Skeleton Backbone, aligned with action class labels from the frozen CLIP Text Encoder during Skeleton-Language Supervision. (b) SKI-VLM: Engages in online distillation between SkeletonCLIP and a Vision-Language Model (VLM), both trainable. For inference, the VLM alone performs zero-shot action recognition on unseen classes. (c) SKI-LVLM: Projects SkeletonCLIP features into LLM… view at source ↗
Figure 3
Figure 3. Attention Map Visualization: Comparison be￾tween ViFiCLIP and SKI-VLM. While ViFiCLIP struggles to identify the critical areas responsible for actions, SKI￾VLM accurately focuses on the relevant joints, such as hands and face, for actions like Sneeze/Cough. B Effectiveness of SKI-VLM for ADL In [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Illustration of how SkeletonCLIP can be easily integrated with VLMs like ViFiCLIP, FROSTER and XCLIP ( [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Impact of α in SKI-VLM for NTU48 and NTU110 In [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 33 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Aytar, Y.; Vondrick, C.; and Torralba, A. 2016. SoundNet: Learning Sound Representations from Unlabeled Video. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, 892–900. Red Hook, NY, USA: Curran Associates Inc. ISBN 9781510838819

    work page 2016

  5. [5]

    Caba Heilbron, F.; Escorcia, V.; Ghanem, B.; and Carlos Niebles, J. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 961--970

    work page 2015

  6. [6]

    Chen, Y.; Guo, J.; He, T.; and Wang, L. 2024. Fine-Grained Side Information Guided Dual-Prompts for Zero-Shot Skeleton Action Recognition

    work page 2024

  7. [7]

    Cheng, K.; Zhang, Y.; He, X.; Chen, W.; Cheng, J.; and Lu, H. 2020. Skeleton-Based Action Recognition with Shift Graph Convolutional Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2020

  8. [8]

    E.; Stoica, I.; and Xing, E

    Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\

    2023

Show all 57 references
  1. [9]

    Crasto, N.; Weinzaepfel, P.; Alahari, K.; and Schmid, C. 2019. MARS: Motion-Augmented RGB Stream for Action Recognition . In CVPR

    2019

  2. [10]

    Darcet, T.; Oquab, M.; Mairal, J.; and Bojanowski, P. 2023. Vision Transformers Need Registers. ArXiv, abs/2309.16588

    2023 arXiv

  3. [11]

    Das, S.; Dai, R.; Koperski, M.; Minciullo, L.; Garattoni, L.; Bremond, F.; and Francesca, G. 2019. Toyota Smarthome: Real-World Activities of Daily Living. In Int. Conf. Comput. Vis

    2019

  4. [12]

    Das, S.; Dai, R.; Yang, D.; and Bremond, F. 2021. VPN++: Rethinking Video-Pose embeddings for understanding Activities of Daily Living. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1--1

    2021

  5. [13]

    Das, S.; Sharma, S.; Dai, R.; Bremond, F.; and Thonnat, M. 2020. Vpn: Learning video-pose embedding for activities of daily living. In European Conference on Computer Vision, 72--90. Springer

    2020

  6. [14]

    Duan, H.; Zhao, Y.; Chen, K.; Lin, D.; and Dai, B. 2022. Revisiting Skeleton-based Action Recognition. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2959--2968

    2022

  7. [15]

    K.; Sun, Y.; Patel, P.; and Black, M

    Feng, Y.; Lin, J.; Dwivedi, S. K.; Sun, Y.; Patel, P.; and Black, M. J. 2024. ChatPose: Chatting about 3D Human Pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2024

  8. [16]

    V.; Joulin, A.; and Misra, I

    Girdhar, R.; El-Nouby, A.; Liu, Z.; Singh, M.; Alwala, K. V.; Joulin, A.; and Misra, I. 2023. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15180--15190

    2023

  9. [17]

    Gupta, P.; Sharma, D.; and Sarvadevabhatla, R. K. 2021. Syntactically Guided Generative Embeddings for Zero-Shot Skeleton Action Recognition. 2021 IEEE International Conference on Image Processing (ICIP), 439--443

    2021

  10. [18]

    Gupta, S.; Hoffman, J.; and Malik, J. 2016. Cross Modal Distillation for Supervision Transfer

    2016

  11. [19]

    Guzhov, A.; Raue, F.; Hees, J.; and Dengel, A. R. 2021. Audioclip: Extending Clip to Image, Text and Audio. ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 976--980

    2021

  12. [20]

    Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the Knowledge in a Neural Network. arXiv:1503.02531

    2015 arXiv

  13. [21]

    Huang, X.; Zhou, H.; Yao, K.; and Han, K. 2024. FROSTER: Frozen CLIP is a Strong Teacher for Open-Vocabulary Action Recognition. In International Conference on Learning Representations

    2024

  14. [22]

    Kay, W.; Carreira, J.; Simonyan, K.; Zhang, B.; Hillier, C.; Vijayanarasimhan, S.; Viola, F.; Green, T.; Back, T.; Natsev, P.; et al. 2017. The kinetics human action video dataset . arXiv preprint arXiv:1705.06950

    2017 arXiv

  15. [23]

    Kuehne, H.; Jhuang, H.; Garrote, E.; Poggio, T.; and Serre, T. 2011. HMDB: a large video database for human motion recognition . In 2011 International Conference on Computer Vision, 2556--2563. IEEE

    2011

  16. [24]

    Lin, B.; Zhu, B.; Ye, Y.; Ning, M.; Jin, P.; and Yuan, L. 2023. Video-LLaVA: Learning United Visual Representation by Alignment Before Projection. arXiv preprint arXiv:2311.10122

    2023 arXiv

  17. [25]

    Liu, J.; Shahroudy, A.; Perez, M.; Wang, G.; Duan, L.-Y.; and Kot, A. C. 2019. NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2019

  18. [26]

    Lu, J.; Clark, C.; Zellers, R.; Mottaghi, R.; and Kembhavi, A. 2022. Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks. arXiv preprint arXiv:2206.08916

    2022 arXiv

  19. [27]

    Maaz, M.; Rasheed, H.; Khan, S.; and Khan, F. S. 2024. Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)

    2024

  20. [28]

    Meta. 2024. The Llama 3 Herd of Models. arXiv:2407.21783

    2024 arXiv

  21. [29]

    Ni, B.; Peng, H.; Chen, M.; Zhang, S.; Meng, G.; Fu, J.; Xiang, S.; and Ling, H. 2022. Expanding Language-Image Pretrained Models for General Video Recognition

    2022

  22. [30]

    Pan, J.; Lin, Z.; Zhu, X.; Shao, J.; and Li, H. 2022. ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning. In Koyejo, S.; Mohamed, S.; Agarwal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Advances in Neural Information Processing Systems, volume 35, 26462--26477....

    2022

  23. [31]

    Qian, R.; Li, Y.; Xu, Z.; Yang, M.-H.; Belongie, S.; and Cui, Y. 2022. Multimodal open-vocabulary video classification via pre-trained vision and language models. arXiv preprint arXiv:2207.07646

    2022 arXiv

  24. [32]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning

    2021

  25. [33]

    U.; Maaz, M.; Khan, S.; and Khan, F

    Rasheed, H.; Khattak, M. U.; Maaz, M.; Khan, S.; and Khan, F. S. 2023. Finetuned CLIP models are efficient video learners. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition

    2023

  26. [34]

    K.; Wang, P.; Bremond, F.; Xue, L.; and Das, S

    Reilly, D.; Chakraborty, R.; Sinha, A.; Govind, M. K.; Wang, P.; Bremond, F.; Xue, L.; and Das, S. 2024. LLAVIDAL: A Large LAnguage VIsion Model for Daily Activities of Living. arXiv, 2406.09390

    2024 arXiv

  27. [35]

    Reilly, D.; and Das, S. 2024. Just Add ! Pose Induced Video Transformers for Understanding Activities of Daily Living. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2024

  28. [36]

    Rogez, G.; Weinzaepfel, P.; and Schmid, C. 2019. LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images . IEEE Transactions on Pattern Analysis and Machine Intelligence

    2019

  29. [37]

    Ruan, L.; Hu, A.; Song, Y.; Zhang, L.; Zheng, S.; and Jin, Q. 2023. Accommodating Audio Modality in CLIP for Multimodal Processing. In AAAI Conference on Artificial Intelligence

    2023

  30. [38]

    Shahroudy, A.; Liu, J.; Ng, T.-T.; and Wang, G. 2016. NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis. In IEEE Conf. Comput. Vis. Pattern Recog

    2016

  31. [39]

    Shi, L.; Zhang, Y.; Cheng, J.; and Lu, H. 2019. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12026--12035

    2019

  32. [40]

    A.; Varol, G.; Wang, X.; Farhadi, A.; Laptev, I.; and Gupta, A

    Sigurdsson, G. A.; Varol, G.; Wang, X.; Farhadi, A.; Laptev, I.; and Gupta, A. 2016. Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding . In European Conference on Computer Vision(ECCV)

    2016

  33. [41]

    Simonyan, K.; and Zisserman, A. 2014. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, 568--576

    2014

  34. [42]

    R.; and Shah, M

    Soomro, K.; Zamir, A. R.; and Shah, M. 2012. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. CoRR, abs/1212.0402

    2012 arXiv

  35. [43]

    Su, Y.; Lan, T.; Li, H.; Xu, J.; Wang, Y.; and Cai, D. 2023. PandaGPT: One Model To Instruction-Follow Them All. arXiv preprint arXiv:2305.16355

    2023 arXiv

  36. [44]

    Sun, Q.; Fang, Y.; Wu, L.; Wang, X.; and Cao, Y. 2023. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389

    2023 arXiv

  37. [45]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lample, G. 2023. LLaMA: Open and Efficient Foundation Language Models. ArXiv, abs/2302.13971

    2023 arXiv

  38. [46]

    Wang, M.; Xing, J.; and Liu, Y. 2021. ActionCLIP: A New Paradigm for Video Action Recognition. ArXiv, abs/2109.08472

    2021 arXiv

  39. [47]

    Wang, W.; Lv, Q.; Yu, W.; Hong, W.; Qi, J.; Wang, Y.; Ji, J.; Yang, Z.; Zhao, L.; Song, X.; Xu, J.; Xu, B.; Li, J.; Dong, Y.; Ding, M.; and Tang, J. 2023. CogVLM: Visual Expert for Pretrained Language Models. ArXiv, abs/2311.03079

    2023 arXiv

  40. [48]

    Wu, H.-H.; Seetharaman, P.; Kumar, K.; and Bello, J. P. 2021. Wav2CLIP: Learning Robust Audio Representations from Clip. ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4563--4567

    2021

  41. [49]

    Yan, S.; Xiong, Y.; and Lin, D. 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Thirty-second AAAI conference on artificial intelligence

    2018

  42. [50]

    Yan, T.; Zeng, W.; Xiao, Y.; Tong, X.; Tan, B.; Fang, Z.; Cao, Z.; and Zhou, J. T. 2024. CrossGLG: LLM Guides One-shot Skeleton-based 3D Action Recognition in a Cross-level Manner. arXiv preprint arXiv:2403.10082

    2024 arXiv

  43. [51]

    Yang, T.; Zhu, Y.; Xie, Y.; Zhang, A.; Chen, C.; and Li, M. 2023. AIM: Adapting Image Models for Efficient Video Understanding. In International Conference on Learning Representations

    2023

  44. [52]

    Zhang, D.; Li, S.; Zhang, X.; Zhan, J.; Wang, P.; Zhou, Y.; and Qiu, X. 2023. SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities. arXiv:2305.11000

    2023 arXiv

  45. [53]

    Zhang, H.; Li, X.; and Bing, L. 2023. Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. ArXiv, abs/2306.02858

    2023 arXiv

  46. [54]

    Zhang, Z. 2012. Microsoft Kinect Sensor and Its Effect. In IEEE MultiMedia, volume 19

    2012

  47. [55]

    Zhou, Y.; Cheng, Z.-Q.; Li, C.; Geng, Y.; Xie, X.; and Keuper, M. 2022. Hypergraph Transformer for Skeleton-based Action Recognition. arXiv preprint arXiv:2211.09590

    2022 arXiv

  48. [56]

    Zhou, Y.; Qiang, W.; Rao, A.; Lin, N.; Su, B.; and Wang, J. 2023. Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization. In Proceedings of the 31st ACM International Conference on Multimedia, 5302--5310

    2023

  49. [57]

    Zhu, B.; Lin, B.; Ning, M.; Yan, Y.; Cui, J.; Wang, H.; Pang, Y.; Jiang, W.; Zhang, J.; Li, Z.; Zhang, W.; Li, Z.; Liu, W.; and Yuan, L. 2023. LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment. ArXiv, abs/2310.01852

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.