Pith. sign in

REVIEW 5 major objections 5 minor 47 references

ResNetVLLM -- Multi-modal Vision LLM for the Video Understanding Task

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A non-pretrained ResNet plus an LLM claims state-of-the-art zero-shot video understanding.

desk verdict The SOTA claim rests on a training recipe that cannot run as described, and the method section omits the loss and projection details, so the reported numbers are not supported. read the letter →

arxiv 2504.14432 v1 pith:GDCXPHVV submitted 2025-04-20 cs.CV cs.AI

Ahmad Khalil , Mahmoud Khalil , Alioune Ngom This is my paper · ORCID
classification cs.CVcs.AI
keywords zero-shotvideounderstandingquestionansweringResNetvisualencoderlargelanguagemodelmulti-modalLLMVideo-ChatGPTbenchmarkcross-modallearningcaptioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ResNetVLLM is a proposed video-understanding model that joins a randomly initialized, non-pretrained ResNet visual encoder to a large language model, training them together on video-text pairs and then evaluating without seeing question-answer training data. The paper's central claim is that this design beats every compared video LLM on four open-ended video QA benchmarks—MSRVTT-QA, MSVD-QA, TGIF-QA FrameQA, and ActivityNet-QA—and on all five generative quality axes of the Video-ChatGPT benchmark. If true, expensive pretrained video encoders are unnecessary for strong zero-shot video understanding, and a simple two-stage recipe running about four hours on two GPUs can learn visual and semantic representations in one architecture. The paper introduces no new dataset; it trains on Video-ChatGPT-100K.

What carries the argument

The central object is the ResNetVLLM architecture: a vanilla, non-pretrained 2D ResNet visual encoder connected to a LLaVA-based LLM transformer, with [CLS] tokens and flattened global average-pooled features projected into the sequence the language model reads. The random weight initialization of the ResNet is the mechanism intended to guarantee zero-shot fairness by preventing prior video knowledge from entering the model. Training proceeds in two stages: a $150$-epoch warm-up that optimizes the ResNet alone with SGD, followed by a $50$-epoch joint stage that optimizes the ResNet and the LLM transformer together with AdamW.

What would settle it

An independent reproduction using only the details given in the paper—100 frames per video at 6-frame intervals, random $224\times224$ crops, a randomly initialized ResNet warm-up for $150$ epochs with SGD at learning rate $0.01$, joint training for $50$ epochs with AdamW at learning rate $0.00015$, and evaluation on MSVD-QA—would disprove the claim if the resulting accuracy does not approach the reported $78.3$.

Watch

Extended reading notes

Core claim

The paper's discovery claim is that a vanilla, randomly initialized 2D ResNet can serve as the visual encoder of a video LLM and outperform models built on pretrained video features in zero-shot settings. The model samples frames at regular intervals, extracts features with the ResNet, flattens the global average-pooled features, projects them into a LLaVA-based language model, and is trained first with a ResNet-only warm-up and then jointly with the LLM. The authors report state-of-the-art accuracy on MSVD-QA, MSRVTT-QA, TGIF-QA FrameQA, and ActivityNet-QA, and a 3.55 mean score on the Video-ChatGPT generative benchmark, above the previous best 2.42. They attribute the gains to joint learning of visual and semantic representations in a shared space, which they say minimizes information loss and makes the embeddings discriminative and semantically consistent.

Load-bearing premise

For the central claim to hold, the described two-stage training must be a well-defined procedure—with a concrete warm-up objective and a compatible way of feeding ResNet features into the language model—so that someone else could reproduce the reported scores.

Editorial extensions

If this is right

  • Pretrained video encoders become optional rather than necessary for competitive zero-shot video QA, lowering the barrier to building new video LLMs.
  • The two-stage recipe of a random-encoder warm-up followed by joint LLM fine-tuning becomes a reusable training template for other vision-language tasks.
  • The reported scores give the video-LLM community new reference points: $78.3$ on MSVD-QA, $63.5$ on MSRVTT-QA, $59.9$ on TGIF-QA, and $54.8$ on ActivityNet-QA.
  • A direct corollary is that frozen CLIP-style visual features are not required for strong video understanding, contrary to the usual design assumption.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the magnitude of the reported gains suggests the LLM's language prior, trained on vast text, may carry more of the semantic and temporal load than the visual encoder, since the random ResNet is trained on only 100K video-text pairs for a few hours.
  • Editorial inference: because the model feeds flattened global ResNet features rather than tokenized spatial features, it likely captures global scene content better than fine-grained object details; a testable extension would probe the model on questions about small objects or spatial relations.
  • Editorial inference: the same random-encoder-plus-LLM design could be transferred to audio or sensor streams, although the paper only demonstrates video.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript proposes ResNetVLLM, a video understanding framework that combines a randomly initialized 2D ResNet visual encoder with a LLaVA-based language model. It claims state-of-the-art zero-shot video question-answering results on MSRVTT-QA, MSVD-QA, TGIF-QA FrameQA, ActivityNet-QA, and the VideoChatGPT generative benchmark. Training is described as a two-stage procedure in which the ResNet is first warmed up alone and then the full model is jointly trained on Video-ChatGPT-100K for 50 epochs on two NVIDIA V100 GPUs in about four hours.

Significance. The reported findings, if reproducible, would be significant: a non-pretrained ResNet trained jointly with an LLM in a few hours outperforming models that rely on CLIP or pretrained video encoders would challenge common assumptions about pretraining. The paper uses standard benchmarks and compares against recent baselines, which is an appropriate experimental framing. However, the manuscript provides no code, checkpoints, evaluation scripts, error bars, or significance tests, and the training protocol as stated is both underdetermined and computationally implausible. Until those issues are resolved, the claimed state-of-the-art results are not verifiable and the significance of the contribution cannot be assessed.

major comments (5)
  1. [IV-A, Warm-Up phase] The Warm-Up phase is specified only as training the ResNet for 150 epochs with SGD, a learning rate of 0.01, and a weight decay of 0.0001; no loss function, labels, output head, or data split is given. Without a defined objective, this phase cannot be executed or reproduced, and any resulting encoder is not well-defined. This is load-bearing because the central claim of learning from a non-pretrained encoder depends on this phase actually training the encoder.
  2. [IV-A, Joint-Training phase] The paper states that the entire model, including the ResNet encoder and the LLaVA transformer, is trained for 50 epochs over the 100K-pair Video-ChatGPT-100K dataset with AdamW on 2 NVIDIA V100 GPUs in approximately 4 hours. Under standard dense AdamW fine-tuning, the weights, gradients, and optimizer states of a 7B or 13B transformer exceed the memory of two V100s, and 5 million sample-epochs of such a model require far more compute than two V100s can deliver in four hours. The description therefore omits essential implementation details such as parameter-efficient adaptation (LoRA/adapters), offloading, gradient checkpointing, or reduced sequence length; as written, the experiment is not executable, and the results in Tables I and II cannot be attributed to the stated training protocol.
  3. [III-C, Model Formulation] The visual features are described as flattened global ResNet features that are "projected" and merged with the tokenized text, but no projection layer, feature dimension, or injection mechanism into the LLaVA transformer is defined. This leaves the architecture underdetermined and makes it impossible to reproduce the method from the text alone.
  4. [IV-A and IV-C, ActivityNet overlap] Training uses the Video-ChatGPT-100K instruction dataset, which is built from videos including ActivityNet, while evaluation includes ActivityNet-QA and the ActivityNet-200-based VideoChatGPT generative benchmark. The paper neither reports whether the training and evaluation videos overlap nor enforces disjoint splits for the zero-shot evaluation; if overlap exists, the zero-shot claim is compromised. The authors should state the overlap and provide a contamination analysis.
  5. [Tables I and II] All reported numbers are single-run results without error bars, multiple seeds, or significance tests, and no code, checkpoints, or evaluation scripts are provided. Given the large claimed gaps over prior work (e.g., Table I Mean 3.55 vs. 2.42 for Video-ChatGPT, and Table II MSVD-QA 78.3 vs. 67.5 for BT-Adapter), this lack of reproducibility evidence is a central obstacle to accepting the empirical claims.
minor comments (5)
  1. [Abstract] The phrase "with a Large Language Model (LLM." is missing the closing parenthesis and should read "(LLM).".
  2. [Sections III and IV] The term "LLaV A" appears with an unintended space; the standard spelling "LLaVA" should be used throughout.
  3. [IV-B, Evaluation Metrics] The equations for CI, DO, CU, TU, and C define symbols such as fCI, wc, ws, Ci, and Si only in prose and are not algorithmically specified; please either provide precise computational definitions or defer to the official VideoChatGPT evaluation implementation.
  4. [IV-A, Implementation Details] Bayesian Optimization is mentioned twice, but neither the objective function nor the hyperparameter search space is described, which makes the tuning procedure unreproducible.
  5. [Table II caption] The caption contains a duplicated article: "THE THE ZERO-SHOT QUESTION-ANSWER EVALUATION".

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper's claims are empirical benchmark comparisons, not derivations.

full rationale

The paper's central claims are empirical performance numbers (Tables I and II) on external benchmarks (VideoChatGPT benchmarking, MSRVTT-QA, MSVD-QA, TGIF-QA FrameQA, ActivityNet-QA). There is no derivation chain in which an output is defined in terms of its own target, and no fitted parameter is renamed as a prediction. The method section describes an architecture and a training procedure, but no equation in the paper reduces a claimed result to its own input. The only potentially circular-looking concern is that the Video-ChatGPT-100K training data might overlap with the ActivityNet-based evaluation, but the paper never states that overlap, so under the hard-evidence rule this cannot be counted as a demonstrated circular step. The missing loss specification for the Warm-Up phase and the implausible training budget are reproducibility/correctness concerns, not circularity. There are no load-bearing author self-citations; the cited prior work (LLaVA, Video-ChatGPT, ResNet) is external. Accordingly, the paper's claims are self-contained empirical assertions and receive a circularity score of 0.

Assumptions & free parameters 13 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced; ResNetVLLM is an architecture, not a postulated entity. The main free choices are training hyperparameters and the unspecified feature projection. The axioms are domain assumptions about the training data, the zero-shot protocol, and the GPT-based evaluation, none of which are independently verified in the paper.

free parameters (13)
  • Warm-up learning rate = 0.01
    Chosen by Bayesian optimization; no search range or validation split disclosed.
  • Warm-up weight decay = 0.0001
    Chosen by Bayesian optimization; no search range or validation split disclosed.
  • Warm-up epochs = 150
    Selected by the authors.
  • Joint-training learning rate = 0.00015
    Chosen by Bayesian optimization; no search range or validation split disclosed.
  • Joint-training weight decay = 0.05
    Chosen by Bayesian optimization; no search range or validation split disclosed.
  • Joint-training epochs = 50
    Selected by the authors.
  • Number of sampled frames = 100
    Selected by the authors.
  • Frame interval = 6
    Selected by the authors.
  • Random crop size = 224 x 224
    Selected by the authors.
  • Training clips per video = 1
    Selected by the authors.
  • Test clips per video = 25
    Selected by the authors.
  • ResNet depth and width
    Not specified in the paper; affects capacity and is a free architectural choice.
  • Feature projection dimension
    The projection of ResNet features into the LLM token space is never specified.
assumptions (5)
  • domain assumption Video-ChatGPT-100K contains sufficiently diverse video-text pairs to learn transferable visual representations from a randomly initialized ResNet.
    Section IV-A uses this dataset as the only training signal; no analysis supports the transfer claim.
  • ad hoc to paper The warm-up phase has a well-defined objective that teaches the ResNet meaningful features, even though the loss is not stated.
    Section IV-A says the ResNet is trained for 150 epochs but omits the loss, labels, and output head.
  • domain assumption The zero-shot protocol is valid: training on Video-ChatGPT-100K does not expose the evaluation benchmarks.
    ActivityNet-QA is built from ActivityNet videos, and Video-ChatGPT-100K includes instruction data derived from ActivityNet; Section III-D and IV-A rely on this.
  • domain assumption GPT-3.5 based scoring in the VideoChatGPT benchmark provides accurate, comparable quality scores.
    Section IV-B describes the scoring but provides no evidence of reliability or comparability.
  • domain assumption The LLaVA transformer can consume projected ResNet features without architectural changes.
    Section III-C states visual features are merged with tokenized system command but does not specify the projector architecture.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ResNetVLLM -- Multi-modal Vision LLM for the Video Understanding Task." pith.science (2026). https://pith.science/paper/GDCXPHVV

@misc{pith2026250414432,
  author       = {Pith},
  title        = {Pith review of: ResNetVLLM -- Multi-modal Vision LLM for the Video Understanding Task},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GDCXPHVV}},
  note         = {Machine review of arXiv:2504.14432}
}
read the original abstract

In this paper, we introduce ResNetVLLM (ResNet Vision LLM), a novel cross-modal framework for zero-shot video understanding that integrates a ResNet-based visual encoder with a Large Language Model (LLM. ResNetVLLM addresses the challenges associated with zero-shot video models by avoiding reliance on pre-trained video understanding models and instead employing a non-pretrained ResNet to extract visual features. This design ensures the model learns visual and semantic representations within a unified architecture, enhancing its ability to generate accurate and contextually relevant textual descriptions from video inputs. Our experimental results demonstrate that ResNetVLLM achieves state-of-the-art performance in zero-shot video understanding (ZSVU) on several benchmarks, including MSRVTT-QA, MSVD-QA, TGIF-QA FrameQA, and ActivityNet-QA.

Figures

Figures reproduced from arXiv: 2504.14432 by the authors.

Figure 1
Figure 1. Overview of ResNetVLLM. volume of tokens to model spatial-temporal dependencies across successive video frames simultaneously [12]. To tackle these challenges, recent methods in zero-shot VideoLLMs primarily leverage visual features obtained from pre-trained video understanding models. These methods use either pooling operations [13], [14] or query aggregation [15]–[17] on the token sequence of the video before pass… view at source ↗
Figure 2
Figure 2. Overall architecture of the proposed ResNetVLLM framework. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Training Phase of ResNetVLLM. approach has proven effective in video understanding tasks [44]. Lastly, the projected visual features are merged with the tokenized system command and fed into the LLM to produce the video description. In the zero-shot setting, the ResNet feature encoder is initialized with random weights to prevent prior knowledge from influencing the model’s performance. During training, both the Res… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Sample Output of ResNetVLLM [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 28 canonical work pages

  1. [1]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023, 1, 4

    arXiv 2023

  2. [2]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023, 1, 4

    arXiv 2023

  3. [3]

    Stanford alpaca: An instruction-following llama model,

    R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model,” 2023, 1, 4

    work page 2023

  4. [4]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” 2023, 1, 4

    work page 2023

  5. [5]

    Chatgpt,

    ——, “Chatgpt,” https://openai.com/blog/chatgpt/, 2023, 1, 4, 5, 9

    work page 2023

  6. [6]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. Gonzalez et al. , “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” https://vicuna. lmsys.org, 2023, 1, 4, 9

    work page 2023

  7. [7]

    Instructblip: Towards general-purpose vision- language models with instruction tuning,

    W. Dai, J. Li, D. Li, A.-M.-H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision- language models with instruction tuning,” 2023, 1

    work page 2023

  8. [8]

    Llama-adapter v2: Parameter-efficient visual instruction model,

    P. Gao, J. Han, R. Zhang, Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yue et al. , “Llama-adapter v2: Parameter-efficient visual instruction model,” arXiv preprint arXiv:2304.15010 , 2023, 1, 4, 9, 12

    arXiv 2023

Show all 47 references
  1. [9]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” arXiv preprint arXiv:2304.08485, 2023, 1, 4, 6, 9

    2023 arXiv

  2. [10]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” arXiv preprint arXiv:2304.10592 , 2023, 1, 4

    2023 arXiv

  3. [11]

    Locality and compositionality in zero-shot learning,

    T. Sylvain, L. Petrini, and D. Hjelm, “Locality and compositionality in zero-shot learning,” in International Conference on Learning Represen- tations, 2020, 1

    2020

  4. [12]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in Proceedings of the International Conference on Machine Learning (ICML) . PMLR, 2021,...

    2021

  5. [13]

    Valley: Video assistant with large language model enhanced ability,

    R. Luo, Z. Zhao, M. Yang, J. Dong, M. Qiu, P. Lu, T. Wang, and Z. Wei, “Valley: Video assistant with large language model enhanced ability,” arXiv preprint arXiv:2306.07207 , 2023, 2, 3, 5, 7, 9

    2023 arXiv

  6. [14]

    Video-chatgpt: Towards detailed video understanding via large vision and language models,

    M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “Video-chatgpt: Towards detailed video understanding via large vision and language models,” arXiv preprint arXiv:2306.05424, 2023, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15

    2023 arXiv

  7. [15]

    Video-llama: An instruction-tuned audio-visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio-visual language model for video understanding,” arXiv preprint arXiv:2306.02858, 2023, 2, 3, 5, 7, 9, 12

    2023 arXiv

  8. [16]

    Videochat: Chat-centric video understanding,

    K. Li, Y . He, Y . Wang, Y . Li, W. Wang, P. Luo, Y . Wang, L. Wang, and Y . Qiao, “Videochat: Chat-centric video understanding,” arXiv preprint arXiv:2305.06355, 2023, 2, 3, 5, 7, 9, 12

    2023 arXiv

  9. [17]

    Moviechat: From dense token to sparse mem- ory for long video understanding,

    E. Song, W. Chai, G. Wang, Y . Zhang, H. Zhou, F. Wu, X. Guo, T. Ye, Y . Lu, J.-N. Hwanget al., “Moviechat: From dense token to sparse mem- ory for long video understanding,” arXiv preprint arXiv:2307.16449 , 2023, 2, 3, 5, 7

    2023 arXiv

  10. [18]

    Learning spatiotemporal features with 3d convolutional networks,

    DuTran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in Proceedings of the IEEE International Conference on Computer Vision , 2015, pp. 4489–4497

    2015

  11. [19]

    Quo vadis, action recognition? a new model and the kinetics dataset,

    J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 6299–6308

    2017

  12. [20]

    Joint learning of attended zero-shot features and visual-semantic mapping,

    Y . Li and D. Wang, “Joint learning of attended zero-shot features and visual-semantic mapping,” in BMVC, 2019

    2019

  13. [21]

    A review of generalized zero-shot learning methods,

    F. Pourpanah, M. Abdar, Y . Luo, X. Zhou, R. Wang, C. P. Lim, and X.- Z. Wang, “A review of generalized zero-shot learning methods,” arXiv preprint arXiv:2011.08641, 2020, 2

    2011 arXiv

  14. [22]

    Learning a deep embedding model for zero-shot learning,

    L. Zhang, T. Xiang, and S. Gong, “Learning a deep embedding model for zero-shot learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 2021–2030, 2, 7

    2017

  15. [23]

    Video question answering via gradually refined attention over appearance and motion,

    D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y . Zhuang, “Video question answering via gradually refined attention over appearance and motion,” in Proceedings of the 25th ACM International Conference on Multimedia, 2017, pp. 1645–1653

    2017

  16. [24]

    Tgif-qa: Toward spatio- temporal reasoning in visual question answering,

    Y . Jang, Y . Song, Y . Yu, Y . Kim, and G. Kim, “Tgif-qa: Toward spatio- temporal reasoning in visual question answering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 2758–2766

    2017

  17. [25]

    Activitynet- qa: A dataset for understanding complex web videos via question answering,

    Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y . Zhuang, and D. Tao, “Activitynet- qa: A dataset for understanding complex web videos via question answering,” in Proceedings of the AAAI Conference on Artificial In- telligence, vol. 33, 2019, pp. 9127–9134

    2019

  18. [26]

    Latent dirichlet allocation,

    D. M. Blei, A. Y . Ng, and M. I. Jordan, “Latent dirichlet allocation,” Journal of Machine Learning Research , vol. 3, pp. 993–1022, 2003

    2003

  19. [27]

    Efficient estimation of word representations in vector space,

    T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” in Proceedings of the Interna- tional Conference on Learning Representations (ICLR) , 2013

    2013

  20. [28]

    Skip-thought vectors,

    R. Kiros, Y . Zhu, R. Salakhutdinov, R. S. Zemel, R. Urtasun, A. Torralba, and S. Fidler, “Skip-thought vectors,” in Advances in Neural Information Processing Systems (NeurIPS) , 2015, pp. 3294–3302

    2015

  21. [29]

    Distributed representations of sentences and documents,

    Q. Le and T. Mikolov, “Distributed representations of sentences and documents,” in Proceedings of the International Conference on Machine Learning (ICML), 2014, pp. 1188–1196

    2014

  22. [30]

    A neural proba- bilistic language model,

    Y . Bengio, R. Ducharme, P. Vincent, and C. Jauvin, “A neural proba- bilistic language model,” Journal of Machine Learning Research, vol. 3, pp. 1137–1155, 2003

    2003

  23. [31]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165 , 2020

    2005 arXiv

  24. [32]

    Palm: The power of language model pools,

    A. Chowdhery, E. Clark, U. Khandelwal, Y . Lin, A. Sabharwal, A. Vaswani, and X. Zhang, “Palm: The power of language model pools,” arXiv preprint arXiv:2202.06755 , 2022

    2022 arXiv

  25. [33]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin et al. , “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068 , 2022

    2022 arXiv

  26. [34]

    Flamingo: A visual language model for few-shot learning,

    J.-B. Alayrac et al., “Flamingo: A visual language model for few-shot learning,” 2022

    2022

  27. [35]

    Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture,

    T. Gupta, A. Kamath, A. Kembhavi, and D. Hoiem, “Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022

    2022

  28. [36]

    Class-agnostic object detection with multi-modal transformer,

    M. Maaz, H. Rasheed, S. Khan, F. S. Khan, R. M. Anwer, and M.-H. Yang, “Class-agnostic object detection with multi-modal transformer,” in Proceedings of the European Conference on Computer Vision (ECCV) . Springer, 2022

    2022

  29. [37]

    Bridg- ing the gap between object and image-level representations for open- vocabulary detection,

    H. Rasheed, M. Maaz, M. U. Khattak, S. Khan, and F. S. Khan, “Bridg- ing the gap between object and image-level representations for open- vocabulary detection,” in Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  30. [38]

    Open-vocabulary semantic segmentation with mask-adapted clip,

    F. Liang, B. Wu, X. Dai, K. Li, Y . Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu, “Open-vocabulary semantic segmentation with mask-adapted clip,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023

    2023

  31. [39]

    Language-grounded indoor 3d semantic segmentation in the wild,

    D. Rozenberszki, O. Litany, and A. Dai, “Language-grounded indoor 3d semantic segmentation in the wild,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2022

    2022

  32. [40]

    Expanding language-image pretrained models for general video recognition,

    B. Ni, H. Peng, M. Chen, S. Zhang, G. Meng, J. Fu, S. Xiang, and H. Ling, “Expanding language-image pretrained models for general video recognition,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2022

    2022

  33. [41]

    Actionclip: A new paradigm for video action recognition,

    M. Wang, J. Xing, and Y . Liu, “Actionclip: A new paradigm for video action recognition,” arXiv preprint arXiv:2109.08472 , 2021

    2021 arXiv

  34. [42]

    Finetuned clip models are efficient video learners,

    H. Rasheed, M. U. Khattak, M. Maaz, S. Khan, and F. S. Khan, “Finetuned clip models are efficient video learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  35. [43]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015

    2015

  36. [44]

    Temporal segment networks: Towards good practices for deep action recognition,

    L. Wang, Y . Xiong, Z. Wang, Y . Qiao, D. Lin, X. Tang, and L. Van Gool, “Temporal segment networks: Towards good practices for deep action recognition,” in European conference on computer vision . Springer, 2016, pp. 20–36

    2016

  37. [45]

    Activi- tynet: A large-scale video benchmark for human activity understanding,

    F. Caba Heilbron, V . Escorcia, B. Ghanem, and J. C. Niebles, “Activi- tynet: A large-scale video benchmark for human activity understanding,” in Proceedings of the ieee conference on computer vision and pattern recognition, 2015, pp. 961–970

    2015

  38. [46]

    One for all: Video conversation is feasible without video instruction tuning,

    R. Liu, C. Li, Y . Ge, Y . Shan, T.-H. Li, and G. Li, “One for all: Video conversation is feasible without video instruction tuning,” arXiv preprint arXiv:2309.15785, pp. 5, 9, 10, 12, 2023

    2023 arXiv

  39. [47]

    Zero-shot video question answering via frozen bidirectional language models,

    A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid, “Zero-shot video question answering via frozen bidirectional language models,” in NeurIPS 35, 2022, pp. 124–141. APPENDIX A SAMPLE OUTPUT OF RESNETVLLM Fig. 4. Sample Output of ResNetVLLM

    2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.