Pith. sign in

REVIEW 3 major objections 6 minor 51 references

RTime-QA: A Benchmark for Atomic Temporal Event Understanding in Large Multi-modal Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RTime-QA measures whether video models grasp atomic temporal events, and finds they mostly do not.

desk verdict The benchmark part is worth your time; the instruction-tuning claim needs external validation before you trust the 65.9 number. read the letter →

arxiv 2505.19125 v1 pith:HMN3JRCE submitted 2025-05-25 cs.CV

Yuqi Liu , Qin Jin , Tianyuan Qu , Xuan Liu , Yang Du , Bei Yu , Jiaya Jia This is my paper
classification cs.CV
keywords atomictemporaleventunderstandingnegativesamplesvideoquestionansweringlargemultimodalmodelsinstructiontuningbenchmarkstrictaccuracyorder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that existing video-language benchmarks can be answered from static appearances, so they do not measure whether large multimodal models actually track temporal order. To fix this, the authors build RTime-QA, 822 human-annotated multiple-choice questions where each video of an atomic temporal event is paired with a correct description and a temporally opposite distractor that is visually similar. On this benchmark the best models score only 34.6–38.7 strict accuracy against 97.3 for humans, while image-only models fall near random. The paper also contributes RTime-IT, a 14,096-sample instruction-tuning set built with the same temporal-negative design, and reports that fine-tuning Qwen2-VL on it raises strict accuracy to 65.9. The central claim is that atomic temporal events are a distinct, currently unsolved capability that can be isolated and improved with paired temporal negatives.

What carries the argument

The load-bearing device is the temporal negative pair $(T, \bar{T})$: two concise captions that differ only in temporal semantics while describing nearly identical spatial scenes. Each benchmark item is built from a quadruple $(V, \bar{V}, T, \bar{T})$ where $V$ and $\bar{V}$ are visually similar videos with opposite temporal events, and the strict-accuracy metric only credits a model that gets both directions right. This design removes the static-image shortcut that the paper says lets image-only models ace older video benchmarks. The atomicity requirement—brief, single-event captions averaging about six words—forces the model to commit to the specific temporal progression in the clip.

What would settle it

Fine-tune a model on RTime-IT and evaluate it on a held-out temporal-negative QA set built from a different video source with independent annotations and paraphrased options; if strict accuracy collapses back toward the zero-shot level, the reported improvement is mostly template or distribution overfitting, not general temporal understanding.

Watch

Extended reading notes

Core claim

RTime-QA's organizing unit is the triplet $(V, T, \bar{T})$: a video $V$ depicting an atomic temporal event—a brief event whose identity is fixed by temporal progression rather than static appearance—a correct short description $T$, and a temporally negative description $\bar{T}$ that shares the same objects and static appearance but describes the opposite event, such as folding versus unfolding a chair or upward versus downward motion. Each question is a forced choice between $T$ and $\bar{T}$, and the strict-accuracy metric requires a model to answer correctly for both $V$ and $\bar{V}$, so always choosing one description is counted as failure. The paper reports that most evaluated large multimodal models score below random choice on strict accuracy, with Qwen2-VL at 34.6 and Qwen2.5-VL at 38.7 against 97.3 for humans, and that training on RTime-IT improves Qwen2-VL's strict accuracy to 65.9 and accuracy from 65.9 to 77.9. This is presented as evidence that current models lack robust atomic temporal event understanding and that instruction tuning explicitly structured around temporal negatives can substantially improve it.

Load-bearing premise

The RTime-IT improvement is claimed as a gain in general temporal understanding, but the only test used to demonstrate it, RTime-QA, is built from the same source dataset and the same annotation template as RTime-IT, so some or all of the gain could be matching the benchmark's distribution rather than learning temporal semantics generally.

Editorial extensions

If this is right

  • Modern video large multimodal models are far from human-level on atomic temporal events, with the best reported models scoring 34.6–38.7 strict accuracy versus 97.3 for humans.
  • Models trained only on images perform worst, supporting the benchmark's claim that single-frame shortcuts are not enough here.
  • Frame count matters: Qwen2-VL improves from 5.1 with 2 frames to 34.6 with 32 frames, unlike older benchmarks where extra frames gave little gain.
  • Instruction tuning on temporal-negative pairs is a promising lever: Qwen2-VL's strict accuracy nearly doubles after training on RTime-IT.
  • Publicly available video training data may lack enough temporal emphasis, since models trained with private video data lead the table.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because RTime's original captions are long descriptions, RTime-QA is essentially testing how well models perform with minimal temporal contrast; we would expect performance on the same videos to rise if the model is also given the longer caption as context, which the paper does not test.
  • The large drop from accuracy to strict accuracy suggests many models pick whichever caption is more plausible regardless of the video; an independent diagnostic could report the share of same-answer errors to quantify this shortcut directly.
  • If RTime-IT's benefit transfers, it should appear on other temporal video benchmarks and in retrieval accuracy on the source corpus; that transfer test is absent from the paper.
  • The paper's own footnote calls the project work in progress, so the 65.9 figure is a first snapshot rather than a settled endpoint.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper presents RTime-QA, a multiple-choice video-language benchmark focused on atomic temporal event understanding. The benchmark is constructed from the authors' prior RTime dataset: 822 questions, each pairing a video with a correct description and a temporally opposite distractor, with all annotations human-verified. The paper also introduces RTime-IT, a 14,096-sample instruction-tuning dataset with the same structure, and evaluates eight LMMs plus five vision-language alignment models. The main empirical results are that state-of-the-art Qwen2-VL achieves only 34.6 strict-ACC (vs. 97.3 for humans), that increasing the number of sampled frames improves Qwen2-VL, and that fine-tuning Qwen2-VL on RTime-IT raises strict-ACC from 34.6 to 65.9. The paper concludes that RTime-QA is a challenging temporal benchmark and that RTime-IT effectively improves LMM temporal understanding.

Significance. The benchmark addresses a genuine gap: many existing video QA benchmarks can be solved by image-only models, and the design here forces a model to distinguish temporally opposite descriptions. The evidence that RTime-QA is challenging is reasonably strong—image-centric LLaVA1.5 is near chance, the best LMMs are far below human performance, and performance scales with the number of frames. The release of the dataset and the paired-question Strict-ACC protocol are useful contributions. However, the claim that RTime-IT improves general temporal understanding is not yet supported, because the instruction set and the evaluation set are drawn from the same source distribution and share the same template; this is a load-bearing weakness that requires additional external evaluation and control experiments.

major comments (3)
  1. [Section 3.4, Table 5] The claim that RTime-IT 'effectively enhance[s] LMMs' capacity in temporal understanding' is not established because RTime-IT and RTime-QA are constructed from the same source dataset (RTime) using the same annotation process and the same question template ('Which sentence accurately describes ...? A ... B ... Answer:'). Excluding overlapping videos removes video-level leakage but does not remove template-level, annotation-style, or source-distribution leakage. Training for 6 epochs on 14,096 such samples could memorize these regularities, including answer-position biases, sentence-length/style cues, and the specific temporal contrast patterns, which then transfer directly to the 822 RTime-QA items. To support the transfer claim, the authors should evaluate the fine-tuned model on independent temporal benchmarks (e.g., VITATECS, TemporalBench, Perception Test, MVBench, Next-QA) and include a control condition, such as fine-tuning on RTime-IT with shuffled answer options or on a similarly sized instruction set with a different template. Without such controls, the 34.6-to-65.9 improvement cannot be attributed to general temporal understanding rather than overfitting to the benchmark distribution.
  2. [Section 4.1, Table 2] The Strict-ACC metric and the 'below random' results need a position-bias analysis. Because each quadruple yields two questions with opposite correct letters, a model that always picks option A will obtain around 50% ACC and near 0% Strict-ACC, explaining the very low Strict-ACC numbers for models like LLaVA1.5 (3.9 vs. random 26.8). The paper does not report whether the correct answer is balanced across A and B in the benchmark, nor whether models exhibit a systematic letter preference. Since Strict-ACC is the central metric used to establish that RTime-QA is challenging, the authors should either rotate option order in the benchmark or report accuracy under permuted options, and show that the gap to random persists.
  3. [Section 3.2] The paper asserts that the benchmark is 'carefully-curated' and that annotators excluded quadruples where the event could be inferred from a single static image, but it provides no inter-annotator agreement statistics and no direct verification that the final 822 questions are not solvable from spatial appearance alone. The single-frame and few-frame results for Qwen2-VL in Table 3 are suggestive, but a dedicated single-frame or image-only baseline on the final benchmark (e.g., evaluating a strong image-based model or human annotators with one frame) would directly validate the central 'temporal-only' design claim. Reporting these statistics and controls would also make the quality-control process reproducible.
minor comments (6)
  1. [Section 4.2, Table 5] The number 65.9 appears both as the zero-shot ACC of Qwen2-VL in Table 2 and as the fine-tuned Strict-ACC in Table 5, which is confusing; please disambiguate the two uses in the abstract and Section 4.2.
  2. [Section 3.2, Table 1] The paper should clarify the relation between 822 questions and the underlying quadruples (V, ¯V, T, ¯T): if each quadruple yields two questions, then there are 411 quadruples, and Strict-ACC is computed per quadruple rather than per question. Table 1 should report exact counts instead of '0.8K'.
  3. [Section 4.1] The evaluation protocol is underspecified: the paper does not state the number of frames used for each model (other than Qwen2-VL in Table 3), the exact prompt template, the decoding parameters, or how many runs were averaged. These details are necessary for reproducibility of Tables 2–5.
  4. [Table 2] The human evaluation protocol is not described: the paper does not specify the number of human annotators, their instructions, or whether they viewed the same video and option format as the models. This information is needed to interpret the 97.3 human strict-ACC figure.
  5. [Section 3.4] For RTime-IT, the paper only states that videos do not overlap with RTime-QA; it does not say whether RTime-IT videos were also filtered against WebVid, VideoChatGPT, or other public training corpora, which is relevant to potential leakage when fine-tuning a model like Qwen2-VL.
  6. [Throughout] There are several typos and formatting issues, including 'Evalutation' in the Section 4.1 heading, 'epoches' in Section 4.2, and inconsistent spacing in model names such as 'LLaV A' and 'Vide-oLLaV A'. The footnote 'Work in progress' should also be removed for a formal submission.

Circularity Check

1 steps flagged · score 5.0 of 10

RTime-IT's effectiveness claim is supported only by a same-source, same-template evaluation; the 34.6-to-65.9 gain may reflect in-distribution fitting rather than general temporal understanding.

  1. fitted input called prediction [Section 3.4 and Section 4.2, Table 5]
    "The annotation process for short-sentence instructions is similar to that used in RTime-QA. We begin by selecting videos, denoted as V, which contain temporal negative samples ¯V and do not overlap with RTime-QA. Using the video annotation process outlined in Section 3.2, we generate question triples ... formulated into instructional data as follows: ⟨V⟩ Which sentence accurately describes the events happened in the video? A. ⟨T⟩ B. ⟨¯T⟩ Answer: A'. ..."

    RTime-QA and RTime-IT are both derived from the authors' prior RTime dataset and use the same multiple-choice template and the same temporal-negative annotation process. The only safeguard stated is that RTime-IT videos 'do not overlap with RTime-QA'; the shared source distribution, template, caption style, and annotation pipeline are not removed. Therefore the 34.6-to-65.9 Strict-ACC gain in Table 5 measures how well Qwen2-VL fits the RTime-derived instruction distribution, not an independently established gain in general temporal understanding.

full rationale

The benchmark-difficulty claim has independent content: human performance is 97.3 Strict-ACC while the best model reaches only 34.6, and frame-number scaling shows increasing frames helps, which is a self-contained empirical result. The RTime citation is a data source rather than a theorem, so the benchmark construction is not circular by itself. However, the RTime-IT effectiveness result is the paper's central functional claim, and it is evaluated only on RTime-QA, which shares RTime as its source, the exact multiple-choice phrasing, and the temporal-negative annotation procedure with RTime-IT. The stated exclusion of overlapping videos removes item-level leakage but not distribution-level leakage. Consequently, the Table 5 improvement is best interpreted as in-distribution fine-tuning gain, not as demonstrated generalization to temporal understanding generally. This is a partial circularity: the evidence for the capability claim and the training data come from the same pipeline, so the conclusion reduces to fitting the authors' own RTime-derived distribution.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. Its load-bearing assumptions concern the validity of the source RTime temporal negative pairs, the success of the single-frame filter, the strict-ACC metric's validity, and the distribution independence of RTime-IT and RTime-QA.

assumptions (4)
  • domain assumption The RTime source dataset's temporal negative pairs (V, \bar V) are valid: \bar V depicts the temporally opposite event with largely identical static appearance.
    The entire benchmark inherits this property from Du et al. (2024); if violated, the benchmark would not measure temporal distinctions. Invoked in Section 3.1's choice of RTime as the primary data source.
  • domain assumption The human annotation filtering (guideline 3 in Section 3.2) successfully removes all samples where the correct choice can be inferred from a single static image.
    This filter is the safeguard that makes RTime-QA a temporal benchmark rather than a spatial one; it relies on subjective annotator judgment with no reported inter-annotator agreement.
  • domain assumption The paired-question strict-ACC metric is a valid measure of genuine temporal comprehension: a model must correctly answer both (V, T, \bar T) and (\bar V, T, \bar T).
    The metric is reasonable, but it assumes the two questions are equally answerable and that the correct answers are unambiguous for a model that truly understands the videos.
  • domain assumption RTime-IT training data and RTime-QA test data are sufficiently disjoint in distribution that improvement on RTime-QA after fine-tuning on RTime-IT indicates general temporal understanding gains.
    Both datasets are derived from RTime using the same annotation template; the paper only excludes exact video overlap, not distribution overlap. This is the weakest assumption for the RTime-IT effectiveness claim (Section 3.4, Table 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of RTime-QA: A Benchmark for Atomic Temporal Event Understanding in Large Multi-modal Models." pith.science (2026). https://pith.science/paper/HMN3JRCE

@misc{pith2026250519125,
  author       = {Pith},
  title        = {Pith review of: RTime-QA: A Benchmark for Atomic Temporal Event Understanding in Large Multi-modal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HMN3JRCE}},
  note         = {Machine review of arXiv:2505.19125}
}
read the original abstract

Understanding accurate atomic temporal event is essential for video comprehension. However, current video-language benchmarks often fall short to evaluate Large Multi-modal Models' (LMMs) temporal event understanding capabilities, as they can be effectively addressed using image-language models. In this paper, we introduce RTime-QA, a novel benchmark specifically designed to assess the atomic temporal event understanding ability of LMMs. RTime-QA comprises 822 high-quality, carefully-curated video-text questions, each meticulously annotated by human experts. Each question features a video depicting an atomic temporal event, paired with both correct answers and temporal negative descriptions, specifically designed to evaluate temporal understanding. To advance LMMs' temporal event understanding ability, we further introduce RTime-IT, a 14k instruction-tuning dataset that employs a similar annotation process as RTime-QA. Extensive experimental analysis demonstrates that RTime-QA presents a significant challenge for LMMs: the state-of-the-art model Qwen2-VL achieves only 34.6 on strict-ACC metric, substantially lagging behind human performance. Furthermore, our experiments reveal that RTime-IT effectively enhance LMMs' capacity in temporal understanding. By fine-tuning on RTime-IT, our Qwen2-VL achieves 65.9 on RTime-QA.

Figures

Figures reproduced from arXiv: 2505.19125 by the authors.

Figure 1
Figure 1. Although the two videos share identical spa [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The annotation pipeline of RTime-QA. We start by generating reference sentences with commercial LLMs. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples of RTime-QA. The test samples in RTime-QA can be categorized into four types. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Zero-shot response from different LMMs. 5 Conclusion We introduce RTime-QA, a novel benchmark de￾signed to assess the temporal event understanding capabilities of LMMs. Through careful selection and annotation, we formulate 822 multiple-choice QA in RTime-QA, each feat…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 17 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Anthropic. 2024. Claude. https://www.anthropic.com/claude

    work page 2024

  5. [5]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

    arXiv 2023

  6. [6]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  7. [7]

    Max Bain, Arsha Nagrani, G \"u l Varol, and Andrew Zisserman. 2021. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1728--1738

    work page 2021

  8. [8]

    Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, et al. 2024. Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models. arXiv preprint arXiv:2410.10818

    arXiv 2024

Show all 51 references
  1. [9]

    David Chen and William B Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190--200

    2011

  2. [10]

    Yang Du, Yuqi Liu, and Qin Jin. 2024. Reversed in time: A novel temporal-emphasized benchmark for cross-modal video-text retrieval. In ACM Multimedia 2024

    2024

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  4. [12]

    Han Fang, Pengfei Xiong, Luhui Xu, and Wenhan Luo. 2022. Transferring image-clip to video-text retrieval via temporal relations. IEEE Transactions on Multimedia, 25:7772--7785

    2022

  5. [13]

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. 2024. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075

    2024 arXiv

  6. [14]

    Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala. 2021. Agqa: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11287--11297

    2021

  7. [15]

    Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. 2023. Video-text as game players: Hierarchical banzhaf interaction for cross-modal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2023

  8. [16]

    Wonkyun Kim, Changin Choi, Wonseok Lee, and Wonjong Rhee. 2024. An image grid can be worth a video: Zero-shot video question answering using a vlm. arXiv preprint arXiv:2403.18406

    2024 arXiv

  9. [17]

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pages 706--715

    2017

  10. [18]

    Jie Lei, Tamara L Berg, and Mohit Bansal. 2022. Revealing single frame bias for video-and-language learning. arXiv preprint arXiv:2206.03428

    2022 arXiv

  11. [19]

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024 a . Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299--13308

    2024

  12. [20]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888--12900. PMLR

    2022

  13. [21]

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023 a . Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355

    2023 arXiv

  14. [22]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. 2024 b . Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

    2024

  15. [23]

    Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. 2023 b . Unmasked teacher: Towards training-efficient video foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19948--19960

    2023

  16. [24]

    Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun, and Lu Hou. 2023 c . Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models. arXiv preprint arXiv:2311.17404

    2023 arXiv

  17. [25]

    Yanwei Li, Chengyao Wang, and Jiaya Jia. 2025. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323--340. Springer

    2025

  18. [26]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122

    2023 arXiv

  19. [27]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. 2024 a . Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434

    2024 arXiv

  20. [28]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 b . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306

    2024

  21. [29]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 c . Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  22. [30]

    Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. 2022. Ts2-net: Token shift and selection transformer for text-video retrieval. In European conference on computer vision, pages 319--335. Springer

    2022

  23. [31]

    Yuqi Liu, Luhui Xu, Pengfei Xiong, and Qin Jin. 2023. Token mixing: parameter-efficient transfer learning from image-language to video-language. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1781--1789

    2023

  24. [32]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2024. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)

    2024

  25. [33]

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212--46244

    2023

  26. [34]

    Meta. 2024. Llama3.2: Revolutionizing edge ai and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices

    2024

  27. [35]

    OpenAI. 2022. Gpt3.5. https://openai.com/index/gpt-3-5-turbo-fine-tuning-and-api-updates/

    2022

  28. [36]

    OpenBMB. 2024. Minicpm-v 2.6: A gpt-4v level mllm for single image, multi image and video on your phone. https://github.com/OpenBMB/MiniCPM-V?tab=readme-ov-file

    2024

  29. [37]

    OpenGVLab. 2024. Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy. https://internvl.github.io/blog/2024-07-02-InternVL-2.0/

    2024

  30. [38]

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. 2024. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processin...

    2024

  31. [39]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  32. [40]

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2024. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36

    2024

  33. [41]

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355

    2023 arXiv

  34. [42]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024 a . Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  35. [43]

    Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4581--4591

    2019

  36. [44]

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. 2024 b . Internvideo2: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403.15377

    2024 arXiv

  37. [45]

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671

    2023 arXiv

  38. [46]

    Wenhao Wu. 2024. Freeva: Offline mllm as training-free video assistant. arXiv preprint arXiv:2405.07798

    2024 arXiv

  39. [47]

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777--9786

    2021

  40. [48]

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288--5296

    2016

  41. [49]

    Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858

    2023 arXiv

  42. [50]

    Yuanhan Zhang, Bo Li, Haotian Liu, Yong Jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. 2024. Llava-next: A strong zero-shot video understanding model. https://llava-vl.github.io/blog/2024-04-30-llava-next-video/

    2024

  43. [51]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.