Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TUNA, a new benchmark for fine-grained temporal video understanding, tests whether video-language models actually track camera motion, scene changes, actions, and object attributes over time, and finds that even top models score only…

desk verdict A substantial human-annotated benchmark whose central MCQ temporal-indispensability claim is internally contradicted by its own appendix. read the letter →

arxiv 2505.20124 v2 pith:CDVJHVQW submitted 2025-05-26 cs.CV cs.MM

Fanheng Kong , Jingyuan Zhang , Hongzhi Zhang , Shi Feng , Daling Wang , Linhao Yu , Xingguang Ji , Yu Tian
show 2 more authors
Victoria W. Fuzheng Zhang
This is my paper
classification cs.CVcs.MM
keywords videounderstandingbenchmarktemporaldensecaptioningquestionansweringcameramotionactionsequencemultimodallargelanguagemodelsevent-levelevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces TUNA, a benchmark for fine-grained temporal understanding of short, dense dynamic videos, built on a manually annotated dataset called TUNA-1K. It argues that existing video benchmarks either isolate individual elements like actions or bias toward long videos, entangling temporal ability with long-context modeling. TUNA instead requires models to understand four dynamic elements together—camera state, background scene, subject action, and object attribute—across chronologically ordered events. On this benchmark, the paper reports that even the best model, GPT-4o, reaches only 58.5 percent F1 on dense captioning and about 60 percent accuracy on multiple-choice questions, while open-source models match commercial ones on the question task but lag on captioning. The results point to consistent model weaknesses in camera-motion understanding and action-sequence reasoning.

What carries the argument

The central machinery is the three-level annotation structure of TUNA-1K: a narrative-level whole-video caption, an event-level chronological decomposition into temporal events, and an atomic-level set of visual elements, each typed as camera, scene, action, or attribute and weighted from 1 to 3 by importance. This structure powers TUNA-CAP's evaluation pipeline, which uses GPT-4o to split candidate captions into events, match them to reference events in temporal order, and classify each visual element's relationship to the candidate event as entailment, lack, or contradiction, then computes weighted precision, recall, and F1. The same element structure drives TUNA-MCQ's ten task types, which cover camera motion, camera transition, scene description, scene transition, action recognition, action sequence, action-subject matching, object recognition, object appearance, and object location, with a temporal-indispensability filter meant to reject questions answerable from a single frame.

What would settle it

A direct falsification would be to take the released TUNA-MCQ questions and evaluate models on single-frame inputs versus 16-frame inputs; if accuracy on one random frame matches or closely approaches the 16-frame accuracy on a substantial fraction of questions, the claim that answers require full video context collapses for those items. Additionally, inspecting the released construction code or instructions would settle whether the main-text rule or the Appendix C.2 rule was actually applied.

Watch

Extended reading notes

Core claim

The paper's central claim is that TUNA isolates and measures temporal understanding in video models more directly than previous benchmarks, through two complementary tasks. TUNA-CAP evaluates dense video captioning by splitting generated captions into events, matching them to reference events, and classifying each reference visual element as entailed, lacking, or contradicted by the candidate caption. TUNA-MCQ provides 1,432 multiple-choice questions that, according to the main text, specifically require full video context and cannot be answered from a single frame or limited frames. The evaluation of 21 models shows that all tested models struggle with camera motion (best accuracy 53.9 percent) and action sequences (best 67.6 percent), while scene description and object appearance are comparatively easier. The paper also reports that open-source models such as Qwen2-VL-72B and LLaVA-Video-72B match GPT-4o on the question task, yet commercial models lead clearly on dense captioning.

Load-bearing premise

The central QA claim rests on the temporal-indispensability filter actually removing every question that a single frame could answer; the main text says it does, while Appendix C.2 describes a rule that would keep such questions, so the filter's real behavior is the load-bearing uncertainty.

Editorial extensions

If this is right

  • If TUNA is a valid measure, model improvements should prioritize camera-motion perception and action-sequence reasoning, since these are the weakest and most consistent failure points across all tested models.
  • The strong human-correlation results (Kendall's tau 57.2, Spearman's rho 76.7, Pearson r 69.9) suggest that element-level, relationship-classification-based caption evaluation could replace n-gram and direct LLM scoring methods.
  • The finding that open-source models match commercial ones on TUNA-MCQ but lag on TUNA-CAP implies a specific gap in open-ended holistic video generation and description, not in discriminative temporal question answering.
  • The paper's frame-scaling results indicate that adding frames helps on complex videos up to a point, but going from 32 to 64 frames can hurt on the most complex videos, pointing to context-length and token-compression limits.
  • The observed performance drop as video complexity increases (more events or visual elements) suggests that future benchmarks should report complexity-stratified scores to avoid masking model weaknesses on hard temporal content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: The benchmark's central QA guarantee is testable by running each TUNA-MCQ question with only one randomly sampled frame as input; if a substantial subset is still answerable correctly, then the claim that answers require full video context is weaker than stated.
  • Editorial extension: The event-splitting and element-weighting approach could be transferred to other video domains, such as egocentric or long-form video, by segmenting videos into events first, although the paper itself does not claim this.
  • Editorial extension: The annotation scheme may also serve as a fine-grained temporal consistency metric for video generation models, since it decomposes a video into checkable camera, scene, action, and attribute elements over time.
  • Editorial extension: Readers should treat the temporal-indispensability guarantee as contingent on the main-text description of the filter; Appendix C.2 gives a different selection rule, and which rule was actually applied would need to be confirmed from the released data and code.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces TUNA-1K, a manually annotated dataset of 1,000 short, dense dynamic videos with narrative-level captions, event-level decompositions, and atomic visual elements (camera, scene, action, attribute) with importance weights. On top of this dataset, the authors build a two-task benchmark: TUNA-CAP, an automated captioning evaluation that splits model captions into events, matches them to reference events, and classifies element-level entailment/lack/contradiction via GPT-4o to compute weighted precision/recall/F1; and TUNA-MCQ, a set of 1,432 multiple-choice questions generated from error-prone points extracted by an LMM, refined by crowdsourcing, and filtered by a temporal-indispensability criterion. The paper evaluates 21 video LMMs, reporting that all models struggle with camera motion, action sequence, and multi-subject content, while open-source models are competitive on MCQ but lag on captioning.

Significance. If the benchmark works as claimed, it would be a valuable fine-grained temporal understanding benchmark: it explicitly targets camera state and scene transitions alongside actions and attributes, provides human-annotated ground truth, releases data and code, and includes a human-correlation study (Table 3) showing that the TUNA-CAP metric tracks human judgments better than existing captioning metrics. The interpretable element-level scoring and the multi-dimensional breakdown across dynamic elements and visual characteristics are useful contributions. The central validity claim, however, rests on the assertion that TUNA-MCQ questions require full video context; the internal contradiction between the main text and Appendix C.2 on the temporal-indispensability filter undermines that claim as written, and the captioning metric's inability to penalize ungrounded hallucinated content limits the interpretation of the reported precision scores. These issues are fixable but load-bearing.

major comments (3)
  1. [Limitations; Eq. (1)] The temporal-indispensability filter is defined in two mutually exclusive ways. Section 3.2.3 states that a question is 'temporal-indispensable only if it cannot be correctly answered using a single frame but requires n frames (default n=16) for accurate comprehension.' Appendix C.2 states the opposite: 'A question is deemed temporal-indispensable if it can be accurately answered using both a single frame and multiple frames,' operationalized as 'answered correctly by both 1-frame and 16-frame inputs.' If the appendix reflects the actual implementation, then single-frame-answerable questions are deliberately retained, and the central claim in Section 1 and §3.2.3 — that TUNA-MCQ 'answers cannot be derived from a single frame or limited frames' — is false for the benchmark as constructed. This directly affects the interpretation of every TUNA-MCQ result in Table 4. The authors must clarify which rule was used, correct the inconsistent text, and provide the actual filter statistics (e.g., how many questions were removed by the filter and how the retained set behaves under 1-frame versus 16-frame answering).
  2. [Limitations; Eq. (1)] The TUNA-CAP precision formula in Eq. (1) does not penalize hallucinated content that fails to match reference events, because event matching discards unmatched candidate events before relationship classification. The Limitations section concedes this: 'if the model outputs visual elements that fail to match the annotated events or elements, our method cannot assess their precision... even if this content contains substantial hallucinatory information, our method would be unable to provide a valid assessment.' Consequently, the statement in Section 4.2 that precision 'reflects the correctness of the content mentioned in the descriptions' is too strong; a model that emits long fabricated but irrelevant captions can receive the same precision as a concise accurate model. I recommend reporting an additional ungrounded-content ratio or hallucination rate that measures content not matched to any reference event, so that the precision metric cannot be gamed by verbosity.
  3. [§3.2.2, §4.2, Table 3] GPT-4o is used as the automatic judge for TUNA-CAP (event splitting, event matching, and relationship classification) and is also the best-scoring closed-source model in Table 2. This creates a risk that the evaluation favors GPT-4o's caption style, particularly in the entailment/contradiction judgments. The human-correlation study in Table 3 is based on only 40 videos (687 visual elements) and does not report judge agreement or a comparison against a second LLM judge. To establish that the ranking is not an artifact of judge self-preference, the authors should report inter-judge agreement between GPT-4o and another strong LLM (e.g., Claude or GPT-4.1) on a subsample, and ideally provide the correlation of GPT-4o's judgments with human judgments separately from the final metric computation.
minor comments (6)
  1. [Table 1 and §1] The sample count is inconsistent: Table 1 lists 2,432 samples for TUNA, but the text says TUNA-MCQ comprises 1,432 questions and TUNA-1K has 1,000 videos. Clarify whether 2,432 is the sum of captioning and MCQ samples, and state the exact counts per task.
  2. [Appendix B.1 and C.3] The paper mentions that 'a few samples (less than 5)' for Gemini in TUNA-CAP and 'less than 10' in TUNA-MCQ did not receive responses. Since scores are computed only on samples with responses, the exact denominators differ across models; please report the number of missing responses per model so that small differences in Table 4 are not over-interpreted.
  3. [Table 3] The human-correlation study uses 40 videos; please provide confidence intervals or significance tests beyond p-values, and describe how the 40 videos were sampled to ensure representativeness across the four visual characteristics and 12 domains.
  4. [Figure 26 prompt] The prompt template for MCQ generation contains formatting errors: the JSON example has '"correct_option", ""' instead of '"correct_option": ""', and the reference action-sequence example lists 'answer': '11' for a question about object-picking order, which appears to be a copy-paste error. Please correct these in the appendix.
  5. [Throughout] There are several typos: 'vidoe' in the Table 1 caption, 'Mulitmodal' in Appendix E, 'categorise' in Figure 23, and 'Temporally Dense' in section 2. A light proofread is needed.
  6. [Appendix C.2] The appendix says the filter is applied using LLaVA-Video-7B, but Section 3.2.3 does not state which model performs the filtering; please state the model and frame counts (1 vs 16) explicitly in the main text, since this is essential for reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

Self-definitional contradiction in TUNA-MCQ temporal-indispensability filter: Appendix C.2's rule retains single-frame-answerable questions, inverting §3.2.3's definition.

  1. self definitional [Section 3.2.3 (Multi-Choice QA Generation) vs Appendix C.2 (Quality Review)]
    "a question is considered temporal-indispensable only if it cannot be correctly answered using a single frame but requires n frames (default n=16) for accurate comprehension. ... A question is deemed temporal-indispensable if it can be accurately answered using both a single frame and multiple frames. Specifically, we deem the question to be temporal-indispensable if it can be answered correctly by both 1-frame and 16-frame inputs."

    Section 3.2.3 defines 'temporal-indispensable' as NOT answerable from a single frame, while Appendix C.2 defines and operationalizes it as answerable by BOTH 1-frame and 16-frame inputs. Since 'correctly by both' entails answerable from one frame, the appendix's selection rule is the logical negation of the main text's definition. If Appendix C.2 is the actual filter, then the 1,432 TUNA-MCQ questions are selected precisely for being single-frame-answerable, so the benchmark's load-bearing claim that answers 'cannot be derived from a single frame or limited frames' is false by construction. The reported 'temporal-indispensability' property is therefore self-contradictory and cannot support the claimed full-video-context requirement.

full rationale

The core TUNA-1K data is human-annotated and cross-reviewed, and TUNA-CAP's automatic metric is externally validated against human judgments (Table 3), so the model rankings are not fitted quantities and the benchmark construction is largely independent. No load-bearing self-citation or imported uniqueness theorem was found. The one central circular/definitional flaw is the TUNA-MCQ temporal-indispensability filter: Section 3.2.3 and Appendix C.2 give mutually exclusive operational definitions of the same property. If the appendix procedure was followed, the benchmark's central claim that TUNA-MCQ requires full video context is violated by construction; if it is a typo, the paper must state the correction and report the actual filter. The use of GPT-4o as the automatic judge while it is also the best-scoring evaluated model is a reviewer concern, but it is not a constructional circularity because GPT-4o's judgments are not fitted to its own score. The score reflects the self-definitional contradiction at the load-bearing validity step.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The benchmark's validity rests on annotation quality, the GPT-4o judge, frame sampling, and the temporal filter. None of these are derived; they are domain assumptions or hand-chosen parameters. The main free parameters are small in number but directly affect every reported score.

free parameters (4)
  • Element weights w_ij
    Annotator-assigned importance weights in {1,2,3} enter the precision, recall, and F1 equations in Section 3.2.2; they are hand-labeled heuristics, not learned or derived.
  • Temporal-indispensability frame count n = 16
    Hand-chosen default frame count used to filter TUNA-MCQ questions; the central claim that questions need temporal context depends on this threshold.
  • Default frame sampling count = 32
    Default 32 uniformly sampled frames, with per-model variations, determines all captioning and QA scores.
  • Error-prone point extraction frames = 8
    8 frames are used when asking a video LMM to find inconsistencies with ground-truth descriptions; a hand-chosen parameter of the MCQ construction pipeline.
assumptions (5)
  • domain assumption Human annotations in TUNA-1K are accurate and consistent ground truth
    No inter-annotator agreement statistics are reported; the full benchmark and all model scores rest on the quality of these annotations (Section 3.1, A.2.4).
  • domain assumption GPT-4o's event splitting, matching, and relationship classification are faithful proxies for human judgment on the full dataset
    TUNA-CAP metrics are computed by GPT-4o (Section 3.2.2), but human correlation is shown only on 40 videos and 687 elements (Appendix B.2.4), not across all 1,000 videos.
  • domain assumption Uniformly sampled frames capture all key events in every video
    The paper asserts that 32 frames, or model-specific sampling, are sufficient for videos up to 38 seconds (Appendix B.1); this is not verified per video and affects all reported scores.
  • standard math Standard weighted precision, recall, and F1 definitions apply
    Equations 1 to 3 in Section 3.2.2 use conventional formulas; no non-standard assumptions are introduced.
  • domain assumption The 1,000 videos selected from ten sources are representative of dense dynamic open-domain video
    Selection is biased toward Film (414 videos) and Daily Life (114), with stock footage and academic datasets filling the rest; representativeness is assumed, not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos." pith.science (2026). https://pith.science/paper/CDVJHVQW

@misc{pith2026250520124,
  author       = {Pith},
  title        = {Pith review of: TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CDVJHVQW}},
  note         = {Machine review of arXiv:2505.20124}
}
read the original abstract

Videos are unique in their integration of temporal elements, including camera, scene, action, and attribute, along with their dynamic relationships over time. However, existing benchmarks for video understanding often treat these properties separately or narrowly focus on specific aspects, overlooking the holistic nature of video content. To address this, we introduce TUNA, a temporal-oriented benchmark for fine-grained understanding on dense dynamic videos, with two complementary tasks: captioning and QA. Our TUNA features diverse video scenarios and dynamics, assisted by interpretable and robust evaluation criteria. We evaluate several leading models on our benchmark, providing fine-grained performance assessments across various dimensions. This evaluation reveals key challenges in video temporal understanding, such as limited action description, inadequate multi-subject understanding, and insensitivity to camera motion, offering valuable insights for improving video understanding models. The data and code are available at https://friedrichor.github.io/projects/TUNA.

Figures

Figures reproduced from arXiv: 2505.20124 by the authors.

Figure 1
Figure 1. Performance of several advanced models on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of TUNA-1K construction. We collect and filter high-quality, short videos featuring dynamic temporal content from various sources. Each video is then categorized based on its visual characteristics and domain. Trained annotators provide temporally dense descriptions, followed by cross-validation. Video experts continuously review annotations, guiding annotators to refine their works, thus ensuring quality o… view at source ↗
Figure 3
Figure 3. An instance in TUNA-1K consists of three levels of description: (a) an overall caption (Narrative-level), (b) a chronological sequence of events (Event-level), and (c) fine-grained visual elements (Atomic-level) along with their types and weights. A complete sample can be found in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (23 more)
Figure 4
Figure 4. Figure 4: Overview of the evaluation workflow for TUNA-CAP. We first split candidate caption into multiple events and match them to reference events in TUNA-1K. Then we discard the mismatched events (useless content or inconsistent chronology), and connect the matched candidate …
Figure 5
Figure 5. Figure 5: Performance comparison of different input frames with different video complexity for models trained in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison across different video complexities. Video Complexity. We partition TUNA-1K based on the number of events and visual elements to investigate how increasing video complexity af￾fects model performance. As shown in [PITH_FULL_IMAGE:figures/full_fi…
Figure 7
Figure 7. Figure 7: The sample distribution of TUNA-1K, videos covering 4 visual characteristics and 12 domains. As shown in [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Sample distribution of domains in the TUNA￾1K, covering 12 domains. There are 12 domains contained in TUNA-1K, including: (1) Film, (2) Daily Life, (3) Cooking, (4) Sports Activity, (5) Driving, (6) Animals & Pets, (7) Natural Landscape, (8) Cityscape, (9) Urban Activi…
Figure 9
Figure 9. Figure 9: Visual statistics of the number of events and the number of visual elements in T [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: The whole performance comparison on TUNA-CAP across different video complexities. by type and weighted by importance on a scale of 1-3. The types include camera, scene, action, and attribute. Quality Review. For quality assurance, cross￾inspections are performed betwe…
Figure 11
Figure 11. Figure 11: Performance comparison across different number of input frames. As shown in [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Sample distribution of task types in the [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Sample distribution of correct option in the [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Several video understanding benchmark examples and analysis. [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: A detailed example in TUNA-1K [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Several examples in TUNA-MCQ, involving Camera Motion, Camera Transition, Scene Description and Scene Transition tasks [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Several examples in TUNA-MCQ, involving Action Recognition, Action Sequence, and Action-Subject Matching tasks [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: Several examples in TUNA-MCQ, involving Object Recognition, Object Appearance, and Object Location tasks [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]
Figure 19
Figure 19. Figure 19: The default prompt used for the TUNA-CAP experiments in Section 4.2. Given a chronological video caption, split it into multiple chronologically evolving events. All events spliced together should be equal to the original caption. Video Caption: {model_generated_capti…
Figure 20
Figure 20. Figure 20: The prompt used to split events for the T [PITH_FULL_IMAGE:figures/full_fig_p027_20.png]
Figure 21
Figure 21. Figure 21: The prompt used to match events for the T [PITH_FULL_IMAGE:figures/full_fig_p027_21.png]
Figure 22
Figure 22. Figure 22: The prompt used to classify relationships for the T [PITH_FULL_IMAGE:figures/full_fig_p028_22.png]
Figure 23
Figure 23. Figure 23: The prompt used to classify videos for the T [PITH_FULL_IMAGE:figures/full_fig_p028_23.png]
Figure 24
Figure 24. Figure 24: The default prompt used for the TUNA-MCQ experiments in Section 4.3. You are an AI visual assistant specialized in analyzing videos. Given 8 frames uniformly sampled from a video clip and a human-annotated video description, your task is to extract the elements of the…
Figure 25
Figure 25. Figure 25: The prompt used to generate error-prone points for the T [PITH_FULL_IMAGE:figures/full_fig_p029_25.png]
Figure 26
Figure 26. Figure 26: The prompt used to generate multi-choice QAs for the T [PITH_FULL_IMAGE:figures/full_fig_p030_26.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval

    cs.CV 2025-05 conditional novelty 6.0 of 10

    UNITE combines curated multimodal training data and a modality-masked contrastive loss to achieve strong retrieval performance across text, image, and video tasks.

Reference graph

Works this paper leans on

71 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, et al. 2024. Understanding alignment in multimodal llms: A comprehensive study. arXiv preprint arXiv:2407.02477

    arXiv 2024

  2. [2]

    Hidehisa Arai, Keita Miwa, Kento Sasaki, Yu Yamaguchi, Kohei Watanabe, Shunsuke Aoki, and Issei Yamamoto. 2024. Covla: Comprehensive vision-language-action dataset for autonomous driving. arXiv preprint arXiv:2408.10845

    arXiv 2024

  3. [3]

    Satanjeev Banerjee and Alon Lavie. 2005. W05-0909 METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . pages 65--72, Ann Arbor, Michigan

    work page 2005

  4. [4]

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. 2024. https://openai.com/research/video-generation-models-as-world-simulators Video generation models as world simulators

    2024

  5. [5]

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. 2024. https://doi.org/10.18653/v1/2024.findings-acl.807 The revolution of multimodal large language models: A survey . pages 13590--13618, Bangkok, Thailand and virtual meeting

    doi:10.18653/v1/2024.findings-acl.807 2024

  6. [6]

    Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, et al. 2024. Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models. arXiv preprint arXiv:2410.10818

    arXiv 2024

  7. [7]

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng-Neng Hwang, Saining Xie, and Christopher D Manning. 2024. Auroracap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.03051

    arXiv 2024

  8. [8]

    David Chan, Suzanne Petryk, Joseph Gonzalez, Trevor Darrell, and John Canny. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.841 CLAIR : Evaluating image captions with large language models . pages 13638--13646, Singapore

    doi:10.18653/v1/2023.emnlp-main.841 2023

Show all 71 references
  1. [9]

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. 2024 a . Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325

    2024 arXiv

  2. [10]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024 b . How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821

    2024 arXiv

  3. [11]

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476

    2024 arXiv

  4. [12]

    Andrei de Souza Inácio and Heitor Silvério Lopes. 2023. https://doi.org/10.1016/j.mlwa.2023.100488 Evaluation metrics for video captioning: A survey . Machine Learning with Applications, 13:100488

    2023

  5. [13]

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. 2024. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37:89098--89124

    2024

  6. [14]

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. 2024. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075

    2024 arXiv

  7. [15]

    Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. 2024. Miradata: A large-scale video dataset with long durations and structured captions. arXiv preprint arXiv:2407.06358

    2024 arXiv

  8. [16]

    Fanheng Kong, Jingyuan Zhang, Yahui Liu, Hongzhi Zhang, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Victoria W, Fuzheng Zhang, and Guorui Zhou. 2025. Modality curation: Building universal embeddings for advanced multimodal information retrieval. arXiv preprint arXiv:2505.19650

    2025 arXiv

  9. [17]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  10. [18]

    Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. 2024 b . Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends in Computer Graphics and Vision , 16(1-2):1--214

    2024

  11. [19]

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024 c . Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895

    2024 arXiv

  12. [20]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR

    2023

  13. [21]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. 2024 d . Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

    2024

  14. [22]

    Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. 2024 e . Videovista: A versatile benchmark for video understanding and reasoning. arXiv preprint arXiv:2406.11303

    2024 arXiv

  15. [23]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122

    2023 arXiv

  16. [24]

    Chin-Yew Lin. 2004. W04-1013 ROUGE : A package for automatic evaluation of summaries . pages 74--81, Barcelona, Spain

    2004

  17. [25]

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689--26699

    2024

  18. [26]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306

    2024

  19. [27]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 b . Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  20. [28]

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. 2024 c . Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542

    2024 arXiv

  21. [29]

    Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. 2025. Llava-plus: Learning to use tools for creating multimodal agents. In European Conference on Computer Vision, pages 126--142. Springer

    2025

  22. [30]

    Tingkai Liu, Yunzhe Tao, Haogeng Liu, Qihang Fang, Ding Zhou, Huaibo Huang, Ran He, and Hongxia Yang. 2024 d . https://doi.org/10.18653/v1/2024.acl-long.772 D e VA n: Dense video annotation for video-language models . pages 14305--14321, Bangkok, Thailand

    2024 doi

  23. [31]

    Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Chang Wen Chen, and Ying Shan. 2024 e . E.t. bench: Towards open-ended event-level video-language understanding. In Neural Information Processing Systems (NeurIPS)

    2024

  24. [32]

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. 2024 f . Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476

    2024 arXiv

  25. [33]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2023. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424

    2023 arXiv

  26. [34]

    Neelu Madan, Andreas M gelmose, Rajat Modi, Yogesh S Rawat, and Thomas B Moeslund. 2024. Foundation models for video understanding: A survey. arXiv preprint arXiv:2405.03770

    2024 arXiv

  27. [35]

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212--46244

    2023

  28. [36]

    mixkit. 2023. mixkit. https://mixkit.com/videos/

    2023

  29. [37]

    Thong Nguyen, Yi Bin, Junbin Xiao, Leigang Qu, Yicong Li, Jay Zhangjie Wu, Cong-Duy Nguyen, See-Kiong Ng, and Luu Anh Tuan. 2024. Video-language understanding: A survey from model architecture, model training, and data perspectives. arXiv preprint arXiv:2406.05615

    2024 arXiv

  30. [38]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Hello gpt-4o

    2024

  31. [39]

    Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. 2023. Kosmos-g: Generating images in context with multimodal large language models. arXiv preprint arXiv:2310.02992

    2023 arXiv

  32. [40]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . pages 311--318, Philadelphia, Pennsylvania, USA

    2002

  33. [41]

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. 2024. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processin...

    2024

  34. [42]

    Pexels. 2023. Pexels. https://www.pexels.com/videos/

    2023

  35. [43]

    pixabay. 2023. pixabay. https://pixabay.com/videos/

    2023

  36. [44]

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. 2024. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720

    2024 arXiv

  37. [45]

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

    2024 arXiv

  38. [46]

    Darshana Saravanan, Darshan Singh, Varun Gupta, Zeeshan Khan, Vineet Gandhi, and Makarand Tapaswi. 2024. Velociti: Can video-language models bind semantic concepts through time? arXiv preprint arXiv:2406.10889

    2024 arXiv

  39. [47]

    Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. 2024. Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models. arXiv preprint arXiv:2410.23266

    2024 arXiv

  40. [48]

    Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, and Hao Li. 2024. Vidgen-1m: A large-scale dataset for text-to-video generation. arXiv preprint arXiv:2408.02629

    2024 arXiv

  41. [49]

    Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. 2023. Video understanding with large language models: A survey. arXiv preprint arXiv:2312.17432

    2023

  42. [50]

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566--4575

    2015

  43. [51]

    Jiawei Wang, Liping Yuan, and Yuchen Zhang. 2024 a . Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634

    2024 arXiv

  44. [52]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024 b . Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  45. [53]

    Yuxuan Wang, Difei Gao, Licheng Yu, Weixian Lei, Matt Feiszli, and Mike Zheng Shou. 2022. Geb+: A benchmark for generic event boundary captioning, grounding and retrieval. In European Conference on Computer Vision, pages 709--725. Springer

    2022

  46. [54]

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777--9786

    2021

  47. [55]

    Tianwei Xiong, Yuqing Wang, Daquan Zhou, Zhijie Lin, Jiashi Feng, and Xihui Liu. 2024. Lvd-2m: A long-take video dataset with temporally dense captions. arXiv preprint arXiv:2410.10816

    2024 arXiv

  48. [56]

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. 2024. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994

    2024 arXiv

  49. [57]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800

    2024 arXiv

  50. [58]

    Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. 2024 a . Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601

    2024 arXiv

  51. [59]

    Jingyuan Zhang, Hongzhi Zhang, Zhou Haonan, Chenxi Sun, Jiakang Wang, Fanheng Kong, Yahui Liu, Qi Wang, Fuzheng Zhang, et al. 2025. Data metabolism: An efficient data design schema for vision language model. arXiv preprint arXiv:2504.12316

    2025 arXiv

  52. [60]

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. 2024 b . Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320

    2024 arXiv

  53. [61]

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024 c . Long context transfer from language to vision. arXiv preprint arXiv:2406.16852

    2024 arXiv

  54. [62]

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. 2024 d . Direct preference optimization of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01258

    2024 arXiv

  55. [63]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  56. [64]

    Yiqun Zhang, Fanheng Kong, Peidong Wang, Shuang Sun, SWangLing SWangLing, Shi Feng, Daling Wang, Yifei Zhang, and Kaisong Song. 2024 e . https://doi.org/10.18653/v1/2024.acl-long.417 STICKERCONV : Generating multimodal empathetic responses from scratch . pages 7707--7733, Bang...

    2024 doi

  57. [65]

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024 f . Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713

    2024 arXiv

  58. [66]

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923

    2023 arXiv

  59. [67]

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. 2024. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264

    2024 arXiv

  60. [68]

    Luowei Zhou, Chenliang Xu, and Jason Corso. 2018. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32

    2018

  61. [69]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

  62. [70]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  63. [71]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.