Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Video captions now get fine-grained grading across 21 dimensions

desk verdict Valuable fine-grained video caption benchmark with a sensible evaluation pipeline, but the QA ground truth is under-audited and the Gemini circularity is real. read the letter →

arxiv 2505.23484 v1 pith:ZYZLYEHA submitted 2025-05-29 cs.CV

Shi-Xue Zhang , Hongfa Wang , Duojun Huang , Xin Li , Xiaobin Zhu , Xu-Cheng Yin This is my paper
classification cs.CV
keywords videocaptionevaluationfine-grainedbenchmarkvision-languagemodelstext-to-videogenerationquestion-answeringunderstandingqualitymetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VCapsBench is a benchmark for judging how good a machine-written video caption is, built around 5,677 videos and 109,796 yes/no/unanswerable questions about what actually appears in each clip. The questions are organized into 21 dimensions that matter for text-to-video generation, such as camera movement, shot type, object position, relative position, blur, and lighting. The paper proposes that caption quality can be reduced to three numbers: Accuracy, Inconsistency Rate, and Coverage Rate, computed by an LLM that reads the caption and tries to answer each question from it. If the benchmark works, it gives caption generators and text-to-video systems a cheap, fine-grained way to see exactly which kinds of detail their captions get right, get wrong, or omit. It also provides a reference ranking of ten current vision-language models on these dimensions.

What carries the argument

The object that carries the argument is the ternary QA pair: each question has a ground-truth answer of 'yes', 'no', or 'unanswerable', and the evaluator LLM must assign one of these to the caption under test. This turns caption evaluation into a series of checkable factual claims rather than a similarity score. Around it sit the 21 dimensions that organize which claims matter (content and entity, visuals and composition, color and lighting, cinematography and atmosphere), the three derived metrics, and the protocol of querying the evaluator three times per caption and taking a consensus to reduce output variability.

What would settle it

Take a random sample of, say, 300 videos from the benchmark, have two independent annotators re-verify every QA pair against the raw video, and measure pairwise agreement and the share of answers that change; if a substantial fraction (for example, more than a few percent) of answers is disputed or corrected, the benchmark's ground truth is not stable enough to support the claimed rankings.

Watch

Extended reading notes

Core claim

The paper claims that video caption quality can be reliably evaluated by turning each caption into answers on a fixed bank of video-specific questions and scoring those answers. For every video, corrected QA pairs define what is true, false, and undeterminable from the video; a caption is then judged by whether an LLM reading only the caption answers each question correctly. From those judgments the paper derives Accuracy (AR), the share of questions the caption answers correctly; Inconsistency Rate (IR), the share of answered questions on which the caption is wrong; and Coverage Rate (CR), the share of questions the caption addresses at all. Using this setup on 5,677 videos with 109,796 QA pairs across 21 fine-grained dimensions, the paper reports that Gemini-2.5-Pro-Preview produces the most accurate and complete captions, while open-source models lag especially on entity shape, relative position, blur, and camera movement.

Load-bearing premise

The load-bearing premise is that the 109,796 question-answer pairs are correct enough to be treated as ground truth; the paper says human reviewers re-examined a latitude-based sample and corrected errors, but it does not report inter-annotator agreement or the fraction of pairs that were wrong, so a large hidden error rate would poison all three metrics.

Editorial extensions

If this is right

  • Caption generators can be debugged by dimension: a low AR on camera movement or relative position tells the developer exactly what class of detail to add or fix.
  • Text-to-video prompt engineering can be steered by CR: captions that omit shot type or lighting can be expanded before being fed to a generation model.
  • Reference-based and embedding-based metrics can be supplemented by question-answering evaluation, which checks spatio-temporal facts directly rather than lexical or semantic similarity.
  • The 21-dimension question bank gives the community a common protocol for comparing vision-language models' video-description ability, with a published leaderboard of ten models.
  • Because each question carries its own category, scores can be aggregated per dimension, which is not possible with a single holistic quality score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The QA-pair scaffold could be reused as a training signal: since errors are localized to specific dimensions, a captioner could be optimized or fine-tuned to maximize AR and CR on this benchmark, effectively turning evaluation into a reward.
  • The reported correlation between longer captions and higher CR suggests coverage is partly a verbosity effect; a stricter metric might reward concise captions that still cover many dimensions, or penalize padding.
  • The absolute scores depend on the evaluating LLM (the paper shows two different LLM evaluators give different numbers), so cross-model comparisons are only meaningful within a fixed evaluator; a calibration set or normalization would make scores portable.
  • The benchmark's dimensions are text-to-video-centric; the same question-answering mechanism could extend to audio events, scene transitions, or temporal ordering, which are currently folded into coarse categories like action and camera movement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces VCapsBench, a benchmark for evaluating the quality of video captions generated by vision-language models (VLMs). It consists of 5,677 videos from 10 public datasets and 109,796 ternary (yes/no/unanswerable) question-answer pairs organized into 21 fine-grained dimensions across four broad categories. The authors propose three metrics—Accuracy Rate (AR), Inconsistency Rate (IR), and Coverage Rate (CR)—computed through an LLM-based 'TextQA expert' that answers each question using only the caption text. They evaluate ten VLMs, using Gemini-2.5-Pro-Preview and GPT-4.1 as evaluators, and report that Gemini-2.5-Pro-Preview achieves the best AR and CR and the lowest IR. A 100-sample manual evaluation in Table 4 is presented as evidence that the automatic pipeline is consistent with manual judgment.

Significance. If the QA ground truth is reliable, VCapsBench would be a valuable large-scale resource for fine-grained video caption evaluation, filling a gap left by image-only caption benchmarks such as CapsBench and by holistic video understanding benchmarks like MVBench. The metric definitions are clear, the dataset is large and diverse, and the inclusion of an 'Unanswerable' option is a sensible safeguard against LLM hallucination in evaluation. The use of two independent evaluators and the manual consistency check are genuine strengths. However, the significance is contingent on a quantitative demonstration that the 109,796 QA pairs are correct; the paper currently does not provide that evidence, which is the main risk to the benchmark's validity.

major comments (4)
  1. [Section 3.3 (Human Correction)] The load-bearing assumption of the benchmark is that the 109,796 QA answers are correct ground truth, since AR, IR, and CR (Eq. 1-3) are all defined against those answers. The 'Human Correction' step is described only qualitatively: 'Human reviewers re-examine these QA-pairs, deleting those with unreasonable or incorrect questions and correcting those with erroneous answers.' The paper reports no number of reviewers, no inter-annotator agreement, no fraction of pairs deleted or corrected, and no description of the 'latitude' sampling procedure. Please provide a quantitative audit of this step, including per-category deletion/correction rates and an inter-annotator agreement study on a representative sample. Without this, the 'human-verified' claim is unsupported, and any systematic bias in the LLM-generated QA pairs propagates directly into all reported metrics.
  2. [Section 4.2 (Evaluation Analysis)] The circularity with Gemini is acknowledged but not bounded. The QA pool is generated by pipelines that use Gemini (among other models) to propose questions and answers, and Gemini-2.5-Pro-Preview then ranks first on AR, IR, and CR. This is not limited to a coverage-rate inflation: if the QA question distribution is biased toward content that Gemini captions mention, AR and IR are also affected because the metrics are computed on that biased question set. Please report the per-source decomposition of QA pairs (e.g., which pairs came from which generator), and rerun the ranking on a subset of QA pairs generated without Gemini involvement (e.g., the direct-from-video pipeline) to test whether the observed Gemini advantage persists.
  3. [Section A.2, Table 4] The 100-sample manual evaluation validates the overall AR/IR/CR pipeline, but it does not validate per-pair correctness of the QA ground truth. Manually judging whether a caption is accurate/coherent/complete is a different task from verifying whether each stored answer to a yes/no question is in fact the correct answer to that question for the video. Please add a per-pair human verification study in which annotators label each QA pair as correct/incorrect (and perhaps also as ambiguous), with agreement rates and a breakdown of errors across the 21 dimensions.
  4. [Section 4, Experimental Setup; Tables 1 and 3] The results are reported as point estimates without confidence intervals or significance tests. Several differences discussed in the text are small (for example, some dimension-level AR and IR gaps between adjacent models are around one to two percentage points), so without error bars or bootstrap intervals it is unclear whether the fine-grained rankings are statistically meaningful. Please provide bootstrap confidence intervals or another form of uncertainty quantification for the headline numbers in Tables 1 and 3.
minor comments (5)
  1. [Section 3.3] The model name 'Geinimi' should be 'Gemini'.
  2. [Section 2 (Related Work)] The reference to DSG appears as 'DSG [30]]' with an extra closing bracket; please correct the citation formatting.
  3. [Sections 3.2, 3.3, and 3.4] The dataset name is spelled inconsistently as both 'ChronomicBench' and 'ChronomicBench'; the cited work is 'ChronoMagic-Bench', so please use a consistent spelling matching the reference.
  4. [Section A.2, Table 4 caption] The caption refers to 'method b' and 'method c' without defining them; please identify these as the two LLM evaluators (e.g., Gemini-2.5-Pro-Preview and GPT-4.1) used in the comparison.
  5. [Section A.2] The text 'As shown in Figure 1' should refer to the relevant figure in the appendix (e.g., Fig. 10 or Fig. 11), not the main-text Figure 1.

Circularity Check

1 steps flagged · score 5.0 of 10

Gemini-derived QA ground truth makes the Gemini-first ranking partly circular; the paper itself concedes the QA pairs were built from Gemini captions.

  1. self definitional [Sec. 3.3 (Annotation Details) and Sec. 4.2 (Evaluation Analysis)]
    "Although Gemini surpasses open-source VLMs in every aspect, showcasing its advanced video comprehension abilities, it is crucial to acknowledge that Gemini’s captions were utilized in creating QA-pairs, potentially influencing its elevated CR."

    The benchmark's ground-truth QA answers are constructed from the outputs of the model that is then ranked first. Section 3.3 describes a pipeline where VLMs (including Gemini) produce captions and LLMs (including Gemini) turn those captions into QA pairs, later filtered by Gemini1.5. AR/IR/CR are defined as agreement between an LLM's answer and the stored ground-truth answer. A Gemini caption is therefore, by construction, more likely to agree with Gemini-derived, Gemini-filtered answers. The paper's admission that this 'potentially influenc[ed] its elevated CR' concedes that the top ranking is partly an artifact of construction. The qualitative Human Correction step reports no correction counts or inter-annotator agreement, so it does not show that this built-in advantage was removed.

full rationale

The central evaluation chain is: videos --> VLMs (including Gemini) propose captions --> LLMs (including Gemini) propose QA pairs --> Gemini1.5 filters/deduplicates them --> human reviewers spot-correct --> the pipeline scores candidate captions by agreement with the stored answers. Because the stored 'ground truth' originates from the very model family that then wins, the Gemini-first headline result is partially forced by the benchmark's construction rather than by independent measurement. The paper itself states that Gemini's captions were used in creating QA pairs and that this 'potentially influenc[ed] its elevated CR.' This is the specific reduction: the score is agreement with answers derived from the evaluated model. At the same time, there is real independent content: the benchmark is large, the QA dimensions are defined independently of any single model, a second LLM expert (GPT-4.1) produces a similar ranking, and a 100-sample manual check in Table 4 confirms the overall pipeline ordering. The human-correction step is described without quantitative detail, so it cannot be shown to eliminate the Gemini bias, but the manual check and second expert prevent the whole paper from reducing to a self-citation or a pure tautology. I therefore rate the circularity as partial: the Gemini-first claim is substantially overdetermined by the QA-generation process, but the benchmark retains independent value. Score 5 rather than 6 because the paper explicitly flags the issue and because the appendix provides a second evaluator and a small manual validation.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark introduces no fitted parameters or new theoretical entities. It depends on assumptions about annotation quality and the validity of the chosen dimensions.

assumptions (3)
  • domain assumption The human-corrected QA pairs are accurate ground truth for video content.
    The paper relies on the correctness of the QA pairs to compute AR, IR, and CR, but the correction process is only described qualitatively in Section 3.3.
  • domain assumption The 21 fine-grained dimensions are critical for text-to-video generation.
    The selection of dimensions is motivated by the needs of text-to-video generation, but no empirical evidence or ablation is provided to show they are the most important ones.
  • domain assumption The LLM evaluator's answers to questions based on captions are reliable indicators of caption content.
    The metrics depend on the LLM's ability to correctly answer yes/no/unanswerable questions from the caption text, as described in Section 3.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation." pith.science (2026). https://pith.science/paper/ZYZLYEHA

@misc{pith2026250523484,
  author       = {Pith},
  title        = {Pith review of: VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZYZLYEHA}},
  note         = {Machine review of arXiv:2505.23484}
}
read the original abstract

Video captions play a crucial role in text-to-video generation tasks, as their quality directly influences the semantic coherence and visual fidelity of the generated videos. Although large vision-language models (VLMs) have demonstrated significant potential in caption generation, existing benchmarks inadequately address fine-grained evaluation, particularly in capturing spatial-temporal details critical for video generation. To address this gap, we introduce the Fine-grained Video Caption Evaluation Benchmark (VCapsBench), the first large-scale fine-grained benchmark comprising 5,677 (5K+) videos and 109,796 (100K+) question-answer pairs. These QA-pairs are systematically annotated across 21 fine-grained dimensions (e.g., camera movement, and shot type) that are empirically proven critical for text-to-video generation. We further introduce three metrics (Accuracy (AR), Inconsistency Rate (IR), Coverage Rate (CR)), and an automated evaluation pipeline leveraging large language model (LLM) to verify caption quality via contrastive QA-pairs analysis. By providing actionable insights for caption optimization, our benchmark can advance the development of robust text-to-video models. The dataset and codes are available at website: https://github.com/GXYM/VCapsBench.

Figures

Figures reproduced from arXiv: 2505.23484 by the authors.

Figure 1
Figure 1. Illustration of video caption evaluation by VCapsBench. Evaluate the detail, comprehen [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An example of video caption and question-answer pairs in our VCapsBench. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) Video source distribution; (b) Video duration distribution; (c) Video resolution distribu [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The pipeline of QA-pairs generation, which includes multiple data processing pipelines and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The question length and category distribution. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Results of Gemini-2.5-Pro-Preview captioning evaluation, organized by category. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The relationship between CR, IR and caption length, where Gemini-2.5-Pro-Preview as a [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Representative video from VCapsBench. ttiidfVCBh“Q2572 [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Results of GPT-4.1 captioning evaluation, organized by category. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: The relationship between CR, IR and caption length, where GPT-4.1 as a TextQA expert. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Video caption word length distribution. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Animation Needs Attention: A Holistic Approach to Slides Animation Comprehension with Visual-Language Models

    cs.AI 2025-07 conditional novelty 6.0 of 10

    A vision-language model fine-tuned on a new synthetic slide-animation dataset outperforms GPT-4.1 and Gemini-2.5-Pro at describing slide animations, especially on synthetic evaluation data.

  2. AVC-DPO: Aligned Video Captioning via Direct Preference Optimization

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Using preference pairs synthesized from the model's own prompt-varied outputs, DPO fine-tuning improves Qwen2.5-VL-7B's video captioning on the VDC benchmark from 43.9 to 51.1 average VDCSCORE.

Reference graph

Works this paper leans on

45 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    arXiv 2024

  2. [2]

    St-llm: Large language models are effective temporal learners

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. In European Conference on Computer Vision, pages 1–18. Springer, 2025

    2025

  3. [3]

    Cogvlm2: Visual language models for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500, 2024

    arXiv 2024

  4. [4]

    Oryx mllm: On- demand spatial-temporal understanding at arbitrary resolution.arXiv preprint arXiv:2409.12961, 2024

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On- demand spatial-temporal understanding at arbitrary resolution.arXiv preprint arXiv:2409.12961, 2024

    arXiv 2024

  5. [5]

    Video-language alignment pre-training via spatio-temporal graph transformer

    Shi-Xue Zhang, Hongfa Wang, Xiaobin Zhu, Weibo Gu, Tianjin Zhang, Chun Yang, Wei Liu, and Xu-Cheng Yin. Video-language alignment pre-training via spatio-temporal graph transformer. arXiv e-prints, pages arXiv–2407, 2024

    work page 2024

  6. [6]

    Aria: An open multimodal native mixture-of-experts model

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2024

    arXiv 2024

  7. [7]

    Video generation models as world simulators, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators, 2024

    2024

  8. [8]

    Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models

    Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024

    arXiv 2024

Show all 45 references
  1. [9]

    Videotetris: Towards compositional text-to-video generation

    Ye Tian, Ling Yang, Haotian Yang, Yuan Gao, Yufan Deng, Jingmin Chen, Xintao Wang, Zhaochen Yu, Xin Tao, Pengfei Wan, et al. Videotetris: Towards compositional text-to-video generation. arXiv preprint arXiv:2406.04277, 2024

    2024 arXiv

  2. [10]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  3. [11]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024

    2024 arXiv

  4. [12]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024

    2024 arXiv

  5. [13]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  6. [14]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  7. [15]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22195–2...

    2024

  8. [16]

    Videovista: A versatile benchmark for video understanding and reasoning

    Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. Videovista: A versatile benchmark for video understanding and reasoning. arXiv preprint arXiv:2406.11303, 2024. 10

    2024 arXiv

  9. [17]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024

    2024 arXiv

  10. [18]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  11. [19]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024

    2024 arXiv

  12. [20]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. arXiv preprint arXiv:2407.15754 , 2024

    2024 arXiv

  13. [21]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pages 65–72, 2005

    2005

  14. [22]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  15. [23]

    Spice: Semantic propositional image caption evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 , pages 382–398. Springer, 2016

    2016

  16. [24]

    Cider: Consensus-based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015

    2015

  17. [25]

    Infometic: An informative metric for reference-free image caption evaluation

    Anwen Hu, Shizhe Chen, Liang Zhang, and Qin Jin. Infometic: An informative metric for reference-free image caption evaluation. arXiv preprint arXiv:2305.06002, 2023

    2023 arXiv

  18. [26]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021

    2021 arXiv

  19. [27]

    Tiger: Text-to-image grounding for image caption evaluation

    Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, and Jianfeng Gao. Tiger: Text-to-image grounding for image caption evaluation. arXiv preprint arXiv:1909.02050, 2019

    1909 arXiv

  20. [28]

    Faier: Fidelity and adequacy ensured image caption evaluation

    Sijin Wang, Ziwei Yao, Ruiping Wang, Zhongqin Wu, and Xilin Chen. Faier: Fidelity and adequacy ensured image caption evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14050–14059, 2021

    2021

  21. [29]

    Qace: Asking questions to evaluate an image caption

    Hwanhee Lee, Thomas Scialom, Seunghyun Yoon, Franck Dernoncourt, and Kyomin Jung. Qace: Asking questions to evaluate an image caption. arXiv preprint arXiv:2108.12560, 2021

    2021 arXiv

  22. [30]

    Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation

    Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation. arXiv preprint arXiv:2310.18235, 2023

    arXiv

  23. [31]

    Ella: Equip diffusion models with llm for enhanced semantic alignment

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024

    2024 arXiv

  24. [32]

    Playground v3: Im- proving text-to-image alignment with deep-fusion large language models

    Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Chase Lambert, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Im- proving text-to-image alignment with deep-fusion large language models. arXiv preprint arXiv:2409.10695, 2024. 11

    2024 arXiv

  25. [33]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  26. [34]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In IEEE/CVF Conference o...

    2024

  27. [35]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision ...

    2022

  28. [36]

    Bdd100k: A diverse driving dataset for heterogeneous mul- titask learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous mul- titask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p...

    2020

  29. [37]

    Sharegpt4video: Improving video understanding and gener- ation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and gener- ation with better captions. In The Thirty-eight Conference on Neural Information Processing Sys...

  30. [38]

    Vidgen-1m: A large-scale dataset for text-to-video generation

    Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, and Hao Li. Vidgen-1m: A large-scale dataset for text-to-video generation. arXiv preprint arXiv:2408.02629, 2024

    2024 arXiv

  31. [39]

    Chronomagic-bench: A benchmark for metamorphic evaluation of text-to-time-lapse video generation

    Shenghai Yuan, Jinfa Huang, Yongqi Xu, Yaoyang Liu, Shaofeng Zhang, Yujun Shi, Ruijie Zhu, Xinhua Cheng, Jiebo Luo, and Li Yuan. Chronomagic-bench: A benchmark for metamorphic evaluation of text-to-time-lapse video generation. arXiv preprint arXiv:2406.18522, 2024

    2024 arXiv

  32. [40]

    Finevideo

    Miquel Farré, Andi Marafioti, Lewis Tunstall, Leandro V on Werra, and Thomas Wolf. Finevideo. 2024

    2024

  33. [41]

    Lift: Leveraging human feedback for text-to-video model alignment

    Yibin Wang, Zhiyu Tan, Junyan Wang, Xiaomeng Yang, Cheng Jin, and Hao Li. Lift: Leveraging human feedback for text-to-video model alignment. arXiv preprint arXiv:2412.04814, 2024

    2024 arXiv

  34. [42]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  35. [43]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024

    2024

  36. [44]

    Nvila: Efficient frontier visual language models

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04468, 2024

    2024 arXiv

  37. [45]

    ↑" indicates that the larger the value, the better; The symbol “↓

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025. 12 A Technical Ap...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.