Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

ARGUS: Hallucination and Omission Evaluation in Video-LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Even the best video-LLM tested hallucinates in about 41% of its free-form caption sentences, so ARGUS argues that QA benchmarks overstate video-model reliability.

desk verdict A genuinely useful free-form captioning benchmark, but the Eq. (4) normalization is wrong and likely shifts the leaderboard; fix before building on the numbers. read the letter →

arxiv 2506.07371 v2 pith:YG4YCSML submitted 2025-06-09 cs.CV

Ruchit Rawal , Reza Shirkavand , Heng Huang , Gowthami Somepalli , Tom Goldstein This is my paper
classification cs.CV
keywords videolargelanguagemodelshallucinationevaluationdensecaptioningnaturalinferenceomissiontemporalorderingLLM-as-a-judgebenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to measure how often video large language models make things up when asked to describe a video in their own words, rather than answer a multiple-choice question. It introduces ARGUS, a benchmark of 500 videos with dense human-written captions, and two normalized scores: ArgusCost-H, the fraction of a model's caption sentences that are hallucinated (not entailed by the human caption, or describing events out of order), and ArgusCost-O, the fraction of human-annotated details the model leaves out. Evaluated on 23 models, the benchmark finds that even the strongest model, Gemini-2.0-Flash, hallucinates in about 41% of its caption sentences; most models omit more than they hallucinate, and hallucination and omission rates correlate at 0.65. The point of the exercise is to show that verification-style leaderboards overstate how ready video models are, because free-form generation is a substantially harder and more realistic setting.

What carries the argument

The engine is a normalized cost pair built from sentence-level natural-language entailment plus monotone sequence alignment. Sentences are classified into three types — summary, visual description, and dynamic action — and only dynamic actions incur an order penalty, which counts inversions between the order of events in the model caption and their order in the human caption. A dynamic program assigns each generated sentence to a human sentence (or to the content-error bucket) so that fabricated events cannot scramble the temporal matching, and the final ArgusCost-H and ArgusCost-O are the minimized costs divided by the maximum possible cost, giving a percentage that is comparable across captions of different lengths.

What would settle it

Have a second, independent team write fresh dense captions for a subset of the same 500 videos and recompute ArgusCost-H and ArgusCost-O for the same model outputs; if model rankings or absolute scores shift beyond the reported inter-judge variation (roughly 2–3 points), the metrics are measuring agreement with one particular annotation rather than grounded video understanding.

Watch

Extended reading notes

Core claim

On the authors' own terms, ARGUS establishes a dual-metric evaluation of free-form dense video captioning. Each generated sentence is typed as a summary, a visual description, or a dynamic action, and an LLM judge decides whether it is entailed, contradicted, or undetermined with respect to the human ground-truth caption; a dynamic program then finds the alignment between generated and human sentences that minimizes the combined cost of non-entailment and of event-order inversions among dynamic actions. That minimum, normalized by the worst possible cost, is ArgusCost-H, and reversing the roles of generated and human captions yields ArgusCost-O. The central finding is that every one of the 23 models tested hallucinates at a high rate in free-form generation, with the best model at 41%, and that stronger models reduce dynamic-action errors faster than they reduce fabricated visual details.

Load-bearing premise

The whole score assumes the human-written captions are accurate and complete enough to serve as ground truth, so that anything a model says which is not entailed by them counts as a hallucination and anything they contain that the model omits counts as an omission.

Editorial extensions

If this is right

  • Verification-style QA benchmarks should not be read as evidence of low hallucination, since models that verify facts correctly still fabricate details when generating free-form captions.
  • Omitting information is the more common failure: most models score worse on ArgusCost-O than ArgusCost-H, so caption-length or accuracy metrics alone miss the dominant error mode.
  • Within a model family, scale usually reduces both hallucination and omission, but the InternVL2 family defies this, so scaling is not a guaranteed cure.
  • Supplying more frames consistently reduces omissions for all tested models, while its effect on hallucination varies by model; frame count is therefore a meaningful evaluation parameter.
  • Post-training that reduces hallucination, such as DPO in LLaVA-Next-Video, can cut ArgusCost-H from 59% to 45% while pushing omissions up to 85%, revealing a hallucination–omission trade-off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because undetermined statements — details a human caption neither supports nor rules out — count as hallucinations, ArgusCost-H likely over-penalizes correct but unannotated details; the authors' own human study places most disagreements exactly there. An editor's reading: the absolute numbers are an upper bound on grounded hallucination, while the rankings are more trustworthy than the magnitudes
  • A natural stress-test the paper does not run is to vary the temporal penalty weight lambda; at larger lambda, order errors would count nearly as heavily as content errors, which could re-rank models whose captions scramble events, and would show how much of the reported scores is temporal versus factual.
  • The same NLI-plus-alignment machinery could be pointed at other free-form outputs, such as summaries of surgery videos or surveillance footage, where hallucination and omission have asymmetric real-world costs; the dual-metric design transfers directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. ARGUS is a benchmark for evaluating free-form dense video captioning by Video-LLMs, built around two normalized metrics: ArgusCost-H, which scores hallucinated content via an LLM entailment judge and a dynamic-programming alignment with temporal-order penalties for dynamic actions, and ArgusCost-O, which scores omission of human-caption content by reversing source and target. The authors curate 500 videos with human-written dense captions, evaluate 23 Video-LLMs, and report that even the strongest model, Gemini-2.0-Flash, has ArgusCost-H around 41%, with most models above the y=x line in the hallucination-omission plane. They also provide sensitivity analyses across prompts, decoding temperatures, and judge LLMs, and a human study reporting 91.26% agreement with the judge's verdicts.

Significance. If correct, ARGUS would be a useful complement to QA-style video hallucination benchmarks, because it targets exactly the open-ended generation regime in which verification-style accuracy overstates reliability, and it is the first benchmark in the paper's comparison to measure both hallucination and omission in dense captions. The dataset release, the multi-judge sensitivity analysis, and the human agreement study are strengths that make the central qualitative finding plausible: free-form video captions contain a substantial amount of ungrounded and omitted content across all tested models. The caveat is that the headline contribution is a model leaderboard, so any defect in the normalization or in the completeness of human references can directly affect the stated rankings.

major comments (3)
  1. [Section 3.2.2, Eq. (4)] The normalization denominator in Eq. (4) is not the maximum possible hallucination cost. By Eq. (1), an entailed target sentence of type SUM or VD has base cost 0 for every possible alignment j, so it can never contribute to the base cost. The denominator's first term (n-d) nevertheless counts every entailed SUM/VD sentence as though it could incur penalty 1. The true worst-case base cost is (number of non-entailed targets) plus (number of entailed dynamic-action targets), which equals n - E where E is the number of entailed SUM/VD targets, plus the same lambda*d(d-1)/2 order term. Since Fig. 5 shows a balanced mixture of sentence types and E is typically large, the denominator in Eq. (4) is inflated, deflating ArgusCost-H unevenly across models. The headline comparisons in Section 4.2, such as Gemini-2.0-Flash at 41% versus GPT-4o at 44% and the LLaVA-NV-DPO versus non-DPO gap, can therefore change after correction. The qualitative conclusion that hallucination rates are high is conservative, but the ranking claims require recomputation with the corrected denominator.
  2. [Appendix E, Fig. 13] The paper acknowledges that human captions are not exhaustive and that most human-judge disagreements concern visual-description sentences the LLM flagged as hallucinations but humans judged entailed. This is a load-bearing limitation because the benchmark treats human captions as complete ground truth for both omission and entailment judgments. A model that correctly mentions details not recorded in the human reference is penalized in ArgusCost-H, while ArgusCost-O overstates omissions for details the human annotator did not include. The reported 91.26% human agreement is agreement with the judge's verdicts given the human caption, not evidence that the human captions are complete. The paper should quantify the sensitivity of model rankings to reference incompleteness, for example by reporting results on the subset of videos where captions were cross-validated by multiple authors or by providing human-verified entailment subsets.
  3. [Section 3.2.2, Omission Cost] No explicit equation is given for ArgusCost-O; the text only says that the roles of source and target are reversed. If the same normalized denominator as Eq. (4) is used, it inherits the same error, because entailed sentences in the new target set have zero base cost but are counted as capable of cost 1. Since ArgusCost-O is a central metric presented in Fig. 1 and Fig. 7, the paper must specify its exact formula and report it under the corrected normalization before the omission rankings can be considered reliable.
minor comments (5)
  1. [Appendix E] The sentence 'even the strongest models exhibit substantially higher hallucination rates--sometimes reaching up to 40% disagreement' is confusing, because Fig. 13 reports about 9% average disagreement overall and 6.6% for the visual-description/hallucination category; please reword to distinguish disagreement rates from hallucination rates.
  2. [Section 4.2, Fig. 6, Fig. 1] Model naming is inconsistent: 'LLaVA-NV-DPO', 'LLaV A-Next-Video (DPO)', and 'LLaV A-NV-DPO' are used interchangeably; please choose one consistent name for each model.
  3. [Section 3.2.2, Eq. (2)] The temporal penalty factor lambda is fixed at 0.1 with no sensitivity analysis. Because temporal penalties are reported to make up only 1-4% of total cost, this is unlikely to be rank-changing, but a short lambda-ablation table would make the metric less hand-set and would strengthen the robustness claim.
  4. [Section 1 and Fig. 1] The claim that 'Gemini-2.0-Flash achieves the best performance' in Fig. 1 should state whether 'best' refers to ArgusCost-H alone or a joint criterion over both metrics, since no combined index is defined.
  5. [Dataset release] The paper states that dataset and artifacts are available, but it does not mention release of the evaluation code that computes the DP alignment, the judge prompt, and the normalization; providing this code would make the benchmark fully reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ARGUS evaluates free-form captions against independent human ground truth, and the GPT-4o-as-judge self-bias is explicitly tested with alternative judges and a human agreement study.

full rationale

The paper's derivation chain is self-contained against external ground truth. ArgusCost-H and ArgusCost-O are computed by matching model-generated sentences to human-written captions and applying an LLM entailment judge; the measured quantities (hallucination and omission rates) are defined by comparison to that external reference, not in terms of the models being ranked. The only potentially self-referential element is the use of GPT-4o as judge while GPT-4o is also an evaluated model and contributed initial VDC captions that were later human-corrected. The authors explicitly acknowledge this in Appendix F ('there is a possibility of GPT-4o's scores inflating because of self-bias') and address it with four alternative judges (DeepSeek-R1, DeepSeek-V3, LLaMA-3.3, Qwen-2.5) showing ranking correlations r >= 0.92, plus a 26-participant human study with 91.26% agreement on line-level verdicts. That external evidence prevents the judge choice from being load-bearing circularity. The Eq. (4) normalization issue noted in review (the denominator counts entailed SUM/VD sentences, which have base cost 0 under Eq. (1), as potentially costly) is a real miscalibration/validity concern that could affect score magnitudes and possibly rankings, but it is not a case of a prediction reducing to its inputs by definition; it is a metric-construction bug, not circularity. No self-citation is load-bearing: references to the authors' prior CinePile work appear only in a general list of long-video benchmarks. Overall, the central qualitative claim that even strong Video-LLMs hallucinate at high rates in free-form captioning is conservative and independent of the paper's own fitted values or definitions.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four main assumptions: an LLM judge can reliably perform NLI on video captions, human captions are complete ground truth, the normalization makes scores comparable across caption lengths, and the entailment prompt's silence rules are valid. The only numeric free parameter is lambda=0.1. No new physical or conceptual entities are introduced.

free parameters (1)
  • lambda = 0.1
    Scaling factor for temporal-ordering penalty in Equation 2. Chosen by hand; it controls how strongly out-of-order dynamic actions are penalized and directly affects ArgusCost-H.
assumptions (4)
  • domain assumption LLM judge verdicts are accurate for video-caption NLI
    Section 3.1 relies on GPT-4o to classify each sentence as entailed, contradictory, or undetermined, and to select evidence lines. The paper validates this with judge-consensus correlations and a small human study, but the assumption remains load-bearing.
  • domain assumption Human ground-truth captions are complete enough to define omissions
    Section 3.2 defines ArgusCost-O by checking which human sentences are entailed by the model caption. Appendix E admits that human captions miss fine-grained visual details, so omission scores may be inflated.
  • domain assumption Max-cost normalization fully corrects for caption-length differences
    Equation 4 normalizes by (n-d)+lambda*d(d-1)/2, assuming this captures the worst-case cost. The paper only reports correlations between caption length and scores, not a validation of the normalization.
  • ad hoc to paper Silence about an aspect implies no change or non-occurrence
    The judge prompt in Appendix I.3 instructs the LLM to treat target claims of unchanged aspects or non-events as entailed when the source is silent. This rule lowers ArgusCost-H for such claims and is not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ARGUS: Hallucination and Omission Evaluation in Video-LLMs." pith.science (2026). https://pith.science/paper/YG4YCSML

@misc{pith2026250607371,
  author       = {Pith},
  title        = {Pith review of: ARGUS: Hallucination and Omission Evaluation in Video-LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YG4YCSML}},
  note         = {Machine review of arXiv:2506.07371}
}
read the original abstract

Video large language models have not yet been widely deployed, largely due to their tendency to hallucinate. Typical benchmarks for Video-LLMs rely simply on multiple-choice questions. Unfortunately, VideoLLMs hallucinate far more aggressively on freeform text generation tasks like video captioning than they do on multiple choice verification tasks. To address this weakness, we propose ARGUS, a VideoLLM benchmark that measures freeform video captioning performance. By comparing VideoLLM outputs to human ground truth captions, ARGUS quantifies dual metrics. First, we measure the rate of hallucinations in the form of incorrect statements about video content or temporal relationships. Second, we measure the rate at which the model omits important descriptive details. Together, these dual metrics form a comprehensive view of video captioning performance.

Figures

Figures reproduced from arXiv: 2506.07371 by the authors.

Figure 1
Figure 1. Relationship between Hallucination and Omis￾sion. The hallucination and omission cost metrics are correlated; however, most models exhibit more omissions than hallucina￾tions. Marker size indicates the average caption length per model. Gemini-2.0-Flash achieves the best performance. into open-ended tasks like dense video captioning. The latter is particularly important for models that assist users with perceptual di… view at source ↗
Figure 2
Figure 2. An example annotation (see here) of a video by the Gemini-2-Flash model. Using our framework ARGUS, we iden￾tify both hallucinations and omissions in this dense caption. We present the full human and model-captions in Appendix J. suring both the accuracy and completeness of video under￾standing. To the best of our knowledge, no benchmarks cur￾rently exist that systematically evaluate hallucinations and omissions in … view at source ↗
Figure 3
Figure 3. A Video LLM can correctly answer targeted questions about a video (left; see [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Proposed dynamic programming formulation for sentence alignment. Given sentence-level entailment scores from the LLM-as-a-judge, we compute the matching cost between a tar￾get sentence (ti) and a source sentence (sj ) by combining the en￾tailment score with a temporal …
Figure 5
Figure 5. Figure 5: ArgusBench Statistics: video lengths (left), word-counts (middle), and sentence-length (x-axis) distribution by sentence-type (right). Our dataset has a balanced representation across durations and sentence types, and a high word-count density. SmolVLM2 (256M) SmolVLM2…
Figure 6
Figure 6. Figure 6: ArgusCost-H (Hallucination Cost) across Video-LLMs. Even top performers like Gemini-2.0-Flash produce up to 40% hallucinated content. Although summary errors are low, stronger models still fabricate visual details despite improved dynamic action descriptions. Open-sour…
Figure 7
Figure 7. Figure 7: (Left,Middle) Relationship between hallucination/omission and model-size. For most open-source model families (except In￾ternVL2), larger models tend to have lower hallucination (ArgusCost-H) and omission (ArgusCost-O) costs, indicating that scale generally improves pe…
Figure 8
Figure 8. Figure 8: Sensitivity to different LLMs-as-judge. High Pear￾son ranking correlations across different LLMs-as-judge indicate strong agreement in evaluations and robustness in ranking trends. For intra-prompt analyses, we sample three captions for each model using our default pro…
Figure 9
Figure 9. Figure 9: Relationship between verification accuracy and hallucination cost across 50 video samples. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Sensitivity to intra- and inter-prompt variations. Low standard error both intra-prompt (same prompt, default temperature) and inter-prompt (across prompts) suggests consistent hallucination costs across models. standard error. We find that the standard error is minim…
Figure 11
Figure 11. Figure 11: Sensitivity across decoding temperatures. Hallucination costs show little variation across temperatures, indicating low sensitivity to decoding changes speed is implied. In contrast, mentioning a background flower not noted by the human may be marked as a hallucinatio…
Figure 12
Figure 12. Figure 12: Registration Webpage (Left) and the task webpage (right) from an example video in the human study survey. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Human-study disagreement rate decomposed into “LLM-verdict-type” and the “sentence-type”. Overall, disagreement stands at [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Breakdown of hallucination errors into “contradiction” and “undetermined”. Undetermined errors dominate the majority of [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Contribution of Temporal Penalties in the overall cost. Temporal penalties make a minor proportion of the total costs. [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: Visualizing the correlation between clip duration, ArgusCost-H and ArgusCost-O, and caption density [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Effect of total number of frames on ArgusCost-H and ArgusCost-O. Gemini models and SmolVLM2 show consistent reduction [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Correlation between various video-characteristics and ArgusCost-H. [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Model-wise correlation heatmaps between video filter metrics and ArgusCost-H. [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Temporal Compositional Reasoning in Long-Form Sports Videos

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    SportsTime plus Chain-of-Time Reasoning (temporal-reward GRPO and anchor-observe-infer) modestly lifts open-ended sports VideoQA and step-wise temporal grounding over 4B–8B MLLM baselines.

Reference graph

Works this paper leans on

83 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [1]

    https://huggingface.co/ blog/smolvlm2, 2025

    SmolVLM2: Bringing Video Understanding to Every De- vice — huggingface.co. https://huggingface.co/ blog/smolvlm2, 2025. [Accessed 07-03-2025]. 7

    2025

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  3. [3]

    Are language models better at generating an- swers or validating solutions?, 2025

    Anonymous. Are language models better at generating an- swers or validating solutions?, 2025. Under review. 1

    2025

  4. [4]

    The snli corpus

    Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. The snli corpus. 2015. 13

    2015

  5. [5]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, pages 961–970, 2015. 1

    work page 2015

  6. [6]

    e-snli: Natural language inference with natural language explanations

    Oana-Maria Camburu, Tim Rockt ¨aschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. Advances in Neural Information Processing Systems, 31, 2018. 13

    work page 2018

  7. [7]

    Auroracap: Efficient, performant video detailed captioning and a new benchmark

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng-Neng Hwang, Sain- ing Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.03051, 2024. 5, 16, 17

    arXiv

  8. [8]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13320–13331, 2024. 16

    work page 2024

Show all 83 references
  1. [9]

    Neural- log: Natural language inference with joint neural and logical reasoning

    Zeming Chen, Qiyue Gao, and Lawrence S Moss. Neural- log: Natural language inference with joint neural and logical reasoning. arXiv preprint arXiv:2105.14167, 2021. 13

    2021 arXiv

  2. [10]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 7

    2024 arXiv

  3. [11]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open- source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhan...

    2024 arXiv

  4. [12]

    Vidhal: Benchmarking temporal hallucinations in vi- sion llms

    Wey Yeh Choong, Yangyang Guo, and Mohan Kankan- halli. Vidhal: Benchmarking temporal hallucinations in vi- sion llms. arXiv preprint arXiv:2411.16771, 2024. 2

    2024 arXiv

  5. [13]

    Trans- forming question answering datasets into natural language inference datasets

    Dorottya Demszky, Kelvin Guu, and Percy Liang. Trans- forming question answering datasets into natural language inference datasets. arXiv preprint arXiv:1809.02922, 2018. 13

    2018 arXiv

  6. [14]

    Sketch, ground, and refine: Top-down dense video caption- ing

    Chaorui Deng, Shizhe Chen, Da Chen, Yuan He, and Qi Wu. Sketch, ground, and refine: Top-down dense video caption- ing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 234–243, 2021. 14

    2021

  7. [15]

    diffusers/shot-categorizer-v0

    Hugging Face. diffusers/shot-categorizer-v0. https : / / huggingface . co / diffusers / shot - categorizer-v0, 2025. 20

    2025

  8. [16]

    Eva: Exploring the limits of masked visual representa- tion learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representa- tion learning at scale. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition ...

    2023

  9. [17]

    Tc-bench: Benchmark- ing temporal compositionality in text-to-video and image-to- video generation

    Weixi Feng, Jiachen Li, Michael Saxon, Tsu-jui Fu, Wenhu Chen, and William Yang Wang. Tc-bench: Benchmark- ing temporal compositionality in text-to-video and image-to- video generation. arXiv preprint arXiv:2406.08656, 2024. 5, 16

    2024 arXiv

  10. [18]

    Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 1

    2024 arXiv

  11. [19]

    Trueteacher: Learning factual 9 consistency evaluation with large language models

    Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. Trueteacher: Learning factual 9 consistency evaluation with large language models. arXiv preprint arXiv:2305.11171, 2023. 13

    2023 arXiv

  12. [20]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 8, 15

    2024 arXiv

  13. [21]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vi- sio...

    2022

  14. [22]

    Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models. In P...

    2024

  15. [23]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 3, 8, 15

    2025 arXiv

  16. [24]

    Ciem: Contrastive instruction evaluation method for better instruction tuning

    Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. Ciem: Contrastive instruction evaluation method for better instruction tuning. arXiv preprint arXiv:2309.02301, 2023. 2

    2023 arXiv

  17. [25]

    A better use of audio-visual cues: Dense video captioning with bi-modal transformer

    Vladimir Iashin and Esa Rahtu. A better use of audio-visual cues: Dense video captioning with bi-modal transformer. arXiv preprint arXiv:2005.08271, 2020. 14

    2005 arXiv

  18. [26]

    Multi-modal dense video captioning

    Vladimir Iashin and Esa Rahtu. Multi-modal dense video captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 958–959, 2020. 14

    2020

  19. [27]

    Faith- score: Fine-grained evaluations of hallucinations in large vision-language models

    Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du. Faith- score: Fine-grained evaluations of hallucinations in large vision-language models. arXiv preprint arXiv:2311.01477,

    arXiv

  20. [28]

    Throne: An object-based hallucination benchmark for the free-form generations of large vision-language models

    Prannay Kaul, Zhizhong Li, Hao Yang, Yonatan Duk- ler, Ashwin Swaminathan, CJ Taylor, and Stefano Soatto. Throne: An object-based hallucination benchmark for the free-form generations of large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vi- si...

    2024

  21. [29]

    Do you remember? dense video captioning with cross-modal memory retrieval

    Minkuk Kim, Hyeon Bae Kim, Jinyoung Moon, Jinwoo Choi, and Seong Tae Kim. Do you remember? dense video captioning with cross-modal memory retrieval. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13894–13904, 2024. 14

    2024

  22. [30]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. CoRR, abs/2408.03326, 2024. 2

    2024 arXiv

  23. [31]

    Llms-as-judges: a com- prehensive survey on llm-based evaluation methods

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Llms-as-judges: a com- prehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579, 2024. 3

    2024 arXiv

  24. [32]

    Jointly localizing and describing events for dense video captioning

    Yehao Li, Ting Yao, Yingwei Pan, Hongyang Chao, and Tao Mei. Jointly localizing and describing events for dense video captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7492–7500,

  25. [33]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 2

    2023 arXiv

  26. [34]

    Revisiting the role of lan- guage priors in vision-language models

    Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of lan- guage priors in vision-language models. arXiv preprint arXiv:2306.01879, 2023. 2, 12

    2023 arXiv

  27. [35]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 8, 15

    2024 arXiv

  28. [36]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023. 2

    2023 arXiv

  29. [37]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiu- tian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024. 2

    2024 arXiv

  30. [38]

    Logical rea- soning in large language models: A survey

    Hanmeng Liu, Zhizhang Fu, Mengru Ding, Ruoxi Ning, Chaoli Zhang, Xiaozhang Liu, and Yue Zhang. Logical rea- soning in large language models: A survey. arXiv preprint arXiv:2502.09100, 2025. 3

    2025 arXiv

  31. [39]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural In- formation Processing Systems, 36:46212–46244, 2023. 1

    2023

  32. [40]

    Streamlined dense video captioning

    Jonghwan Mun, Linjie Yang, Zhou Ren, Ning Xu, and Bo- hyung Han. Streamlined dense video captioning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6588–6597, 2019. 14

    2019

  33. [41]

    Neptune: The long orbit to benchmarking long video under- standing

    Arsha Nagrani, Mingda Zhang, Ramin Mehran, Rachel Hor- nung, Nitesh Bharadwaj Gundavarapu, Nilpa Jha, Austin Myers, Xingyi Zhou, Boqing Gong, Cordelia Schmid, et al. Neptune: The long orbit to benchmarking long video under- standing. arXiv preprint arXiv:2412.09582, 2024. 1

    2024 arXiv

  34. [42]

    Minerva: Evaluating complex video reasoning

    Arsha Nagrani, Sachit Menon, Ahmet Iscen, Shyamal Buch, Ramin Mehran, Nilpa Jha, Anja Hauth, Yukun Zhu, Carl V ondrick, Mikhail Sirotenko, et al. Minerva: Evaluating complex video reasoning. arXiv preprint arXiv:2505.00681,

    arXiv

  35. [43]

    OpenAI. Gpt-4o. https://openai.com/index/ hello-gpt-4o/, 2024. 1

    2024

  36. [44]

    Per- ception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Re- casens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Per- ception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing ...

    2024

  37. [45]

    Dense video captioning: A survey of techniques, datasets and evalu- ation protocols

    Iqra Qasim, Alexander Horsch, and Dilip Prasad. Dense video captioning: A survey of techniques, datasets and evalu- ation protocols. ACM Computing Surveys, 57(6):1–36, 2025. 14

    2025

  38. [46]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  39. [47]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 20

    2024 arXiv

  40. [48]

    Cinepile: A long video question answering dataset and benchmark

    Ruchit Rawal, Khalid Saifullah, Miquel Farr ´e, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813, 2024. 1

    2024 arXiv

  41. [49]

    Object hallucination in image cap- tioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. arXiv preprint arXiv:1809.02156, 2018. 2

    2018 arXiv

  42. [50]

    Fenice: Factuality evaluation of summarization based on nat- ural language inference and claim extraction

    Alessandro Scir `e, Karim Ghonim, and Roberto Navigli. Fenice: Factuality evaluation of summarization based on nat- ural language inference and claim extraction. arXiv preprint arXiv:2403.02270, 2024. 13

    2024 arXiv

  43. [51]

    Aligning large multi- modal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi- modal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023. 2

    2023 arXiv

  44. [52]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1, 14

    2023 arXiv

  45. [53]

    Laion-aesthetics predictor v1

    LAION team. Laion-aesthetics predictor v1. https:// github.com/LAION-AI/aesthetic-predictor ,

  46. [54]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078–10093, 2022. 14

    2022

  47. [55]

    Amber: An llm-free multi-dimensional bench- mark for mllms hallucination evaluation

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. Amber: An llm-free multi-dimensional bench- mark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397, 2023. 2

    2023 arXiv

  48. [56]

    End-to-end dense video captioning with parallel decoding

    Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6847– 6857, 2021. 14

    2021

  49. [57]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiao- han Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024. 1

    2024 arXiv

  50. [58]

    Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models

    Yuxuan Wang, Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models. arXiv preprint arXiv:2406.16338, 2024. 2, 12

    2024 arXiv

  51. [59]

    A broad-coverage challenge corpus for sentence understanding through inference

    Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017. 13

    2017 arXiv

  52. [60]

    Anliz- ing the adversarial natural language inference dataset

    Adina Williams, Tristan Thrush, and Douwe Kiela. Anliz- ing the adversarial natural language inference dataset. arXiv preprint arXiv:2010.12729, 2020. 13

    2010 arXiv

  53. [61]

    Joint event detection and description in continuous video streams

    Huijuan Xu, Boyang Li, Vasili Ramanishka, Leonid Sigal, and Kate Saenko. Joint event detection and description in continuous video streams. In 2019 IEEE winter conference on applications of computer vision (WACV), pages 396–405. IEEE, 2019. 14

    2019

  54. [62]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. 7, 8, 15

    2024 arXiv

  55. [63]

    mplug- owl3: Towards long image-sequence understanding in multi- modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug- owl3: Towards long image-sequence understanding in multi- modal large language models. In The Thirteenth Interna- tional Conference on Learning Representations, 2024. 7

    2024

  56. [64]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 14

    2023

  57. [65]

    Eventhallusion: Diagnosing event hal- lucinations in video llms

    Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, and Yu-Gang Jiang. Eventhallusion: Diagnosing event hal- lucinations in video llms. arXiv preprint arXiv:2409.16597,

  58. [66]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 3

    2023

  59. [67]

    Streaming dense video captioning

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 18243–18252, 2024. 14

    2024

  60. [68]

    Yes” (confirming the presence of information in the video) and another “leading/hallucinated

    Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An explo- ration of video understanding in large multimodal models. arXiv preprint arXiv:2412.10360, 2024. 2, 12 11 AR...

    2024 arXiv

  61. [70]

    In our setting, we adapt this score by computing the average aesthetic score across all frames in a video

    LAION Aesthetic Score [53]: The LAION aesthetic score is commonly used to assess the aesthetic quality of an image (ranging between 1-10). In our setting, we adapt this score by computing the average aesthetic score across all frames in a video. While this approximation has li...

  62. [71]

    diffusers/shot-categorizer-v0 [15] to identify the lighting type in each frame of a video

    Lighting Unique Count: We utilize a shot detection model i.e. diffusers/shot-categorizer-v0 [15] to identify the lighting type in each frame of a video. We then compute the number of unique lighting types (e.g., Daylight, Sunny) and use this count as a metric. The underlying a...

  63. [72]

    a person is cooking,

    Subject Presence: The number of distinct entities present in a video may also influence model performance, with more entities potentially leading to increased confusion for VideoLLMs. To quantify this, we employ the Segment Anything Model (SAM) [47] to generate segmentation ma...

  64. [73]

    Treat the **source_caption** as the premise and each line of the **target_caption** as a hypothesis

  65. [74]

    - **Contradiction**: Contains information that directly conflicts and is unsupported by the source_caption

    For each hypothesis line, decide if it is: - **Entailment**: Completely supported by the source_caption. - **Contradiction**: Contains information that directly conflicts and is unsupported by the source_caption. - **Undetermined**: Contains additional details or claims that t...

  66. [75]

    "" {source_caption}

    Follow the output format exactly as specified, without any extra summaries or notes. --- In-Context Examples: {IN_CONTEXT_EXAMPLES} 23 --- **source_caption:** """ {source_caption} """ **target_caption:** """ {target_caption} """ J. Details Regarding Qualitative Examples J.1. F...

    2008

  67. [76]

    SUNFEAST PASTA TREAT,

    **Introductory text on a black screen: ** The video starts with text stating "SUNFEAST PASTA TREAT," followed by " *cooking*," "45 sec." and the date "14th May, 2008."

    2008

  68. [77]

    **Family Scene: ** The initial scenes show a child wearing a green shirt and a chef’s hat looking at his mother standing near a kitchen shelf

  69. [78]

    Sunfeast Pasta Treat

    **Product Focus: ** The video clearly highlights the packaging of the "Sunfeast Pasta Treat" product, with the bag appearing close up to the camera. A pair of hands is shown cutting open the product’s packaging. The contents of the pasta treat are displayed

  70. [79]

    **Cooking Process: ** We see shots of the pasta being cooked, likely to emphasize the simplicity of the preparation. 24

  71. [80]

    **Appearance of the finished product: ** The finished pasta is served on a white plate, decorated with herbs

  72. [81]

    **Mother in the car: ** There are multiple shots of a woman in the car

  73. [82]

    Recipe Card

    **Child’s "Recipe Card": ** Briefly, there is a close-up of a handwritten note that contains instructions

  74. [83]

    Quick and Easy

    **"Quick and Easy" Text Overlay: ** Text emphasizing the convenience of the product appears on screen, accompanying a display of the pasta varieties. **Overall Narrative: ** The commercial likely follows this narrative: * Introduction: Setting the scene, indicating it’s a cook...

  75. [2022]

    A linear estimator on top of CLIP to predict the aes- thetic quality of images. 20

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.