ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models

Dongyan Zhao; Huishuai Zhang; Xiaojun Meng; Yifan Wang; Yueqian Wang

REVIEW 5 major objections 5 minor 4 cited by

ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces the first proactive video QA benchmark and proposes PAUC, a metric that combines response timing with correctness and agrees with human preferences better than timing-agnostic metrics.

desk verdict A useful new benchmark and a time-aware metric for proactive video QA, but the headline claim that PAUC matches human preferences rests on a circular user study. read the letter →

arxiv 2507.09313 v2 pith:ZJK2577A submitted 2025-07-12 cs.CV

Yueqian Wang , Xiaojun Meng , Yifan Wang , Huishuai Zhang , Dongyan Zhao This is my paper

classification cs.CV

keywords proactiveinteractionvideoquestionansweringevaluationmetricareaundercurvelargelanguagemodelsresponsetimingopen-endedmulti-turnQAbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ProactiveVideoQA is a benchmark for the setting in which a video assistant is given a question at the start and must decide on its own when to answer as the video plays; the assistant may respond several times, and each response is judged on both what it says and when it is delivered. The paper's central claim is that this timing dimension is essential to user experience and that existing video QA benchmarks and metrics, which evaluate only final textual content, miss it. To support that claim, the authors propose PAUC, which computes the area under a time-versus-correctness curve for each ground-truth reply turn, and show in a human preference study that PAUC agrees with human judgments better than a correctness-only baseline and approaches human inter-annotator agreement. The benchmark covers web videos, egocentric footage, TV series, and surveillance anomaly detection, with multi-turn open-ended answers and speech input on the TV task. The authors report that current models, including proactive ones, are far from solving the timing problem: replies are often duplicated or late, and adapted offline models frequently outperform dedicated proactive models.

What carries the argument

The central object is the PAUC score. For a ground-truth reply turn $g$ with ideal timespan $(t_{\rm start}, t_{\rm end})$, take the model's responses $\mathit{pred}_1,\dots,\mathit{pred}_P$ at timestamps $\tau_1<\cdots<\tau_P$ inside the span. A judge, GPT-4.1 in this paper, is given the question, the gold answer, and the accumulated responses up to each $\tau_p$, and returns a correctness score $s_p\in\{0,1,2\}$. PAUC draws the polyline through $(\tau_p,s_p)$, seeds it with $(t_{\rm start},0.5)$ to encode that silence is better than a wrong answer, closes it at $(t_{\rm end},s_P)$, and divides the area under the polyline by the maximal area $(t_{\rm end}-t_{\rm start})\times 2$. The hyperparameter $\omega\in[0,1]$ shifts timestamps left as $\tau'_p=t_{\rm start}+(1-\omega)(\tau_p-t_{\rm start})$, so $\omega=0$ keeps real time and $\omega=1$ collapses to content-only scoring; $\omega=0.5$ is the default. Per-turn scores are averaged over all ground-truth turns in a video.

What would settle it

Run a preference experiment that holds content quality fixed and varies only response timestamp: present users with two transcripts of identical accumulated answers, one delivered immediately when the event appears and one delayed until the end of the window, without telling them that timeliness matters. If a large majority does not prefer the earlier delivery, PAUC's time-axis weighting is not justified. A second check is to re-annotate the benchmark's ground-truth reply windows independently and measure how much PAUC scores and model rankings change; if small window perturbations flip rankings, the metric's reliance on exact spans is fragile.

Watch

Extended reading notes

Core claim

On the authors' terms, the discovery is that proactive video question answering can be measured, and the right measure is a curve, not a number: for each ground-truth reply with an ideal time span, PAUC plots the accumulated correctness of the model's responses at the timestamps they occur, then takes the normalized area under that polyline as the score for the turn. Because an early correct answer raises the curve sooner, and an early wrong answer contaminates the accumulated input for every later judge call, the area encodes both timing and content in one value. The human study is the decisive evidence: with the timeliness weight $\omega=0.5$, PAUC's pairwise agreement with human preferences is higher than the $\omega=1$ content-only variant, and in the [WEB], [TV], and [VAD] tasks it approaches or reaches the agreement between two independent human annotators. This is put forward as evidence that PAUC is a more faithful measure of user experience in proactive interaction than metrics that throw away timestamps.

Load-bearing premise

The load-bearing premise is that each ground-truth answer has a true ideal reply window and that receiving correct information earlier is always better for the user; if the window annotations or the 'earlier is better' preference model are wrong, PAUC's scores can diverge from real user experience despite passing its own human-study check.

Editorial extensions

If this is right

If PAUC is accepted, any evaluation of proactive video assistants must report response timestamps, because two systems with identical final text can differ substantially in user experience.
Current proactive models such as MMDuet and VideoLLM-Online are shown to suffer from duplicate, late, or absent responses; this should push training toward redundancy suppression and timing decisions, not just content accuracy.
Offline models adapted by chunking can beat dedicated proactive models on this benchmark, indicating that proactive capability is not automatically delivered by architecture choice.
The PAUC framework's judge can be replaced with BLEU, CIDEr, or accuracy, so the metric extends beyond LLM judges and can be tuned to a deployment's speed-versus-completeness trade-off via $\omega$.
The four-task design gives separate spokes: web QA, egocentric assistant, TV-series understanding, and surveillance monitoring, already covering representative deployment surfaces for proactive interaction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension of PAUC's logic is that the value of a correct response may not be additive over time: users might be interrupted or annoyed by repeated partial answers, so a curve that penalizes interruption events could track user experience even better than the current area-under-curve score.
The same area-under-curve construction could transfer to any event-window task with evolving textual output, such as live captioning, real-time monitoring alerts, or assistant notifications, by swapping in a task-specific correctness judge.
The paper's ground-truth reply windows come from content annotations; an alternative ground truth derived from when users actually interrupt to ask or search could validate or revise those windows.
The result that chunk-adapted offline models often beat dedicated proactive models suggests a training hypothesis: timing may be best learned as a separate lightweight decision module on top of a strong offline video interpreter, rather than through end-to-end proactive training.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here is the short version. The paper contributes a useful benchmark for proactive video QA and a time-aware metric that captures an important dimension most evaluations ignore. The benchmark reuses four existing datasets into multi-turn open-ended proactive tasks, and the experiments cover a wide range of offline and proactive models plus human performance. PAUC is a simple, intuitive construction: area under a time-score curve, rewarding early correct content and penalizing late or wrong output. The paper is also transparent about its assumptions.

Where it gets weak is Section 6.2. The human-preference study instructs annotators to judge which model prediction 'better captures the information present in the ground truth at earlier timestamps' - which is essentially the definition of PAUC. Table 4 therefore cannot independently validate that PAUC tracks user experience; it restates the design assumption. The low kappa (0.31-0.59), the filtering of examples, and the annotators finding the task 'cumbersome and unnatural' make the preference signal even shakier. There is also a notation mismatch in Eq. (1) (q_end vs t_end), omega = 0.5 is picked without a reported sweep, and duplicate responses are not penalized even though Table 5 shows they are pervasive. These are fixable, but the headline validation needs a neutral re-study.

Still, the benchmark and the metric are worth having. The adaptation of offline models to proactive settings is pragmatic, and the cross-model comparison gives the community a starting point. The central claim about human agreement should be read as unproven rather than wrong.

Who should read this: researchers working on streaming or proactive video LLMs, and anyone building evaluation metrics for time-sensitive interaction. It deserves a serious referee, because the benchmark will likely be used. My recommendation: send to review, but require substantial revision of the human study and the metric details before accepting.

Referee Report

5 major / 5 minor

Summary. The paper introduces ProactiveVideoQA, a benchmark for evaluating video LLMs in proactive interaction settings, where the model must autonomously decide when to respond during video playback. The benchmark comprises four tasks (web video QA, egocentric video QA, TV-series QA, and anomaly detection) with multi-turn open-ended annotations. The authors also propose PAUC (Proactive Area Under Curve), a metric that combines response timing and content correctness into an area-under-the-curve score, with a hyperparameter ω to trade off timeliness against correctness. The paper reports extensive experiments on proprietary and open-source models, as well as a human preference study claiming PAUC agrees better with human judgments than a timing-agnostic baseline.

Significance. The benchmark is a genuine and useful contribution: it is the first to target proactive interaction with multi-turn, open-ended QA across diverse video domains, and it fills a real gap in evaluation methodology for streaming/proactive video LLMs. The authors also provide thorough experimental comparisons across many models, including both offline-adapted and purpose-built proactive systems. The PAUC metric is a reasonable first proposal for scoring temporal response quality, and the paper is transparent about the difficulty of human annotation in this setting. However, the central validation claim—that PAUC aligns with human preferences better than timing-agnostic metrics—rests on a human study whose instructions embed PAUC's own design assumption, which substantially weakens the headline result. The benchmark alone is solid enough to justify publication if the metric-validation claim is either fixed with a neutral study or appropriately downgraded.

major comments (5)

[§6.2 and Eq. (1)] The human-preference validation is circular: annotators are explicitly instructed to 'assume the role of users seeking timely and accurate information' and to judge 'which model prediction better captures the information present in the ground truth at earlier timestamps, while also considering text quality.' This is precisely the quantity PAUC (Eq. 1) rewards—early accumulation of correct content within the ground-truth reply timespan. Table 4 therefore compares PAUC against a criterion that already incorporates PAUC's 'earlier is better' assumption, so it cannot independently establish that PAUC tracks user experience; it is instead a consistency check between PAUC and its own design principle. A neutral validation would need preference elicitation from users who are not told to prioritize timing or ground-truth alignment (e.g., free-viewing or task-completion setups), or at least a post-hoc analysis showing that the agreement holds without the timing-heavy instruction.
[§3, Eq. (1)] Equation (1) contains a notation inconsistency: the denominator uses (q_end − q_start) while the text and surrounding definitions define the ground-truth reply timespan as (t_start, t_end). This is not merely cosmetic; it makes the exact formula ambiguous for reproduction. Please reconcile the symbols and ensure the numerator and denominator use the same variables (presumably t_start and t_end). In addition, the text says the initial point (t_start, 0.5) is added to make the polyline continuous, but the numerator's first term (τ1 − t_start) × 0.5 is not explicitly tied to this point in the prose; clarifying this connection would improve readability.
[§3.1 and Table 4] The hyperparameter ω is introduced with a recommended default of 0.5, but no sensitivity analysis or principled selection is provided. Table 3 reports results for ω = 0, 0.5, 1, yet the human-agreement comparison in Table 4 only evaluates ω = 1 versus ω = 0.5. Consequently, the claim that PAUC 'better captures human preferences' is contingent on an arbitrary default; a different ω could reverse or weaken the conclusion. Additionally, the semantics of ω are counterintuitive: the text states that ω = 0 corresponds to 'timeliness is very important' (unshifted x-coordinates), while ω = 1 collapses all timestamps to t_start, completely removing timing. This inversion should be explained more clearly or the parameter should be renamed/reparameterized to avoid confusion.
[§3, §6.1, Table 5] PAUC does not penalize duplicate or repeated responses. Because s_p is computed on the accumulated set of responses, repeating the same content does not lower s_p, but it still adds a time interval to the curve and can increase the area under the curve. Table 5 shows that the proactive models (MMDuet, VideoLLM-Online) have very high duplicate rates (81–99%), and the paper itself concludes in §6.1 (3) that these models 'tend to repeat previously generated content, leading to lower response quality.' This creates a bias: PAUC can reward a model that responds early with low-quality content and then repeats it, which inflates its score relative to a model that responds later with better content. The metric should either discount repeated information or treat identical consecutive responses as not contributing new area, so that the scoring is consistent with the paper's own qualitative finding.
[§6.2] The human-validation sample is small and heavily filtered: only 100 turns per task (50 for [VAD]) are included, with criteria requiring both models to have responded within the reply span, at least one model to have responded in more than one round, and both to have at least one response with PAUC > 0. This filtering removes easy cases where metrics may agree and concentrates on borderline, difficult comparisons, which could overstate or understate the agreement. The reported inter-annotator kappa values (0.31–0.59) are low, and the claim that PAUC 'approaches the level of agreement observed between human annotators' is made without confidence intervals or significance tests. Please report statistical uncertainty and, ideally, re-run the preference study on an unfiltered random sample to assess the robustness of the agreement findings.

minor comments (5)

[Abstract and §1] The phrase 'users increasingly expect multimodal systems to be more initiative' should be corrected to 'more proactive' or 'more initiative-taking.'
[Eq. (1)] The denominator in Eq. (1) should read (t_end − t_start) × S, not (q_end − q_start) × S; the subscript 'q' appears to be a leftover from a draft with different variable names.
[§6.1] The observation that human annotators perform poorly because the task is 'cumbersome and unnatural' raises a question about construct validity: if the annotation protocol is unnatural for humans, does the benchmark measure a realistic proactive interaction paradigm? Please discuss this limitation explicitly near the human-performance results.
[Table 2] The row for ProactiveVideoQA uses an asterisk under the 'Multi-Answer' column, but it is unclear whether 'Open-Ended' is also marked; the legend says '*' means true for some sub-tasks, which is ambiguous for a benchmark that is claimed to be fully open-ended. Clarify the table entries and legend.
[General] There are several typos, e.g., 'recommand' should be 'recommend' in §3.1, and some references use inconsistent formatting ('ArXiv' vs 'arXiv'). Please proofread the final version.

Circularity Check

1 steps flagged · score 6.0 of 10

The §6.2 human-preference validation is circular: annotators are instructed to prefer predictions that capture ground-truth information at earlier timestamps, which is precisely the quantity PAUC is defined to reward, so Table 4 does not independently validate PAUC.

self definitional [Section 6.2 (Table 4) vs. Section 3, Eq. (1)]
"Annotators are instructed to assume the role of users seeking timely and accurate information from the video. They are asked to judge which model prediction better captures the information present in the ground truth at earlier timestamps, while also considering text quality (e.g., avoiding hallucinations and maintaining fluency)."

The human-preference criterion is constructed from the same objective that PAUC optimizes. Eq. (1) computes PAUC by integrating correctness scores over time, with earlier correct responses expanding the area under the curve; the initial point (t_start, 0.5) and the per-timestamp accumulations also penalize late arrival of correct content. The instruction to prefer the prediction that 'better captures the information present in the ground truth at earlier timestamps' is effectively a verbal restatement of this area-under-curve logic. Therefore, the Table 4 contrast between ω=0.5 and ω=1 measures whether annotators followed the timing preference embedded in PAUC's design, not whether PAUC independently matches natural, unconstrained user experience.

full rationale

The benchmark construction is largely self-contained: ProactiveVideoQA reuses existing annotations (Shot2story, TVQA, Ego4D, UCF-Crime) and the PAUC formula is explicitly defined from ground-truth reply timespans, with no fitted parameter being relabeled as a prediction. No load-bearing self-citation chain is used. The circularity is localized to the central validation claim. Section 6.2 instructs annotators to rank model predictions by how well they capture ground-truth information at earlier timestamps, which is exactly what PAUC's area-under-curve computation rewards; hence the reported agreement between PAUC ω=0.5 and these human preferences is partly guaranteed by construction. The default ω=0.5 is asserted without an independent selection procedure, and the paper itself reports low inter-annotator kappa values (0.31–0.59) and describes the annotation task as 'cumbersome and unnatural' in §6.1, further weakening the claim that the elicited preferences are stable, naturalistic user judgments. These issues undermine the metric-validation claim but do not invalidate the benchmark or the PAUC definition as a proposed evaluation tool. Overall circularity score 6: one or more central comparison steps reduce by construction, while the benchmark itself remains an independent contribution.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The PAUC metric rests on an assumed ideal ground truth with reply timespans, on GPT-4.1 as a correctness judge, and on a human preference model that equates earlier accurate responses with better user experience. The benchmark inherits annotations from four existing datasets and adds LLM-generated or manually written answers. No new physical entities are introduced.

free parameters (3)

omega (ω) = 0.5 (default)
Balances timeliness versus correctness in PAUC; no sweep or principled justification is given, and the value is used in the human-agreement validation.
initial score for no response = 0.5
Chosen to reflect preference for silence over wrong answers; the specific value is arbitrary.
correctness score scale = 0,1,2
Coarse 3-point scale chosen after experiments because finer scales showed low inter-evaluator consistency.

assumptions (4)

domain assumption Existence of ideal ground truth with reply timespans
Section 3 explicitly states 'we assume the existence of an ideal ground truth for a quantitative and objective measurement of proactive model performance.'
domain assumption GPT-4.1 reliability as judge
Section 3 uses GPT-4.1 to assign cumulative correctness scores; the paper does not validate its agreement with human content judgments.
domain assumption Human annotator preferences reflect user experience
Section 6.2 relies on a small human preference study (4 annotators, 100 turns per task) with low inter-annotator kappa.
domain assumption Source dataset annotations are valid
Section 4.1 reuses annotations from Shot2story, Ego4D, TVQA, and UCF-Crime, and adds LLM-generated QA for EGO and manual descriptions for VAD.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models." pith.science (2026). https://pith.science/paper/ZJK2577A

@misc{pith2026250709313,
  author       = {Pith},
  title        = {Pith review of: ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZJK2577A}},
  note         = {Machine review of arXiv:2507.09313}
}

read the original abstract

With the growing research focus on multimodal dialogue systems, the capability for proactive interaction is gradually gaining recognition. As an alternative to conventional turn-by-turn dialogue, users increasingly expect multimodal systems to be more initiative, for example, by autonomously determining the timing of multi-turn responses in real time during video playback. To facilitate progress in this emerging area, we introduce ProactiveVideoQA, the first comprehensive benchmark to evaluate a system's ability to engage in proactive interaction. Since model responses are generated at varying timestamps, we further propose PAUC, the first metric that accounts for the temporal dynamics of model responses. This enables a more accurate evaluation of systems operating in proactive settings. Through extensive benchmarking of various baseline systems on ProactiveVideoQA and a user study of human preferences, we show that PAUC is in better agreement with human preferences than traditional evaluation metrics, which typically only consider the textual content of responses. These findings demonstrate that PAUC provides a more faithful assessment of user experience in proactive interaction scenarios. Project homepage: https://github.com/yellow-binary-tree/ProactiveVideoQA

Figures

Figures reproduced from arXiv: 2507.09313 by the authors.

**Figure 2.** An example of user journey map. The content in the figure is fictional only for demonstration purpose. swers are overly simplistic (e.g., “Inform me when [event] happens”), thereby evaluating only the timing of the response while ignoring the quality of its content; or (2) they have only a single round of response, rather than supporting multi-turn, contextaware interactions. A more fundamental issue is the lack … view at source ↗

**Figure 3.** An Illustration of the PAUC metric. the intuition that providing no response is preferable to giving entirely incorrect answers, which receive a score of 0 from the LLM evaluator. The final PAUC score for this ground-truth reply turn is defined as the ratio of the area under this curve to the maximum possible area ( by (t end−t start)×S) as calculated by Eq. (1), where S is the maximum score. P AUC = [(τ1 − t start… view at source ↗

**Figure 4.** An Illustration of the effects of ω. (Vedantam et al., 2014), or accuracy, depending on specific application requirements. 4 The ProactiveVideoQA Benchmark To make use of PAUC, a benchmark is required for evaluating proactive models. We introduce ProactiveVideoQA, the first comprehensive benchmark designed for proactive interaction. To encompass prevalent scenarios in proactive interaction, ProactiveVideoQA focuse… view at source ↗

Figures from the paper (2 more)

**Figure 5.** Figure 5: Example data from different tasks. [WEB] [EGO] [TV] [VAD] # videos 500 326 450 101 # examples 500 326 500 101 # reply turns 1328 1575 500 107 reply / example 2.66 4.83 1.00 1.06 video len (s) 16.59 360.00 75.57 121.03 reply span len (s) 5.51 29.20 12.08 17.96 [PITH_FU…

**Figure 6.** Figure 6: Qualitative demonstration of ProactiveV [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding
cs.CV 2026-07 conditional novelty 6.0 of 10

An open 4B video MLLM with inflated-3D ViT tokenization and adaptive streaming perception outperforms comparable open models on general, long-video, and streaming benchmarks while using fewer visual tokens.
GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video
cs.CV 2026-07 conditional novelty 6.0 of 10

Current MLLMs can deliver procedural instructions in streaming video but systematically fail at real-time error detection and corrective coaching on the new GuideMe benchmark.
Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents
cs.AI 2025-10 conditional novelty 6.0 of 10

On PROBE, a new synthetic benchmark for proactive workplace problem solving, the strongest LLM agents reach only ~40% end-to-end success, with root-cause identification as the main failure point.
Proact-VL: A Proactive VideoLLM for Real-Time AI Companions
cs.CV 2026-03 conditional novelty 5.0 of 10

Proact-VL adds a learned 'when to speak' gate to a streaming video LLM and a 561-hour gaming dataset, reporting better timing and quality than prior live-commentary systems.

Reference graph

Works this paper leans on

35 extracted references · 11 canonical work pages · cited by 4 Pith papers

[1]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-vl technical rep...

arXiv 2025
[2]

Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao, Yong Jae Lee, and Jianwei Yang. 2024. Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models. ArXiv, abs/2410.10818

arXiv 2024
[3]

Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. 2024 a . Videollm-online: Online video large language model for streaming video. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18407--18418

work page 2024
[4]

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yiming Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Hui Deng, Jiaye ...

arXiv 2024
[5]

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37 -- 46

1960
[6]

Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. 2024. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. ArXiv, abs/2406.14515

arXiv 2024
[7]

Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. 2024. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. ArXiv, ...

arXiv 2024
[8]

Mingfei Han, Linjie Yang, Xiaojun Chang, and Heng Wang. 2023. Shot2story: A new benchmark for comprehensive understanding of multi-shot videos

work page 2023

Show all 35 references

[9]

Faren Huo, Yeying Zhao, Chunlei Chai, and Fei Fang. 2023. A user experience map design method based on emotional quantification of in-vehicle hmi. Humanities and Social Sciences Communications, 10:1--10

2023
[10]

Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. 2018. Tvqa: Localized, compositional video question answering. In Conference on Empirical Methods in Natural Language Processing

2018
[11]

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . Llava-onevision: Easy visual task transfer. ArXiv, abs/2408.03326

2024 arXiv
[12]

Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. 2024 b . Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension. ArXiv, abs/2404.16790

2024 arXiv
[13]

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023 a . Seed-bench: Benchmarking multimodal llms with generative comprehension. ArXiv, abs/2307.16125

2023 arXiv
[14]

Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024 c . Llms-as-judges: A comprehensive survey on llm-based evaluation methods. ArXiv, abs/2412.05579

2024 arXiv
[15]

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. 2023 b . Mvbench: A comprehensive multi-modal video understanding benchmark. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR...

2023
[16]

Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun, and Lu Hou. 2023 c . Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models. In European Conference on Computer Vision

2023
[17]

Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiao wen Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, and Jiaqi Wang. 2025. Ovo-bench: How far is your video-llms from real-world online video understandin...

2025 arXiv
[18]

Yizhi Li, Ge Zhang, Yi Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhen Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, and Chenghua Lin. 2024 d . Omnibench: T...

2024
[19]

Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. 2024. Streamingbench: Assessing the gap for mllms to achieve streaming video understanding. ArXiv, abs/2411.03628

2024 arXiv
[20]

Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Ying Shan, and Chang Wen Chen. 2024. https://arxiv.org/abs/2409.18111 E.t. bench: Towards open-ended event-level video-language understanding . Preprint, arXiv:2409.18111

2024 arXiv
[21]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics

2002
[22]

Rui Qian, Shuangrui Ding, Xiao wen Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction. ArXiv, abs/2501.03218

2025 arXiv
[23]

Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. 2023. Ego4d goal-step: Toward hierarchical understanding of procedural activities. In Neural Information Processing Systems

2023
[24]

Waqas Sultani, Chen Chen, and Mubarak Shah. 2018. Real-world anomaly detection in surveillance videos. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6479--6488

2018
[25]

Lawrence Zitnick, and Devi Parikh

Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2014. Cider: Consensus-based image description evaluation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566--4575

2014
[26]

Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, and Dongyan Zhao. 2024. Videollm knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction format. ArXiv, abs/2411.17991

2024
[27]

Yuxuan Wang, Yueqian Wang, Bo Chen, Tong Wu, Dongyan Zhao, and Zilong Zheng. 2025. Omnimmi: A comprehensive multi-modal interaction benchmark in streaming video contexts. ArXiv, abs/2503.22952

2025 arXiv
[28]

Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen2.5-omni technical report. ArXiv, abs/2503.20215

2025 arXiv
[29]

Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, Lingpeng Kong, Qi Liu, Yuanxing Zhang, and Xu Sun. 2025. Timechat-online: 80\ ArXiv, abs/2504.17343

2025 arXiv
[30]

Tongtong Yuan, Xuange Zhang, Kun Liu, Bo Liu, Chen Chen, Jian Jin, and Zhenzhen Jiao. 2023. Towards surveillance video-and-language understanding: New dataset, baselines, and challenges. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22052--22061

2023
[31]

Pan Zhang, Xiao wen Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen,...

2024 arXiv
[32]

Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024 b . Long context transfer from language to vision. ArXiv, abs/2406.16852

2024 arXiv
[33]

Weinberger, and Yoav Artzi

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675

2019 arXiv
[34]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[35]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-vl technical rep...

arXiv 2025

[2] [2]

Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao, Yong Jae Lee, and Jianwei Yang. 2024. Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models. ArXiv, abs/2410.10818

arXiv 2024

[3] [3]

Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. 2024 a . Videollm-online: Online video large language model for streaming video. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18407--18418

work page 2024

[4] [4]

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yiming Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Hui Deng, Jiaye ...

arXiv 2024

[5] [5]

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37 -- 46

1960

[6] [6]

Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. 2024. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. ArXiv, abs/2406.14515

arXiv 2024

[7] [7]

Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. 2024. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. ArXiv, ...

arXiv 2024

[8] [8]

Mingfei Han, Linjie Yang, Xiaojun Chang, and Heng Wang. 2023. Shot2story: A new benchmark for comprehensive understanding of multi-shot videos

work page 2023